šŸ“Š NEW! Voice of Practitioners 2024: The State of Secrets in AppSec

READ REPORT

šŸ“Š NEW! Voice of Practitioners 2024: The State of Secrets in AppSec

READ REPORT

How to permanently remove files from git and rewrite your git history

In this video I will run through how to permanently delete files and rewrite your git history using two methods. The git reset command for simple scenarios and using the BFG repo cleaner for more complicated real-world examples.The steps on this video can be following at the blog https://blog.gitguardian.com/leaking-secrets-on-github-what-to-do/Link to BFG Repo Cleaner https://rtyley.github.io/bfg-repo-cleaner/

Video Transcript

hey guys welcome to this video in this video we're going to talk about how to rewrite your git history and permanently delete files now before we start it's extremely important to say that this isn't a trivial task and if you have multiple people working on a single git repository that proceeding and rewriting your history is going to affect any changes that they've done that haven't been committed I also want to say that if you've uploaded a sensitive information to get particularly to public get like an API key or a database credential it's extremely important to revoke this key or secret before we proceed the reason is on public it people can and they do monitor this for leaked credentials so that they can exploit it so before you get started please make sure that any secretive information has been revoked so with all the warnings out of the way let's begin to illustrate this I created an extremely simple project on github called hello world and I just have two branches the master branch and a branch I've called feature X now in this scenario what we're going to pretend is that we have this file here called config it's a fight Python file and inside this file we have a secret key that we've accidentally uploaded into our git repository so I'm going to add this into status and I'm going to do a new commit called v commit and push this in to our remote repository it should go without saying but just in case please don't ever upload configuration or environment files into github especially if they have secrets in them but it's also just best practice to get out of the habit of doing this because it will get you into trouble or one day now if we go back to a github page and refresh we can see that now we have four commits and a config file a conflict of pwhile is in there now I had get Guardian install on my repository what this does is it automatically scans for any secrets that may have been leaked you see the Red Cross so that means it's discovered something I would highly recommend everyone to install get Guardian on to their repositories it's free and it will save your life if you do accidentally upload a secret now if you're just working on this by yourself and you need to quickly rewrite your history then there is a simple way to do it but it does cause major troubles of team so the simplest way which will briefly touch on is to take advantage of the git reset function and we can use the soft head flag in this what this does is it takes the head that we were on with our latest commit and it essentially moves it back one so anything that was previously committed it can now be undone the problem with this is that if you push this up into your repository then you're going to get an error this is because that the history that you're trying to push is essentially behind the history on the remote repository to fix this we have to use the force flag so we can go get push force and this will essentially force the remote repository to match the history on our local machine this is extremely dangerous if you are working in teams so please be careful but this is the simplest way if you've quickly done a mistake you immediately find out and you want to revert back to the previous commit after we've run this command if we go back into our remote repository you'll see that we only have three commits now when before it was saying for our config file is gone but if we actually click here onto our repositories history you'll notice the entire fourth commit that we did is now gone but this will only work in very simple scenarios where you or maybe one other person has just done a commit and you realize it back in the real world your get history and your get branches probably looks a little bit more like this and what that means is that simply removing commits back might not solve the problem especially if the file that you're looking for is on different branches if it's back in multiple versions of different commits then you're going to have a real nightmare trying to remove it using this method so in this more complicated example we're going to use actually a different tool for this we're going to use a tool called The BFG repo cleaner and what this tool allows us to do is pinpoint a file by its name and it will search through all the different branches and all the different history within our local machine and it's going to remove that file so moving forward on this example what we're gonna do is let's start by adding back in the config file and the ethics Python file that we previously removed so get add all we're going to commit this using the same tag fourth commit and we're going to push this to our repository and now just for this example I'm going to create another commit and I'm gonna add just any old file so I'm just going to create a random file here called new file because I'm very original and I'm going to commit this back into our git repository and I'm going to use the message fifth commit shoot I forgot to add the files okay add the files and we will do this again add it into a commit with the message v commit so now that we have pushed these up to our remote github repositories we can quickly check that out we see our config file which is in our fourth commit and then we also see a new file which is in our fifth commit so just for illustration purposes you can see that this is what our tree now looks like we have the file that we accidentally uploaded in our fourth commit we have a new commit and we're going to want to go back and delete just that one file from that fourth commit so now that we've got the scenario set up let's head over to the BFG repo cleaner and let's download the latest version and right now that is 1.1 3.0 but while that's downloading what I am going to do is I'm going to clone my entire repository into a new folder so I'm just going to quickly create a new folder here called hello world clone and I'm going to clone the repository into this now I'm doing this just so that I have a backup of my old one and that this one I'm comfortable playing around with now at the moment we were in the master branch by default so let's check out our feature X grunt and we can now see that we have the config file and the new file let's start by getting everything clean so let's just get rid of that config file and trash that now we're going to add we'll add all into this commit and we're going to create a new commit and we're going to call this clean commit so this is our baseline and this is the last commit that we want to do where everything is as if exactly how we would want it to be going forward okay now we're going to grab the Java file that was downloaded from BFG and we go to plunk this straight into our repository file so we're going to a call this Java file type in the file name B if g - 1.1 3.0 da and we wanted to look in our dot git folder this is the directory we're going to use the flag delete files and now we're going to put in the file name that we want to find which is config and you can find these commands on the blog post that I'll link in the comment sections if you just want to copy them alright now let's run this and you're gonna see here that it successfully gone through its or altered some of their sha-1 values but we won't worry about all that too much the next step that we want to do is we just want to make sure and go through our git history to double-check that this config file has actually been removed so now let's have a look into our log file to double-check so I'm gonna go get log and use the flag one-line this just keeps it nice and neat and then we're going to have a look at it now remember we uploaded the config file and the fourth commit so what I'm gonna do is I'm going to copy a sha-1 value in the fourth commit and I'm going to check this out and a detached hit this is basically just searching back through different areas of the history and the git and as you can see new file disappeared which was what we want and our config file hasn't reappeared now what this means is that it means the config file has been deleted throughout again history if you have multiple commits that you want to check it's worth just checking a few of them and going back as far as you can to make sure now we need to push this back up to our remote repository obviously it may be worth considering whether or not it's best just to delete the entire repository it's remote and start fresh because all your history will still be there but in this example I'm again going to use the push a force flag but first let's get back to safety so I'm going to check back to my master branch and I'll go to my feature ex branch again and now I'm going to push this with the force flag to our remote repository all right now we should be getting close to done let's this check out our repository online we can see here that we still have six commits this is what we wanted the clean commit was the latest one we have our fifth and force commit still there now remember it was the fourth commit at our sequences in we can see that our new file is in our fifth commit and if we check out our fourth commit on this branch will see that our FX file is there but the config file has gone so this means that it's worked and we have successfully been able to rewrite our get history and remove that config file from there permanently I hope you found this video useful if you have something specific that you would like me to do please feel free to mention it in the comment section and if you found this video useful please consider subscribing to the gift guardian a YouTube channel where we're going to be uploading a lot of great videos and tutorials like this and we'll be more than happy to create videos that you guys want