[Music] [Applause] [Music] [Applause] hey everyone welcome to another video in this video we're going to be looking at how we can scan files and directories to uncover hidden cigarettes secrets what i'm referring to is api keys credential periods security certificates really really sensitive items that we often use when we're programming or writing code the problem is we want to make sure we kind of have visibility over where these secrets are now one of the worst places they end up is inside our git repositories and there's lots of videos you can look at about how to scan uh your git repositories or how to set up commit hooks to be able to detect these secrets but in this video as i said we're going to be looking at how we can find these inside file directories and file paths now i think it's important to note some of the key differences when we scan git repositories we scan the entire history so that means that if you've committed a secret and then removed it and committed over that and made a whole bunch more commit since since then that secret will still exist in the history of that git repository when we're scanning file directories even if they are git repositories we don't scan the whole history of it we just scan what is available in that file directory so that's just an important note to kind of know when we're scanning these files because if we are going to download a whole bunch of git repositories we scan them and we don't find any secrets doesn't mean that they're not there in the history all right so now that that's out of the way we can move ahead and look at how we can do this so the first step that i'm going to do is i'm going to head over to the ggshield github page now gg shield is the open source cli tool from guardian that we're going to use to detect these secrets so if we scroll down what we're going to be looking at is the scan function so we're going to go down here to scan and as you can see here we have the path option so we're going to be using scan path which is there to scan files and directories with a recursive option recursive option means that it's going to go further into other folders or other directories within that directory so it's going to keep going until it reaches basically the end of all of those directories without this option it's just going to scan that one level so we definitely will most likely want to use the recursive option here so we can go ahead and do this so i have my terminal open here and obviously i have gg shield installed if you don't have ggc installed you can use the command pip3 install gg shield and that will go ahead now i have mine installed already so um that may look different for you so the next thing we're going to do is we're going to be running that command so let's go gt shield scan path now again we want recursive and then we want our final path here so i'm just going to copy in my file path so i have a folder uh so users mckenzie documents repositories and then i have a file here called backup so this backup is the file directory that i want to scan so i'm going to go ahead and do this and unfortunately we get an error so the reason why we get an error is it says down here that get guardian api key is needed so we utilize the get guardian detection engine which is quite powerful but it does mean we need an api key to connect to it so we're quickly going to generate one of these so over in the get guardian dashboard we can go to dashboard.guardian.com if you don't have an account you can create one in a couple of seconds it's totally free you can even sign up using github if you wish and then on the left we have an api tab so we're going to go ahead and create a new api key and i'm going to call it directory scanning and we just want the scan scope we don't need to worry about the incident scope at this stage we're going to go ahead and create that and this is our api key here now remember i'm showing you mine but only because i'm going to delete it later but this is sensitive so don't share it with anyone on youtube like i am and now we need to add this in as an environment variable now we can do this in a couple of ways we can use the export command which will to create an environment variable for this terminal for this kind of terminal window but then we'll have to do that each time i'm going to have to store that that key somewhere we can also add it in to our bash rc file for example so we can load it in regularly or we can add it in using a dot env file because we need to have the dot env file in the directory that we're navigated in so not the directory that we want to scan but the directory that we're currently in which is my root user directory but that's what i'm going to do so i'm going to create a new file called dot we're going to open this file and in here we're just going to create a variable git guardian underscore api underscore key now it has to be named this and we're going to go ahead and save this file here now we're just going to run that exact same command that we put before and this time hopefully it should work so we already got to the next stage and it's asked us it's calculated how many files in total will be scanned so it's 36 do we want to do this yes now obviously it might be a whole lot more in a real life example but here we have the results and we have quite a lot of them so this has been uh a project that i've been using to kind of test out some some different secrets so hopefully you don't have this many secrets inside your directory but maybe you do so you can see here it actually gives us a lot of information so we have the type of key that is here we have a postman api key and what's cool is that we have validity checks as well so it's actually saying that this is an invalid postman key and so we can go down here we have slack box tokens again it's actually invalid but here we have an aws key which is saying that it's valid this is just a cool feature to be able to show that when we're scanning this we can really quickly check which ones are critical and need immediate attention and which ones we can kind of investigate a little bit later on so this is great if we have like this a huge amount of incidents so we can go through and and review all these and it also gives us the file that we are that we're in and if we want we can actually export this information into a json file so that we can review it or add it into a different dashboard later on so let's quickly have a look at how we can export this as a json file so back in my terminal we can run the exact same command as before but we're just going to add one small option in so just quickly looking back at the github page we'll see that we have some different options for instance you'll notice that the secrets were all hidden um those where they are what's in it but we didn't we can show secrets to be able to show all the secrets in the cli and here we have the option of json so we just need to add this after um just where our options go so we move our across and we're just going to add in here json we have two dashes run that and again it's going to prompt us yes do we want to scan the 36 files of course and then this time instead of giving us that nice display it gives us our json format now obviously on the screen this doesn't look all that appealing but when we add this into a json file or we handle it correctly or input it using a program then obviously we can manipulate this data and be able to visualize it in different ways which will help us remediate these incidents so there you have it that's how we can scan files and directories for secrets using ggshield i hope you found the video useful give it a like if you did write me a comment if you're having any trouble or any questions in the comments section or you can reach out to me on twitter at advocate mac always happy to hear from you stay tuned for more gg shield videos and thanks for tuning in [Music] you