CodeSecDays 2024 - Join GitGuardian for a full-day exploration of cutting-edge DevSecOps solutions!

Save my spot!

CodeSecDays 2024 - Join GitGuardian for a full-day exploration of cutting-edge DevSecOps solutions!

Save my spot!

Detect intruders in your software supply chain with honeytokens - CodeSecDays

Learn how to protect your software supply chains from constant attacks by using honeytokens. These decoy secrets can be placed in various parts of the DevOps pipeline to trap attackers and expose them. Join the session with speakers from GitGuardian to discover effective strategies against spearphishing and other threats.

Video Transcript

well thank you Mac uh and hi everyone I know this is the very last session so you've been with us and our amazing partners for the past few hours thank you for sticking around uh I really can't wait to show you what Stanislas and I have in store for you you're going to love honey token well or at least I hope so uh so we've probably heard this in every other session today our attackers have grown smarter in the past few years uh this is no news for you you're contaminating the open source ecosystem manipulating developers with phishing campaigns I myself remember getting suspicious emails asking me to sign into my circle CI account a few months ago and it turned out this was not Circle CI sending it so simply put attackers are using every possible tactic to find a way in and this is where it gets interesting because once once they're in and uh I hope this has not happened to your organization previously they will start looking for hard-coded secrets and we're going to look at certain ways to detect those attackers as soon as they find hard-coded secrets in in your environment but first let's take a look at a few examples of real world attacks in which such scenarios have have happened so I I don't know if you remember but uh in in January uh or late December Circle CI was notified about a suspicious GitHub oauth activity on its accounts uh this triggered an immediate investigation they discovered that this customer's oauth token had been compromised but it was it wasn't just one customer it was all of circle Ci's customers so they started rotating those keys and a few days later they pieced together the whole attack so they discovered that their attackers had deployed malware on one of their Engineers laptops and this has allowed them to steal two-factor-backed uh authentication cookies and this helped them impersonate the engineer before gaining access to Circle Ci's internal systems so from from there they started roaming freely getting access to customer environment variables tokens and keys and it really shows the intention of going after uh your target secrets now another one here I really love is from Wiz I don't know if you can still see my screen yes so this one for me is is quite interesting and I love coming back to to this one whenever I have the chance so in in December uh with research disclosed supply chain vulnerability in IBM Cloud databases uh for postgres health keychain basically involved three exposed secrets and an overly permissive network access to uh to the build servers so they started they they had a postgres instance hosted on the IBM Cloud databases service and what they did was uh well run a first SQL injection attack this has given them the ability to become a super user and Elevate their Privileges and once they did that they realized they were operating in a in a kubernetes cluster and they managed to get the secrets from from this from the Pod they were in and once they pulled those Secrets they started exploring what was around there and they saw that they have direct access to the container registry images of IBM so they simply pulled all the images from this container registry and started scanning them for secrets with an open source tool which is called Yelp detect secrets and once they got their hands on on those secrets that were in the container images manifests they found FTP credentials artifact repository credentials but also the end point that points to the build servers so this is a very interesting supply chain attack where the Wiz researcher research team has shown that they can gain full control and compromise the build system here and also inject malicious images that can then Ripple down and be consumed by all uh IBM customers and really the the key here is that we expose Secrets they uh came across plus uh the different URLs and endpoints they managed to get from different areas and now the last one we have is the Uber breach we're also a hard-coded secret was was used again here so uh this is a quite simple one where the attacker used social engineering to gain access to uh an employee's workstation and from there they infiltrated the Uber Network so uh they got VPN access started scanning the infrastructure saw there were some Powershell scripts uh in in the network share and that Powershell script contained one exposed secret and just one but probably the most dangerous one uh Uber exposed there this was the admin credentials for their privileged access management system so this system hosts all sorts of Secrets and credentials for other tools that Uber is using internally and from from there they managed to well move laterally but this is not just lateral movement it's a full organization-wide I.T takeover really because they got access to the AWS accounts Google Cloud platform all internal tools like slack security tools like Sentinel one and much much more so all of these scenarios whether there were real incidents and breaches in the case of circle CI and Uber or simply a disclosed vulnerability in the case of uh with an IBM really point to the importance of protecting the secrets but also to the intention of attackers so they really want to look for your secrets before deciding what their next move would be so from from from this position we can uh decide to turn the table on attackers we know attackers are looking for our secrets so maybe we can trick them to use some specific secrets that will trigger alerts right this could serve as an early warning system or a detection for our intruders now first when you want to secure your software delivery pipeline of course we must address key issues like securing developer access preventing Cloud misconfigurations protecting your applications in real time setting up all sorts of scanners in your CI CD pipeline to make sure that you're catching the vulnerabilities before they go to production but with all this we can't really completely eliminate the risk and this is very difficult no team no security team in the world is able to say we're Bridge proof because we've put all these controls in place but still you should put them to bring the risk to a minimum now once you've done that what we recommend doing is also assume uh uh adopt a poster where you're assuming a breach it's already happened and this comes in really handy because it's like you now putting your detective hat on and you're being proactive and here intrusion detection can help us collect indicators of compromise and sniff out The Intruders if they manage to find a way in well this really adds an extra layer of security in your software delivery Pipeline and it's a final layer of defense that lets you know if anyone has managed to to get in the idea really behind intrusion detection is to make persistence in your software supply chain uh an impossible mission to achieve because usually when attackers are in um they can be there for days weeks or months on average before being caught and we we don't want that to happen we want to know as early as possible one thing you could do here are you still all seeing my screen because I see a question from uh success key okay so we're still live uh so one solution here to finding uninvited guests is to use uh honey tokens or decoy secrets so please don't provide access to any real resources or data instead they will trigger alerts and reveal information about the attacker who's attempting to use them in case they're they start tampering with the key and you'll be the first to to know and find out so here we're just showing an AWS key which is an inactive honey token so don't try pinging it uh but honey tokens can come in all shapes and sizes these could be database connection URLs servers or it could be even an entire kubernetes cluster it could be a honey infrastructure if you want and one of the goals also behind uh honey tokens is not necessarily to slow down your adversaries that's not the highest value you could uh get from honey tokens the most valuable thing is is really a sensor and this quote from Andy Ellis who's a an ex-chief a security officer at Akamai really really speaks to to the value of Honey tokens here now at good Guardian we've been uh working super hard for the past six months uh to introduce this concept of Honey tokens and intrusion detection to the software delivery Pipeline and all the components that that make it up and a few months ago we released our second product which is get Guardian honey token so if you're familiar with Guardian you all you probably know our first product Secrets detection uh this one finds and eliminates secrets in your uh in your sdlc the idea is to reduce risk reduce your attack surface but honey token complements the approach here and helps you lay traps within the same areas of your software delivery pipeline in order to detect Intruders so our take is you should be using both uh both detecting your own secrets that give access to real resources and real data and real systems in order to reduce risk on that side of exposure but also a couple of that would get honey token for you to be the first to find out if someone is is in your in your systems and for for that matter we're helping you create and deploy AWS Secrets which again are our decoy Secrets here so you may ask why why use AWS Secrets as honey tokens uh well what we noticed is first they leak a lot and second the attackers love them so when we ran uh our uh study on public GitHub last year and the previous one we saw that for every 10 000 commits on GitHub there are about 84 exposed AWS secrets so they're ranking really high in the in the types of sneakers that leak the most but they're also easy to be uh found and detected by attackers because attackers are using tools like bit League trufflehog Yelp detects secrets to find AWS secrets in uh in in different environments now these AWS Secrets can help you do two things uh on the one hand they can help you detect intrusion but git Guardian has also worked on another use case to help you detect leaked code on GitHub repositories so you can deploy honey tokens in your private code base self-hosted or manage devops tools you can even deploy them on your developers laptops and be alerted at the slightest hint of tampering but you can also deploy them in your third-party accounts let's say a hosted service like GitHub actions you can inject a honey token and have it sit next to other secrets in in that Pipeline and see if GitHub actions ever gets compromised or if the service provider gets breached then you will know that before it's even disclosed publicly and now the last use case is detecting if your honey token has leaked on github.com so this is something we can do because we monitor uh GitHub in real time so we can look at all the commits that come to to the public side of GitHub and if we spot one of your honey tokens we'll let you know and this will probably alert you to a potential code leakage now the entry points to uh your software supply chain can be multiple and we saw this in the previous uh section on uh the real the real life scenarios and attacks so you need to aim for uh extensive coverage here we're talking about uh using Secrets use secrets and honey tokens just like you would be uh using secrets in a way that honey tokens will not raise your attacker's suspicion so you want to lay them in your version control system this is your source code you want to have them in your CI CD pipelines but you also could put them in your sequence managers this could fit in your hashicorp Vault for example you could have those in your Registries and also productivity tools developers use day in day out so this is jira Confluence or or Slack and to finish uh before we get to show you the demo I just want to walk you through the different steps that stanislats will show in the git Guardian workspace and product here uh so basically you start by creating a honey token and then you deploy it in one of your sensitive assets again the idea is to aim for an extensive coverage here and then get Guardian will be continuously monitoring all these honey tokens for you uh we'll see if they get tampered with if anyone tries to get access to them and we'll let you know and lastly if an attacker trips on one of these honey tokens well we'll let you know so you you get notified so you can kick off investigation you can look at all the contextual data we're providing so the IP address different blogs and trails Etc so that you can kick off your incident process your incident response process sorry and if the alert is accurate so you're able to contain your attacker very quickly collect evidence and then recover from from this attack and now I'm leaving the floor to Stanislaus to show you the product in action you're going to love this thank you yet for this presentation I'm going to share my screen that if you can confirm set the screen is indeed visible foreign yes okay perfect so as yet mentioned we're going to demo only tokens and we are in fact going to demo two different scenarios reflecting the values that this tool can bring to your organization uh we're going to demo a first scenario of a publicly uh public clicks Can Happen by mistake most of the time there are the mistakes and not willing leaks so we're going to put ourselves in the shoes of a developer who makes mistakes and leaks file containing credentials I will discuss this a bit more in a while and we are going to Showcase also how only tokens can be used to detect an internal Bridge into your VCS so to start off this demo what I'm going to do I'm going to use our only tokens UI to create 200 tokens I'm going to create a first any token as they're mentioned as a moment C type of any tokens that can be created is AWS keys so I'm going to create a first any token only token public demo I can give a name to my token I can write a short description here and I can put some labels on my token so potential use case for the labels could be associating a label to a particular location where you're going to place a little can because see honey tokens it's quite valuable to place them into a lot of different systems for systems like VCS beta bit lab when you place them if we are connected to the VCS already we are going to be able to automatically detect the repository in which the token was placed but for systems like slack if you place the tokens directly on developers machines you will need to track the location through the labels so here I'm going to say okay location public repo just for the demo and I'm going to create my honey token I get confirmations that the honey token has been created I can directly here copy it to my clipboard and what I'm going to do I'm going to put place this on the token into this file uh this file contains basic code at the moment it's a demo file and it contains a tiny token at the end and I'll explain a bit later what it does if you can see it it's here uh it's a dummy key of course but it's here for a good reason so I'm placing my only token here into this python file uh placing myself in the shoes of a developer I have this file which is covered by Honey token now I'm going to commit my change adding the honey token into the file and I'm going to do a mistake so at this stage potentially the developer has been working on his local machine on this file he could be either directly pushing this file into a public repository or more likely it may switch private repository by mistake to a public repository addition accidentally exposing this file to the public internet on GitHub so for the purpose of this demo I'm just going to push and I'm going to push to this repository and you can see it's a public repository so I'm going to refresh so we can catch my push you can see there's a third commit that just appeared here and we can see indeed on GitHub that's a file containing my honey token has been uploaded so the public click just occurred we have a file containing a honey token which was leaked and exposed publicly on public GitHub this file contains some more code and obviously as an organization you would be interested into knowing when this file containing your code is getting Exposed on this public internet that's where what we're going to see in the rest of the demo for the purpose of the second scenario I'm going to create a second honey token this time I'm going to name it only token private repo demo I'm going to States a label which says it's a private triple I'm going to create this one copy it again I'm going to place it into another file this time so it's a it's a fake file that contains a lot of stuff and that is going to contain in addition to the already existing Secrets found within the trial this only token this file now same here I'm going to commit my changes and I'm going to push it and one thing I want to mention here you've seen me committing uh tokens putting tokens contained within the files using my command line you can see the no secrets have been found here which is because I'm using JD shield to protect my code from accidentally leaking Secrets through the use of our pre-commit hook and here because I'm using Garden honey tokens created on my dashboard GG Shield recognizes that these are only tokens not real secrets and it doesn't bother me with those so there's integration between the secrets detection capabilities of the guardian of course and our only tokens capabilities so my point here is I have made my commits I'm going to push my changes into a repository and this repository it's this one I'm going to show it to you it's this one it's a private repository this time on github.com and the the idea behind this repository is that it represents your VCS it's your private repositories as a whole uh so I'm not going to be exposing this publicly but the idea is that we're going to show what could happen if there's an internal verge of your private repository so let's just double check my push as succeeded and I can see if I go in the file that the honey token was successfully added now let's take a look quickly at the UI of Honey token uh to be able to Showcase a bit more I'm going to switch to another workspace where we've got quite a few more totally tokens created and to explain to you how you can navigate this UI so uh the interesting thing is that we try to make this as user friendly as possible so you can filter your token on token types of course as I said currently there's only variable us tokens which are available you can use the different labels that you've put in place to be able to search for particular only tokens uh placed into particular locations so you could also use those to mark for the different environments the honey tokens have been placed in if you've got any tokens that were only placed into test environments you can look for them there you can filter on the source of the Sony tokens so that's going to be available if the honey tokens are placed in repositories and the repositories are connected to git Guardian for Secrets detection using this connection we can find out where the running tokens are placed and allow you to look specifically for the tokens found in a specific repository allowing you to you know look for a of repositories you can filter per statues of the token active means the token has been created but you're in a good place if your token is active because it means no one has tried to use it yet triggered means the token uh has been created and not only hasn't been created someone tried to access it so if your token switch is to triggered that means you should start launching an investigation to understand okay was it triggered by mistake or was it triggered because it was exposed publicly or because it was leaked because someone tried to access it is it someone internal is it someone from the company or is it an external protector you'll need to look into it at this stage and the revoked options or revoked statues for tokens is when you should use after you have been breached after you've determined that okay this there was a token that was triggered we investigated we determined that it was triggered for a good reason uh what you're going to want to do if that's the case is that you're going to want to replace the honey token which is currently in the file that was leaked or breached by your new Honey token so that you know in the future if there's a new unauthorized access to this file so you're going to revoke the previous Sony token and rotate it with a new one exactly like you need to rotate and revoke Secrets except this time you're not going to put your honey token into a vault of course and you can also filter using tags at the months or the only tag we have is a publicly exposed Target which is placed on holy tokens that we see on public GitHub um let's switch back to our workspace where we need the demo and there's one thing I wanted to show you as well it sets thanks to the connection with your VCS you can also have visibility on your perimeter and its status from a honey token point of view here I in front of you created two honey tokens that I placed into repositories I have 10 repositories within my demo workspace and you can see that it's showing to me that I have currently a 20 coverage of my repositories with any tokens two sources where say honey tokens were placed properly and eight sources without any tokens I'm not going to be showing them to you but thanks to this page you can have an easy overview on which repositories are safe from your honey token point of view and which ones are not so safe because certain tokens were triggered here you can see on my two repositories so there's this one which is a public repository and this one which is a private repository my running token placed in my private repo is safe however the public one has been triggered so we are going to look into this and I'm going to click on this it's going to redirect me to the honey token UI pre-filter on the right repository and there I can see indeed okay my token has been triggered I have the publicly exposed diet let's investigate this so I click on my token and I arrive on this all Details page for the honey token where it gives me some information to help me investigate I have the token name of course I have who created it when I have a reminder of the code of the token the labels on the description the source which was detected automatically thanks to the connection with the VCS uh any on the right side of the panel uh we've got this status reminder so here it tells me my only token is in the triggered there has been nine open events and we are going to get the details of those very soon uh the incident has been ongoing for six minutes and I have the public Keys post tag an interesting thing is that if you click on the publicly exposed tag it gets a link to you and you can click on this to see where on the public internet this only token has been found uh that's useful for investigation purpose now let's take a look at the events uh what's interesting here is we have all the interactions that were done with the token uh visible there are three events you can see they are ordered according to time but of course I can change the order and what you can see and what's interesting is that on these nine events the first ones who were on the crime scene where AWS that is the case because from uh I would say a very pragmatic perspective AWS had created a I would say a partnership with GitHub to be among the first one whenever there's a new AWS keys that clicks on public GitHub because in the past there has been quite a few occurrences of AWS customers mistakenly leaking AWS keys on public GitHub this key is being located bias protectors those protectors using the keys to for multiple I would say various varied purposes but there's quite a few stories around this Keys being used for mining Bitcoin and resulting in huge bills for the AWS customers so AWS took action and uh as the GitHub okay please warn us whenever you see an AWS key linked on public GitHub and the first thing AWS will do is this section you can see there's an edit attached user policy AWS takes the initiative and restricts the rights Associated to that key to ensure that exploiting it is not going to be as easy as it could pretty good move from them if USB four minutes later you can see the external requests starting to come in and why the delay um because you can see uh at 50 AWS was on the scene but at that time the token was not yet Exposed on the public GitHub API so AWS was there first and it takes roughly five minutes for GitHub to push to the public API code that has been pushed there's a small relay and from the moment this delay is goes through the the only token and any code that gets published publicly and GitHub is accessible for everyone you can see here among the first ones on the scene where these three IP addresses with public monitoring Garden public monitoring tag attached to it that's our public monitoring solution so we see this token being available publicly and that allows us to tag the only token with publicly exposed attack but we are not the only ones on the scene you can see literally at the same time other IP addresses and these do not belong to git Guardian for these three different IP addresses taking an interest into the Sony token you can see there are different user agents as well and you can see some of these user agents are using travelogue to scan for credentials so that means other people have been monitoring the activity on public GitHub but we knew that and those people are using credential scanners immediately as soon as the code is exposed publicly to look for those credentials so at that stage if we were a developer who had done a mistake and who had been leaking corporate credentials accurs would have found them thanks to the automated use of this credential scanners and their monitoring of public guitar so good good news uh you were using gilgart and honey tokens so you are one of this because in the UI of any tokens of course it's reflected but we also trigger email alerts and we can trigger custom workbook notifications that can go into your slack into any system that receives it and once you as fast as possible whenever there's a public exposure event happening you can also see the timeline of the incident of the running token being triggered there that gives you an overview of how fast things are going now let's take a look at our second scenario here our first scenario was this public exposure scenario as I told you when I leaked this file at the bottom of the file there was this token it was added before I placed it before and the point is that this token is a fake GitHub token as the idea is that if it was a real token form from novice yes threat actor who was monitoring public GitHub using this credential scanner and could then try to use it to access your internal VCS your internal repositories your code base when that happens uh one of the first thing the attacker is going to do is to look for credentials inside the VCS uh they are quite valuable to them they will allow the attacker to move laterally into other systems so they have a few options to do that they can go through the repositories to which the indexes by hand scan them using their eyes that's going to take a while or they can use automated credential scanners like trufflehog like clicks and we're going to show you what happens when they do that so as a reminder the file that is in your internal VCS is that one the token is there it's still not exposed publicly it's still private uh the only token is not triggered you can see it's still active and not triggered but the attacker gained access to your repo so what we're going to do we're going to run travel and use it to scan four Secrets now truffle is currently scanning it's finished scanning it found quite a few things inside this file tons of different secrets and you can see there's a green ones here and this means truffle found a valid Secrets or what it seeks is a valid secret it verified it and ding ding ding Bingo you found the AWS key however when doing so also triggered the only token which was located into your private repo you can see it here this time it's my IP uh see user agent referral and you can see I did this is called triggering the honey token so that means at this stage again your security team can be made aware of the fact that this time your internal VCS your internal repositories were breached and you have an attacker inside at this stage of course we strongly strongly recommend investigating the issue and potentially if you realize that yes this is an external attacker and it's not just today who made a mistake at this stage you should probably take some remediation action depending on your companies that can be quite a lot of different processes but it could be restricting access to the repositories making sure the credentials within there are properly revoked there's a lot of stuff that needs to happen at this stage and that concludes the demo I hope it was clear for you