we found 10 million Secrets last year just last year 10 million secrets in public repositories on GitHub what you're referring to is more like well all the comments you've done so far I can still as a public person click on this link yeah and see what you've done in the past all the way to the very first year you started working on it yeah exactly that and so there's lots of ways that Secrets kind of get leaked too so GitHub has a public API anyone can view it you don't need authentication there's a public Ledger of every single repository yeah and everything that's happening I just want to say thank you for all the love you showed Shelby and I at coupon in Amsterdam as well as RSA in San Francisco over the past couple of weeks I cannot be grateful enough that showing slow slap to us you've supported us to grow on social media and on Spotify and all these podcast platforms so thank you so much for all the love and support you sure thank you so much for coming and saying taking selfies with us and some of you are grateful enough to come on my daily Vlog which I posted LinkedIn and Twitter thank you so much for be powerful life and making it what it is today all right let's get to the episode before I get emotional have you ever wondered are there more secrets than simple username password in a world of cloud apis microservices and everything else that comes with machine to machine communication we have developed a lot more kinds of Secrets float around whether it's AWS access keys or whether it's your API access token it's a lot more information that floats around on the internet as secrets in this episode we had Mackenzie Jackson from great Guardian the recently released a report after scanning all the public repositories on GitHub how many repositories had Secrets just lying around notebook in its name passwords or AWS access Keys a lot more different kind so in this conversation we spoke about what are the kind of Secrets what can you do about finding Secrets why sometimes the only option you have is just to recycle the keys instead of just trying to go and removing them it let's be honest if the key is on the Internet it's just on the internet you never know how many copies have been made of it so you better off just recycling the keys in the first place and as easy as it sounds when I say it's a lot more difficult so When developing some keys make sure you have a recycle process that you're evolving as well in case of a compromise you can recycle those keys even if you just do that at least you're in a much better State than a lot more people they find a lot more in the report so McKenzie and I went into the details of World secrets what kind of secrets are they seeing more publicly used grid Garden is a very popular open source project that you can participate in the number of stars in the game people would be a good enough proof that hey it's a popular software which is used by a lot of people and to free one as well so definitely check that out it's an open source software which you can use to find out secrets in your repository public and private I think I've been using the open source version a couple of times and I think it's been very valuable that's used for me to identify I had access keys in my GitHub history which I had to delete so uh please don't look into my GitHub history for my GitHub repository but I have made sure that those keys don't work anymore so it was really valuable I hope you find this episode about Mackenzie Jackson and if you know someone who's trying to work on a secret management strategy or thinking about doing Secret Management in the organization definitely share this episode with them and if you're here for the second or third time I really appreciate if you can leave us a review on iTunes or subscribe or follow on Spotify Google podcasts like audio platforms or video platforms like YouTube and Linkedin where by the way we're about to hit 10K or maybe 20K depending on how quickly this kind of spreads we are looking at getting the 10K Mark in YouTube very soon so I would really appreciate if you help us cross across that and if you have any questions feel free to reach out as well because we talk about classically all day every day I hope you enjoyed this episode with MacKenzie Jackson and I will see you next episode Peace report because this is a what's your coffee Edition with McKenzie and for people who don't know who you are what's your name and what are you doing in cyber security man yeah yeah I think a lot of people I'm Mackenzie Jackson so we're here in the Netherlands at cubecon I work for a company called kid Guardian we're a code security platform and yeah we're all about code security my role as a developer and security Advocate really sharing the good word of how to build more secure applications interesting and document applications and secrets together because a lot of our audiences developer First Security First we have a good mix of it as well what are secret actually let me start there first let's level the playing field what are secrets for people who don't think it's just at least I think it's just passwords and they just passwords well I yeah that means a technical term for it is a digital authentication credentials yeah right so dear typically passwords but the passwords that are mostly made to be used by other machines so machine to machine identity now password peers you know for an FTP server or a database are still Secrets yeah the majority of of Secrets though are things like API Keys security certificates and encryption keys and these are all used not by humans but by other machines and how we've ended up with so many Secrets is that we've moved away from an architecture of the monolith yeah like the the giant software written in Cobble that you know like is millions of lines long we're have an architecture of Microsystems a micro Services microservice you know SAS platforms everything's interconnected yeah it makes it much more scalable it makes it faster to develop we can focus on what we're doing we can Palm off authentication to opto and search for Golia yeah but it means we need to authenticate that's why we end up with so many of these secrets ah okay cool to a point then because we moved on from monolithic architecture to microservices and have more access Keys access tokens API Keys AWS access keys multi-cloud so how are we dealing with this right now because clearly it's been going over some time it's like identity access management where I tend to be in my mind with sword but then you found oh wait Cloud identity is different yeah they're the same case here as well where it's not just as simple as our password manager yeah so I mean and this is kind of some of the challenges of it is that you have these systems called Secrets managers yeah which manage Secrets funnily enough and the idea behind it is that you want to have all your secrets centralized Rapids and lots of authentication layer yeah with logs and everything and once you have that component then you know you would think job done yeah but the problem is is that all your developers all your team leads different Engineers your devops need access to these secrets ah the more tightly strapped down they are the more likely it is that I'm going to create secrets.txt on my desktop file right and let's talk about the scenario of a Euro developer yeah and I'd say hey build me this feature I need you to connect to this database yeah and I just yeah so the first thing you do is you just hard code those credentials in because all you want to do is just quickly test it that's pretty much you know get it done yeah and then later on you remove them you use environment variables you do right yeah yeah of course but the problem is that in git your entire history is tracked yep so when you go to code review your passwords your secrets are not there the reviewer doesn't see them but a hundred commits ago when you first started out you've hard-coded the secrets they're in you get history yeah moved on from that Secrets manager now they're backed up in your code backup systems yeah you know they may be copied onto messaging systems they may be new wikis but ultimately you don't know yeah because gear doesn't have intense logs of who's accessed what and where your code sprawled to so now your secret is fooled but you don't understand it you still think they're in your secrets manager but they're running around everywhere causing all kinds of trouble and it's funny you mentioned secret sprawl because that was kind of one of the reasons why I kind of thought of you to talk about because you guys came up with the report as well like yeah I'm gonna talk with the final I'll let you talk about the findings but what are some of the highlights for you considering you've been in the secret space for a long time yeah so the state of secrets with this is the third year we've released it and essentially it's come about because you know in Gig outing we build these different services and systems as a result we find a lot of Secrets yeah so we decided that we would create some Pro boner learning so when we find something yeah if we know who's lead it will message them and then essentially what we do is every year as we put in a report what we've found yeah yeah so the highlight of the report is you know basically that just in public repositories on GitHub so there was you know tens of millions of developers use GitHub yep last year there was over a billion commits so a billion contributions made publicly to GitHub yeah wow we scanned all of them oh wait okay we've scanned all of them publicly we found 10 million Secrets last year just last year 10 million secrets in public repositories on GitHub wow it's because we've been working the good space for a long time like a lot of people have and I would have thought the party called out where hey make sure you don't put your secrets in and but it's already good in good history somewhere yeah is that true that you can't take it out well if not easily the main thing is that you don't know they're there can you rewrite your history yes if you're in a team that but like a team of multiple people rewriting your history is totally a pain because if your history is different to my history it throws an error yeah so to be able to do that is you know and but also one person gets if it goes to your remote repository yeah you don't know where it's ended up what machines it's on you know so even if you got rid of it yeah it's still compromised actually I think the that's the good point as well because the clear delineation over here a lot of people would look at GitHub and go well my current Master brand doesn't have any secrets yeah but what you're referring to is more like well all the comments you've done so far I can still as a public person click on this link yeah and see what you've done in the past all the way to the very first year you started working on it yeah exactly with that and so there's there's lots of ways that Secrets kind of get leaked too so GitHub has a public API anyone can view it you don't need authentication there's a public Ledger of every single Repository yeah and everything that's happening yeah lives yeah very easy to scan one of the most interesting events that you can scan for is what they call the public event this is when a private repository yeah is now made public remember that when you make a repository public you're making all of its history public too oh that's something that often contains Secrets yeah but there's also lots of other scenarios where Secrets get leaked publicly so you say okay you're not silly enough to ever hard code a secret and push it publicly okay fair enough I'll believe you yeah but in this scenario where you're trying to debug something yeah so you create a debug log that debug log might have your environment variables printed of it yeah you get add all like you know just capture everything now that debug log is in your git repository you know there's all these types of accidentally accidental leakages that can happen as a result of just code and it happens you know there's this is a funny story a friend of mine was telling me he's a senior developer he's one of the best developers that I know yeah and like if you ask them should I hard code Secrets he would say no yeah his wife was pregnant he was working from home his wife went into labor he just thought oh crap he had just kind of been issued a key for an Amazon web services right right and then just watch the layer but get add all get pushed out the door he came back from maternity leave two weeks later and realized that he had pushed publicly admin access keys for AWS oh my God now like nothing actually came of that the team caught them and that you know but still this is like a scenario where a very experienced developer yeah still makes it you know and it just happens right you know I've done it and I am like I did it whilst making a video about why you shouldn't do it I accidentally pushed the wrong keys to get up so see I can if there are individuals who are listening to this or maybe be just individually working on it may not be working on I guess like with that as a team it's still easy for them to read out their history they can just delete that but as a team because I'm also thinking as a business you have multiple business units yeah it's not just like and you're all feeding into one repository yeah yeah so that makes it even so what do people do then because I think I I don't know what people would do at that point the only thing that you can do this is the only thing that you can do is a secret gets into your git repository private or public or wherever you've lost track of it yeah secret now needs to be revoked okay right to issue a new secret right there's lots of people get a credential strip back all of its permissions leak it on GitHub see how long it takes for someone to find it it'll be less than a minute left in a minute less than a minute we've done this experiment it's lots of times right oh I think it'll be it'll be funny if anyone who's watching or listening they should definitely apply and put a fake key in it yeah definitely I think because it's not like a lot of Honey token as well yeah so this year we actually created honey tokens on A New Concept right right they've been around for a little bit but this year we created a couple of projects around honey tokens so there's an open source project called GG Canary and that's the way it's like the Justin Bieber's wife GT Community [Laughter] are we gonna get sued by Justin Bieber that will create you know you need an AWS account but it will create tokens with the correct amount of permissions yeah and then what will happen is they will trigger if someone tries to use them and it will give you lots of information so you can use that to leak them on GitHub yeah but why these tokens exist is that one of the first things that a militia sector does in any kind of bridge is that immediately when they gain access to something they want to persist access that's right yeah they want to move into different systems right the easiest way to do that is with Secrets yeah so the first thing that they're going to do is enumerate all the data and just try and find low hanging fruit Secrets yeah like persisters if you leave Secrets intentionally that are honey tokens in lots of different places then your alerting systems will light up like a Christmas tree if someone makes access and you will know exactly where they are and you'll know that you have a breach right and if you're worried about your source code getting leaked somewhere then you can put them in your source code and if they ever get leaked and that silicone is ever made public yeah then you'll also know about it interesting was there like a common set of secrets in the 100 million repository that you kind of scanned yeah was it a common type of secret that was identified which is like hey there's more API tokens versus more awfacturers yeah so we have some interesting results there's a lot of different types of Secrets the number one most common type of secret is database credentials okay like I RDS something like a database actual like oh wow like username password for digital in the GitHub wow that's about 25 of what we found 20 of what we found was for cloud infrastructure so there's over a million that's two million actually yeah and we validate these all right also it's not just like you find always so a lot of people kind of say like okay but like how do you really know it's a secret because secrets are high entropy strings they're random strings yeah right URLs unique identifiers all high on sugar strings yeah so we validate secrets we have lots of post validation we check what they actually do before martinumento as a positive so two million cloud provider keys that were valid were leaked publicly on github.com that's like huge because even if they're just for you personally yeah I can still rack up a huge bill for you yeah yeah and then there's lots of other interests in the full report we'll show you we find them for messaging systems attackers love these because you can do internal fishing campaigns yeah post it from within your own slack yeah you know that's and about one percent of the or nearly two percent of the secrets that we found were Hub access tokens so your access token for your private repositories and you've just made public wow oh that's why you're talking the public event that you're talking about yeah yeah everything you've done so far has suddenly become public yeah yeah wow and is there like some sense of responsibility that GitHub or AWS or all these people should be taking over this as well because you almost feel like and I don't want to say that hey it's all the responsibility because I guess you can't know every scenario in the world you're just providing a service yeah yeah so like are they doing something about it actually yes and and so there's a concept of the shared responsibility model yeah and there are a few organizations that do this particularly in the public space now public is interesting to talk about because we throw out big numbers and yeah wow 10 million the real threat is private but just focusing on public if you leak an AWS credential yeah AWS are actively looking for them yeah and they will quarantine them if they find them right which means that they will reduce their permissions someone else can't run up a huge bill or something like that that's not without its risks because you leak a real AWS key and it gets quarantined all of a sudden that key doesn't work you could have just broken production that's right yeah you know so these things come with risks yeah that as well now there are some other providers that also look for their own Secrets but it's a lot to scan GitHub there's a lot of information it's a fire hose and if you're looking for like you know AWS are just looking for one type of key yeah they know but we look for you know nearly 400 different types of credentials on their places we're scanning code 400 times with different detectors wait so because you know how we've kind of started the voluntary developer first and document developers as well one of the challenges and we had a couple of conversations through kubecon and odyssey as well where the conversation always revolves around the fact that you can make a security door but to have an adoption at the developer side is always challenging and secrets is probably clearly as you called out it's been a 100 million or even a million Secrets out there yeah clearly someone somewhere when they find out about it they were like alarming is going off but if they do use open source software the high dokers or whatever do you feel people are ready to even respond to those kind of things oh yeah this is a really interesting status to answer this we have to kind of take a look away from public infrastructure and go into internal infrastructure yeah so at the guardian like the products that we sell we scan internal infrastructure for Secrets right right let's move away from that quick but we also have stats on what we find in these companies when we scan the secrets right so typically when we first scan a company on an average size company they say they have 400 developers it's fairly small but up there right yeah yeah we'll find about 13 000 secrets in their repositories 1 000 of those will be complete completely unique yeah but if you're a security engineer you don't know you just see secrets so you've seen 13 000 Secrets 400 developers you'll probably have four appsec Engineers one per 100 yes so you do the maths that's over this about three and a half thousand secrets that each appsec engineer has to deal with every year so being aware of the problem yeah is like is almost worse than just living in Bliss of not because now you have to investigate each one find out what the secrets does you need to rotate it you need to redeploy it and you need to do all of that three and a half thousand times without causing a server to outage that's right yeah and without without bidding off people the harder thing is kind of remediating that yeah and and you know that's Secrets detection is important because you need to know but you need to know to be able to remediate it as well but there are things going back to individual developers yeah what can developers do is that this is from the security point of view but developers can stop the bleeding yeah yeah and there's simple things that they can do like creating a GitHub like a pre-commit GitHub which checks your commits before your remote repository accepts them for Secrets right right then if you've got something in a debug log that you didn't know about then you're going to be aware of it and you can change it and you don't need to rewrite your history because it hasn't entered the history yet you know you can be proactive and then that stops the bleeding yeah so there are things like from the individual developer shift left mentality there's lots of things that they can do yeah but you need also need it from the top you need visibility over where these stickers are yeah and I think the reason I ask that is also because a lot of the conversation we have around cloud and Cloud security a lot of people are not even ready for instance once like this yeah that's why it's not spoken yeah probably if going mad that child as well a number one reason could also be of your incident is a secret was found on the internet somewhere someone's using it and trying to figure out who owns this yeah and where do I take this and what do I do with it and all of that as well so it's a lot more complexity around just to what you called out you know where was the problem now you're like oh I should probably just quit my job and walk away exactly yeah the problem is too big are there any interesting breaches that you would have heard and with all that I'm not trying to have like a fair uncertainty doubt kind of a mindset but most of our understanding that hey this has happened it's not just a well we don't really have anything important yeah so this is like so many to look at there's one interesting one that happened last year and it's interesting because it illustrates a few things it happened from with Toyota okay car manufacturers yeah they have a product called tconnect mobile application location connects to your car yeah important yeah Toyota had a Consultants working on this as well contractors working on the code it wasn't Toyota it was actually a contractor I accidentally made code public inside that code was the access credentials database credentials to all of the users using tconnect oh all of their information now that was public for many years before a security researcher found it and let them know but who knows who found that secret beforehand but in terms of impact is much worse than that that's an interesting one because Toyota didn't really do anything wrong yeah yeah you know it was a contractor that was connected supply chain issues right but if you know like another recent one Uber they had a worst case scenario where credentials were sold on the dark web to that were for an employee for the VPN service yeah they had MFA MFA was bypassed just by spamming it so the employers whose credentials were lost accepted the MFA request hackers made it onto the network the first thing they did like we talked about they tried to find more secrets yeah there was a Powershell script this Powershell script was used for when a new employee started so you started I run this Powershell script and it created accounts for you on everything uh inside that Powershell script was the worst credentials that you could ever leak it was for the Pam system the privileged access manager system yep basically the keys to the kingdom so what do the attackers do the attackers created accounts for themselves and every single one of uber Services worst case scenario you know and this is a secret there's an internal Network you know so this is I think we're talking about public a lot but internally as well and just like very quickly there's one's like code COV it's a supply chain tool they leaked a secret secret was discovered yep application turned malicious attackers were then able to access the private source code of codecov's customers so like all of this like secrets are just kind of everywhere and like they're always used in attacks because maybe not as initial access but definitely to elevate privileges to move laterally to persist the attack and if you don't have good alerting if you're correctly authenticated how do you even know that an attack is there right they're using the systems in a way that they've designed to it so alarm Bells aren't even going up fair enough considering to pull down the why I've already raised it up is it a funny story to the whole secret thing as well like let's just raise the bottom back over here and do me happy again oh yeah 64 people you're depressed like wow like that's a lot of Secrets because they're like feeling sort of breath by nothing is there a funny story as well yes it's a funny story but I don't know they write I think they might bring down the vibe again one of the funniest things that I I've found in doing this state of Secrets for report is that every year we find secrets in the weirdest of places yeah and so we found 500 commits where their commit message was the private GitHub access token so this is essentially like I'm sending an email and the subject line of my email is my password so obviously some people have been confused of how git works authenticates and the message they think is maybe a header and they're passing a credential through the commit message but we find Secrets areas uh that's a story that I thought that did not bring it up like every security person instead of going wait so it gets worse yeah it's everywhere but now I have to see the car for people sending emails as well yeah yeah because I think that's a good point as well because I think was it Verizon the report that they did for data breach they had I think number one or number two reason was accidental information like so I'm typing in Mackenzie Jackson but I know another McKenzie and Earhart does like try as an autofill yeah and people just like well oh well I guess it's Mackenzie but it's not Mackenzie Jackson it's like MacKenzie I don't know something else yeah yeah and Johnson or something I guess I'll just go with that and without even realizing you send like sensitive information to another person who just happens to be one McKenzie that you knew from ages add Gmail address and like oh next thing you know they're public and I think that to me was quite surprising where even no matter how hard a lot of people have tried secret spoil is gonna happen so I guess where I'm going with this is like where do people start the first thing the important thing to know is identify where your secrets are right so or should be it you should be right you know like so you've got your secrets manager so get a good secret manager and one that you're is actually going to use there's lots of great ones out there but some of them are too heavy for what you need you've got 10 developers you don't need hashicorp Vault yeah you can have it it's great but maybe not and then identify where they've leaked right look in your systems understand where your code's leaking because that's going to give you an idea of how they're leaking yeah what repositories what teams maybe need more training and more understanding about Secrets yeah and then once you have visibility over where your secrets are start putting in some gates to block it get hooks requests on your CI CD pipelines make sure that you've found this pool of blood yeah to stop the bleeding yeah whilst we're working on that as well and so like that's really the best way of trying to combat this situation you mentioned gitoke a couple of times yeah with securely people trying to do this listening on the other side or watching on the other side how hard is implementing GitHub without say if you were a developer do I have to have access to your laptop to do with that yeah that's a lot of this question get hooks are fantastic they're probably the most effective way of stopping something early but they're problematic they're part of the solution you also need to have scanning on the infrastructure the reason why is that githubs are configured locally so yes you need to set them up on the develop this machine there are ways that you can kind of automate this but they need some kind of you know trigger the other thing is that hooks can be bypassed no uh you put a dash in at the end of your commit and the hook will still run so they're not foolproof right right what is foolproof is scanning in real time your infrastructure yeah knowing when sequence leaking you get repositories know where they end up so that's kind of like that's the most important part scan this visibility and knowing where your threats are and then there's things that you can do to prevent the bleedings but you can't rely on them that's right yeah yeah and maybe education could be one way but it's still education great yeah yeah I think it definitely doesn't solve the problem but there are tools I guess what I'm going with is there are tools are available which could be native to a developer platform yeah with security teams can use to enable themselves from finding secrets it and doing all of that as well yeah okay absolutely there's lots of tools that you can use so there's open source sequence detections they give great coverage similar problems when you get too big you're gonna have a lot of noise yeah so the problem comes reducing that noise but you know what like if that's where you start that's where you start like start with an open source solution it doesn't require you to talk to the accounting department like you can get started right yeah and so this is the main thing and there's maintain weird coming from someone that works from a vendor but at the end of the day like just to start somewhere when you get big enough you're going to have issues with remediation and stuff and then look what's available on the market but start at open source yeah keep it simple because at least then you're going to be more aware of the problem I think that's the key takeaway my second last question because I find that a lot of conversation security are harder to sell to people to believe in the idea that hey you give a couple of examples for how breaches are actually yeah been caused by Secrets but the bigger challenge over here also becomes that making people realize it's important to get work secrets and it's not just here I'm going to password manager and data have you found some insights over there as well where what can people use as a tool or mechanism to highlight to say broader node security audience as well that hey we should care about secrets and without using the whole I guess which is important and to know that there's something happened yeah yeah but have you found any other way of identifying it or apart from just finding hey there's a secret you probably should look at that like in the control somewhere I guess absolutely if you're struggling to convince management that you need to invest in like some security tools then like start open source and prove the problem but there's other areas too like vendors will often give you a report of the problem so one thing that like Guardian can do is that we scan all public information yeah we can let you know how many Secrets have been leaked from your employees ah you know like things like that so like show you that this is actually not a hypothetical issue and if we can't find any great you don't need the tool that's fine you know like yeah you're good yeah yeah I mean I shouldn't say that but yeah that really illustrates I said the problem and I think once anyone understands how big of the problem is in internal infrastructure you know because Secret's just late yeah they just do yeah right and sequence managers are important and they help solve the problem but the secrets are going to lead yeah and so once you start scanning your infrastructure and just simply you know get repositories are a good place to start you're going to find that you have a problem yeah it's a good place to start you know and last year we saw so many companies have their private source code leak like Microsoft and video Samsung it's not a few yeah they all had secrets from in the source code big companies like that with great security posture like I'm not picking on them and they have secrets in their source code everyone will yeah right so where can people learn about it because I think that's kind of like my question because I also feel like there's clearly an education piece missing in the fact that secrets are not just username password yeah they're a lot more than that and people will have to go out on the Journey of identifying hey what kind of Secrets do I have yeah which are more than my username password is there like resources for these that you normally Point people to yeah for sure I mean we try and create a lot of resources like the state of Secrets for all report it's great we have a Blog but has you know endless information is about detecting secrets and areas like that but in terms of like General resources I would say like the OS top 10 they talk endlessly about Secrets there's the API and secrets cheat sheet from OAS okay you know like if you look at like how do you actually manage and handle these secrets that's a great resource so there's areas like that where there's lots going on there's a really cool project called wrong Secrets or wasp again and it's an application that where gives you challenges to try and find secrets in the application I guess progressively harder but it lets you know actually like oh my God like I can have a secret in my running application and I didn't even know yeah how can you you find that so there's so some great Educational Tools if you want to learn out more from the get Guardian website but on oasp and also on lots of other resources as well awesome I know well that was all the questions I had where can people find you on the socials to know more about secrets and maybe lower them yes I'm everywhere on social media at Advocate Mac so on LinkedIn on Twitter Advocate yeah awesome I'll put that link in the shows as well thank you so much for watching everyone yeah thank you Mackenzie and we'll see you next episode Peace when you're developing an app security might be treated as an afterthought with functionality requirements in Thai deadlines it's easy to accidentally write vulnerable code or use a vulnerable dependency but sneak can help you secure your coding real time so you don't need to slow down to build securely develop fast stay secure good developer seek foreign