this is the place that uh made nerd cool the most popular password in the united states is password one two three those are some of my previous fans who on earth would actually fall for that i'm sensitive information has been given to the wrong hands the security and we are live so this is the pilot episode for the security repo podcast and i have the great pleasure to introduce mackenzie jackson who's the chief evangelist for get guardian thanks mckenzie for joining me yeah i'm good i'm good david how are you going today good good so we we just mentioned that we have um the in it uh ability to uh to go on tangents so i'm assuming that's probably what's going to happen here um but before before we dive in um i'd love to to hear uh you have you have such a diverse background you just mentioned to me that uh prior to hitting recorded you are um an architect um or you you were you know that's your initial you know uh profession you started so why don't we kind of start with that and um you know talk to me about your kind of your backgrounds just to kind of set the stage uh in terms of what where you are from and and uh just start talking about things for sure yeah so yeah my name is mckenzie uh as you said you know i started off uh kind of in computers and i.t as a teenager uh breaking into things always curious about that um and my parents and everyone in my family is artists of some some sort musicians or pottery or something and i was kind of this i was the square of the family i wanted to get into i.t and uh i kind of made a compromise with myself so i was like if i if i do architecture building architecture i still get to work with computers but it's kind of artistic so i can i can follow on on that and so that's uh you know that's kind of how it started and i was building um my first little business was building websites for musicians because uh there was so many in the kind of in the circle that was my niche kind of prior to the wordpress days or the the the more well-known drag and drop solutions you know but when i graduated in architecture and when i was working in the field i was much more interested in automating my job with scripts the bim systems that we use the building information management systems they have they're quite they're powerful programs that allow you to kind of build in uh some scripts in specific languages and uh you know and this is what i i love to do and part of my job was trying to figure out what we used to call the highest and best use of land you know if we're building an apartment building what is the maximum amount that we can get with regulations and setbacks and council kind of requirements and so i built a little script to do that i didn't tell anyone about it because i thought it was cheating i didn't realize at that time that that was that was going to be valuable so so i used to kind of hide this this work and then uh eventually i just decided do you know what i'd like i that the part that i love about this job is programming the part that i love is is kind of doing these scripts and then uh after after a while a friend of mine came up to me and he goes let's let's let's do a startup and let's let's get going like let's move in a direction and he had got some venture capitalist funds i became the cto and we we started this company called campago which still exists today um so quite proud of that still still a thriving uh thriving uh company there and it was in in healthtech and was this healthstick area and then that's when i first opened up my eyes into uh kind of a lot of the security and compliance and areas because we're working with sensitive data you know we had to you know we had to learn how to make our application secure we had to learn how to kind of use these robust operating areas and infrastructure and testing and again this is kind of where i found my love in this area because you know when you learn about how to protect something you learn about how to how to pull it apart and i was just shocked i just gobsmacked at at you know just how vulnerable uh everything can be and when we're looking at dependencies how the whole internet you know log4j was a good example like everything's built on on these these modules that that if they fall over the whole show falls over and so you know that that by the way that's i can't see it's just it's almost like architecture right yes yeah it's building blocks you know if you are not building a foundation we make it you know it's amazing that you know small errors in in architecture can be disastrous you gotta have to have you know that foundation and it's very similar to what you just mentioned specifically around repositories and i don't think that people are aware and we'll dive into that like in terms of the the entire building blocks of of application and internet as whole is built upon uh this massive repositories of of code that was written by by tens if not millions of thousands of people that um you know our code every day and adding to the system so those are the kind of the foundation building blocks of of the internet yeah in in applications yeah absolutely and and they're and they're so important and open sources is such an important part you know and it's so unique in the development community to have this collaborative platform where so much of what we do is shared um you know and it comes with security risks but overall i think it's a good thing and we and you know and what we what i'm passionate about now and what i get to do as a kind of security evangelist and advocate um is to to really kind of be able to talk about all right what where where are the problems how do we secure them and and where do we go to make sure that we can maintain this fantastic creature that we built um but it doesn't bite us when we're trying to when we're trying to gallop through the field and it's amazing you still you know listen to you still have the entrepreneurship spirit in terms of what you do even though you're part of get guardian but the way you go about it in terms of advocating and in spreading the word is very much entrepreneurial you know like you you you travel a lot you participate in in different you know different events even doing this right now and just you know getting out of the comfort zone um and trying new things and you just mentioned the media you're about to release your own vlog you know from like cool places so it's it's super cool that you have you maintain you know the kind of entrepreneurial spirit by even though you're you know a very different role it just shows it just shows you that it doesn't matter you know where you put somebody if you know there's certain pillars of that person that you know what they want to do they it doesn't matter where you put them they're still going to maintain that uh which is remarkable well thanks thanks david i can always trust you to say a nice word about me it's true you can wire me some money [Laughter] um but uh but so tell me uh let's let's talk about for let's unravel kind of the the kind of the topic at hand um and we talked a bit about kind of the the building blocks of applications so um and you were exposed to firsthand you know you start building scripts and then you you create your own company and you start um you know looking at what's out there in terms of and i'm assuming there's some of that code that you're using as entrepreneur you're using somebody else's code so why don't we talk about um you know this this whole notion of reusing code and what's that mean and in uh maybe just go from there like what what is kind of the underlying uh repository what does that mean for somebody who's just coming into this industry or you're you're a security uh practitioner that wants to know more about this area yeah yeah that's a great question we can dive into that for sure you know i think to start with just kind of a statistic that people like to throw around and you know and it changes um you know it changes a little bit but you know a lot of people say 90 or 95 of your application isn't your own code it's your open source libraries it's the dependencies that you're using it may even be the sas platforms that you're you know you've built it on yeah and so there's there's lots of these different components from a security perspective this is fascinating in in so many levels so for one you're you're dependent on these frameworks these open source libraries um that them themselves could make your application vulnerable or they could be turned malicious so so there's is you know that component of it but we absolutely we're not going to move away from that right and and and i want to be clear and what i say and when i talk about this is that you know i'm not advocating against um you know open source or or anything else um but it also means that you know attackers probably know your code better than you do you know when you look at these massive vulnerabilities like log4j you know people understood that element that was in your application better than you did and that's why they were able to to explore it so there's there's that area when it comes to uh with these with these kind of building blocks and open source source areas now i'll mention to that uh because that the reason why why this this whole underlying repository even exists is you know think about it when you write software right so you always say okay i need a certain function when i say function um i need uh my software to you know maybe like receive input from from the mouse or receive an input from the screen so what i do is i i go out there and i look for a function or a library that does that so i don't need to go out and start writing you know hundreds of lines of codes right is that correct so that's basically kind of the underlying reasoning for even for all these repository to exist is that you know why it's like stories some of the stories already been told why why retail this story you know why not you know write reinvent it so you basically create a you see what's out there and then you reuse that is that is that a correct yeah for sure i mean a great a great example is credit card processing right who in their right mind is going to create an application and build credit card processing into that like if you're building an application let's say you're building a social network right what's going to make your social network great your fantastic credit card processing or the fact that you have a specific algorithm or you have great uh search feature or the way whatever it is that makes your social network better than everyone else's right it's not credit card processing and everything that it's not you can find a library for and because it's and that or or a product like so if we're taking paypal stripe credit card processing these are these are products these are sas kind of components that you can add in um they're kind of different to libraries but then you know if you want to to go down that path uh let's say you want authentication so there is you could write all your own authentication scripts to log someone in or you can take an open source project that that does that too or if you want to create a very specific search function well you can just take an open source library and you know you can modify that you can build upon it if it doesn't meet your needs so that's why and there's a saying in security turtles all the way down it comes from this kind of silly saying that the world's held up on the back of a turtle standing on another turtle saying another turtle and it's turtles all the way down and that's exactly what kind of software dependencies are because you know all right so the the search function is that's your application's dependent on is dependent on these other libraries you know which are dependent on these other modules which are dependent on you know and all the way down to the the lowest level that we you know that you can go down to these modules these open source libraries are built upon this this is why they log4j was such a had such a tremendous impact because people even after all this time they've gone by uh you know they're still you know the there's unknown effects in terms of how many actual applications been written in it has log4j in some library that has been used to buy that application right yeah and i'll give you another example because everyone's sick of hearing about log4j by now i'm sure but you know uh event stream um was uh was an open source library that was used hugely across the internet and it was turned malicious and how it was turned malicious it was dependent on on a library i think it's called i think the like the flat map you're kind of pushing my memory a bit here but um inside event stream which was a dependency in lots of websites and lots of applications there was flat map and uh flat map was kind of a poorly maintained project you know one of these dependencies that hadn't really been updated regularly didn't have a community around it so someone created it it was probably fantastic when they created it and it's embedded into these different projects so what a malicious actor said is that he decided he'd take on the role of maintaining this and made lots of kind of improvements to it built the trust of kind of the owners there then once he'd done that turned it malicious which in turn turned event stream malicious you know so this is kind of an error and you know you could put event stream into your system and you've done your your research event streams well maintained you know but then a dependency of that is compromised a second stage dependency and then your application is vulnerable now too so there's so much things to consider when it comes from that and we haven't even talked about you know like the other sas platforms and other areas that use secrets and how how much they're sprawling through the internet so i mean there's so much to talk about here you know but again you know the reason why we can progress so fast the reason why we can build such great applications the reason why we can speak digitally right now is because we can build stuff on what other one of what everyone else has has been doing and that security community is strong to try and make things secure too and it's an incredible story the fact that somebody actually went ahead and made the investment in maintaining and actually improving a certain library for being used in in future software um projects just for the for the purpose of eventually hacking them and and you talked a bit about numbers and the numbers of the growth of the these uh code repositories is exponential right you have some some some statistics about that like in terms of what is the growth rate of of software repositories oh i do off the top of my head i'm you know like i'm not exactly sure like how much how much the they're growing but you know we've got you github has some great numbers you know something like 75 million developers on on github i do know that 54 million people pushed code to github last year 50 more 54 million unique users this is huge and github is the main place where this collaborate collaboration kind of goes on and and and people come into these these communities so the amount of repositories that that that are kind of being maintained the the active users that are out there the innovation that's happening is is is mind-boggling um you know and and and i think when you explain this into other industries when you talk about the open source community that the the it industry has you know like it doesn't translate it doesn't translate into architecture you know i imagine build you know imagine doing a great detail for for for something like a downpipe drainage system right you spin it you know these are something that you'd spend a lot of time on at your work it's not core to your project no one's going to walk past your building and go damn i'm going to buy an apartment there because that down pipe drainage detail that's fantastic you know but we're still not going to open source that so then every time even you know and in the way that it kind of all works that's not even owned by the firm that's usually owned by the client so we can't even use that same drainage detail in another project you know it's so counter-intuitive it's just the opposite of what we do um you know in in the it industry and this exponential growth that we're facing i think it's what also attracts a lot of people people in here yeah and those numbers are you know immense i mean there are 54 million i mean that's that's quite a bit of code it's almost like it makes you wonder why why not all the code has been written already like what's the you know where is there more to write you know like because it's almost infinite numbers of libraries are we like we're like the universe ever expanding and we never stop i mean it's it's you know is it ever going to slow down or is it like just something they always continue because you think about what else can they write well you know like 54 million people like push you know this is but this is the crazy thing about innovation right because you know someone will come up with because it happens in so many different areas someone will will come up with a better way to do something and that could be even a better uh a better framework to to write a programming language upon something something quite low level you know that you know i think there's a better way to do javascript so we're going to create you know that react was created so then all of a sudden reacts created now all these libraries have to come on top of this and we also expand in our possibilities you know in security or we always have to to maintain and these new projects are created because their computing power is getting getting faster and faster you know these algorithms and cryptography uh algorithms that were secure uh you know 15 years ago were considered secure 15 years ago you know now cannot be and as we get into quantum computing we have to kind of figure out we have to reinvent not reinvent we have to improve everything we do uh constantly and um there's always there's always better ways to to you know to do that and i probably people had the same conversation you know 10 years ago i don't know if you've done this but if you go on to like way back machine and you go into youtube 10 years ago you're like crikey i thought this was a fantastic website 10 years ago you look at it now and it's kind of like unusable and you know that you remember how long because i do that with some doomsday prep you know because i i go back to the 2000s you know and and and you fast forward to you know 2022 and basically talking about the same exact thing where like you know the end of the world is nine oh yeah and uh you know but you're right it's uh there's always the it's not just the the advancement in in computing power i mean you're looking at almost endless iot devices that are out there they're constantly being manufacturers and and created oh exactly that is good for that uh there's you know 5g technologies that we're not available you know to um to be used on mobile devices there's new mobile devices almost daily they're being you know come out and with new capabilities new sensors on it so there's always going to be a need for that now um when you say that guardian scans uh so let's talk about a bit about what the company does you know you mentioned that there this you know you scan the repository and do it automatically what do you what are you scanning for what's the what's the underlying scanning yeah so i mean we talked a lot about kind of open source dependencies and libraries and stuff but there's a whole nother component of these modules these building blocks and that's kind of sas platforms that's tools um you know so when you look at all all of these systems if you have a sophisticated application you might have a thousand different services you're connected to your database which may be managed you've got your cloud infrastructure you know you're you're connected into uh all these different social apps you're connected into credit card processing authentication you know all of these things right because you want to be able to you want to you'd want to focus on what you can do when you want to let everyone else focus on what they're doing and let me double click that that's insane that's right so people don't realize so when you are going to an application because sas application right so people most most likely think okay i'm just going to this website i'm signing up using you know whatever services you're telling me that potentially has hundreds if not thousands of underlying connections to other oh you know absolutely absolutely every everything that would you know like from what you do is from the moment that you first get there you go to your sign on page you know i mean look if you're mad you can build everything at yourself and um you know and i'm sure that there are certainly probably programs in government that that do use very little of these services maybe um well i guess not we saw that with solarwinds but you know everything from from when you've logged into to your application where you're pulling data from where you're pushing where you're storing information credit card processing we've talked about a lot you know how you're hosting your application and then also how you're managing traffic and diverting uh you know diverting traffic into the right areas making sure you know all of this stuff is incredibly complicated and there's and and there's companies that kind of monitor this and then in addition to that you know there's all the tools that you need to build your application you know you'll get your version control systems your git repositories your testing tools and then even if you have the application done right think of some sales and marketing teams they want dashboards where they can view things they want hubspot integrations or salesforce integrations and they want you know like but you need and you need all this stuff you absolutely need all of this stuff um when you're building an application but all of these modules are connected somehow right all of these modules are connected to your application and how does your application let them know who it is and who who they are and it does it through secrets it does it through basically passwords long passwords that are generally what would consider high entropy strings that's a big string that's very random you know you'd and they're they're they're they're everywhere if you go to a youtube video look at the url you'll see a whole bunch of random letters at the end of it that's a high entry string that's kind of what these passwords words look like and these are incredibly sensitive because as secure as your application if you have the key to to get into a component that your application is using then the attacker the adversary can pro correctly authenticate themselves so let me let's kind of break this down if you're an attacker um and you know a company has uh has leaked their aws cloud service key right so now that you can go into their cloud service and you know depending on the security sophistication you are correctly authenticated which means that no red flags are kind of going off at this point so you can shut down services maybe you can access amazon s3 buckets depending on the privileges you know you could do something minor like using computer resources to to to mine data or you could go deep into site inside this infrastructure move laterally and shut down services you can squat in there for a long time and you know so you could just do all of these things and this is just one of those components that you know that we talked about um and so what we're kind of seeing and this relates into our conversation about all these dependencies and libraries because you know when we look at look at these these secrets they're meant to be uh used programmatically they're meant to be used by our application which means that they often end up in source code and it's and it's a complicated topic to talk about how we manage and store these secrets right the tools that we use how we share them but developers need them to test their application to build it they're often an open source code and they often end up in public so get guardian we scan for these secrets as you mentioned we actually we scan it in lots of different areas we scan privately so that a company will know you know where their secrets are because if a secret gets into your source code you've got no visibility over it it's going to be cloned onto all your developers machines it's going to be backed up into different areas you know this may be a highly sensitive secret that very few people in the company have access to but if it's inside your version control system then the intern that started yesterday probably has access to it if he knows what to look for so we we scan it there and then we also scan publicly so these keys can end up on public github they can end up inside these open source repositories they can end up in in public docker images so we basically hunt these down find them and then where we can alert people to them and so just listening to you i think one of one of the biggest issues is the way these secrets are being used you mentioned in the key here is programmatic basically um saying it's not when people think about secrets they mainly i think they think about secrets that they use in the sense that maybe the password username and password which typically is you know on average is not more than eight to ten characters long you know combination thereof but here we're talking about these long strings of combination but it does not matter because it's a machine to machine or application to application that is being used using it and and this is where the kind of the secret sprawl happening because as soon as you start using one application is being automatically being used somewhere else without without a human intervention or oversight right so it's almost like you have to have some system in place to review these yeah i mean look you absolutely do and and when we talk about like okay how are we going to manage these secrets so we've talked about that you know you have thousands of these building blocks at you know hundreds of thousands hundreds or thousands of these different building blocks so you've got hundreds of thousands hundreds or thousands i don't wanna say hundreds of thousands but you know you you have a huge amounts of these um secrets that you need to manage right so where do you want to manage them well you want to manage them in a dedicated system um there's a bunch out there hashicorp vault is one um you know a keyless is is is another there's various different levels of them they're quite complex and set up and they're tightly encrypted they're wrapped in lots of authentication layers there's logs of what happening so storing the secrets you know shouldn't and and having them somewhere isn't isn't uh if we can do that well then that's not that's not where the problem lies the problem lies is that okay so they're in this tight vault right but they need to be distributed still developers still need them okay so you give permission to the developer to use them and you know this developer is quickly trying to get something to happen so this developer has to handle it in a specific way it should be what we call an environment variable it shouldn't be hard coded into source code but it can be hard coded in the source code and it's so much faster to do it so what you what we typically kind of see is a developer will grab this key he'll quickly get his code working his feature he'll hard code it in to do that and then later on he'll remove it and um when it goes to review the key is correctly stored in the environment variables or wherever it's meant to be and and off we go and there's no problem that's visible but everything that we do is tracked everything we do in source code is tracked so that key that was removed is actually still deep in there it's just buried it's just buried in a bunch of code that's been put on top of it so it's hard to find it unless you're a malicious actor and you're specifically looking for it in which case there's lots of tools to be able to to uncover that so that's where the where the problem lies and handling it and there's also a bunch of other ways that these secrets get exposed maybe your application's not working properly so you do a debug log and in that debug log it's probably going to dump out you know a lot of the environment variables that it's using if you accidentally mismanage that debug log that's now incredibly sensitive so now that's another way secrets can get be exposed um they can be exposed in your running application if your application's not handling it well right you may have done everything right but your application is inadvertently uh you know exposing it there's one i saw yesterday where i looked into the source code and i saw that it was taking user data it was encrypting it and then storing it encrypted into the database but in the same line of code that it was encrypting it it had the encryption key so it was basically pointless right so the the key that you used to encrypt and decrypt information was in the lineup so what's the point of encrypting it because you know this is just a sloppy yeah and developer is that or somebody who you know thought they were doing the right thing by encrypting it but they were sloppy about about how they they were about yeah yeah exactly and and also not understanding you know how easily it is to find to find these types of keys um yeah and and maybe not expecting people to kind of go to that effort it's almost never malicious and mistakes happen in um in development the problem is that they can be critical so you know i'm i'm always against kind of shaming people you know mistakes mistakes happen and even if they're even if they're they're critical mistakes then you know we need to we need to learn from them but unless you know and i mean it depends on the situation of course maybe there's some unforgivable things you know but at the end of the day hard coding is secret it's to stated the world it happens everywhere this is huge amount we found six million secrets on github.com last year six million people did this type of error accidentally hard coding something in accidentally pushing something i had a i had a great story from a from a guy that used to work at algolia and i was chatting to him yesterday and he said that he pushed a secret because he was working on something and his wife went into labor he was having a kid and then so he kind of was flustered and then quickly pushed something into into the uh to the get public get repository came back from maternity leave two weeks later to find out that he had set off this huge kind of alarms and everything was going on by by pushing into this secret you know so he wasn't fired nor should he have been and but yeah this is it happens it happens to the best of us when you're handling with sensitive information and you're human it's gonna you know it's gonna expose but we do have checks and balances out there we do have tools to be able to help prevent this identify it like with everything like with open source vulnerabilities but we need to be aware of the risks we need to be aware of the problems and so yeah and that's why we're chatting chatting today yeah and it's amazing it sounds like the you know the attack surface if you call like in a cyber security term it's almost infinite which which i think is is something that um it's most companies are not aware because we we in cyber security you know the pillars are typically are the ones that are you know like the network security the endpoint security you know the uh you know the access management and so on but it seems like they you know this is one of the uh not i would not say overlooked you know areas of of security um but i think it's kind of overlooked specifically around practitioners because you know the end result is that you know these these software developers are the ones who have you know the keys to the kingdom pun intended uh when it comes to um comes to the secrets and they're not necessarily you know cyber security professionals even though they know that they should be doing um you know certain things proper way as you mentioned they're all human um and and you know software development is hard you know it's it's a complex process it's a lot of people you know uh you know pulling levers and making you know making doughnuts you know i'm sure there's some donuts as well uh but you know this this is some very you know high complexity associated with it so you're bound to have you know some some you know some mistakes so you know so you have do you have others i mean that was a cool story and then i'm i'm sure you know it's it's a very it's one of many do you have any other stories because people remember that i'll remember that for sure this story but somebody going out to so what happened when he came back it was you know was the building was burning or you know i think the fire had been put out i think the fire had been put out but it was definitely like oh my god like what is happening and um and there was a number like the meme the the dog me yeah you know when he sits around when it's like a burning building and this is like okay that's everything it's okay and and there was actually other mistakes that were like kind of made into that that compiled it you know the the key that he was given had higher permissions that it really needed so you know like lots of lots of other other areas areas too isn't it isn't it amazing i can't say that there's always you know when something happens it's always like they you know like you know in the army in the air force for example when they when a plane crashes or this major accident it's you know they kill like a almost like they an accident chain yeah where you know it's it's a it's a whole series of events okay you know you know that all combined created that you know catastrophic uh event and you could have stopped it you know at la the 22nd you know step and but nobody did and this is the end result i think that's also the case here that you know there's always some you know um a massive amount of conditions that happened that kind of created that you know catastrophic event yeah uh and and i think what github uh well get guardian it does is that it allows for to put a wrench in that specific uh you know kill chain or process that will eventually will turn into the catastrophic event yeah yeah i mean like absolutely right and you know um and the problem too is that uh it can be so easy to do it you know i'm sure you would have heard that twitch's source code or got leaked at the end of last year you know so we scanned that there was we found about 6 000 credentials in that source code a huge amount but then you look at kind of like how okay how did that happen how did twitch's source code all get leaked and really what happened someone made a mistake someone made a mistake in a server configuration file so probably a yaml file that basically allowed remote access to private git repositories this mistake didn't happen you know like it wasn't it wasn't there for years or months it was there for a short amount of time someone found it and and leaked that uh you know that information but they you know just took so someone wasn't a good citizen i guess yeah when they found it well i you know i disagree i reckon leaking it leaking it's the second best thing you could have done the best thing you could have done was just let them know and tell them that they have a problem the second best thing you can do is leak it publicly because then at least twitch immediately knows that they have an issue um but like you know if you've got much more malicious intention i think you could have done much much much more damage with that because we found 194 aws keys in there now provided we didn't we we have the ability to check if key is valid but in the in the case of an actual breach we never do because we don't they you basically we don't want to kind of confuse any forensics going on because we have to make an api call but but so maybe maybe not all 194 were valid but definitely a few of them were you know so if you had real malicious intentions even if you didn't want to exploit them yourself you can sell you can sell keys um one of the the favorite exploits to do is to sell slack uh web hooks um because uh they're everywhere they're not seemed as critical because it's just a it's just a key to post messages on your your on your slack what what really bad what what terrible thing can you do with that can you shut down the company no can you close stuff off but here's what you can do you know how certain are you that your employees will not respond to a phishing email that i sent right now what if that email or that message comes from the internal slack system saying hey you need to update your password on slack click here or yeah i'll go buy me some gifts yeah yeah yeah you know it comes if it comes internally from a trusted source well man your hit rate's going to go up exponentially so in the dark word slap web tokens for companies are regularly sold and it's kind of like a favorite known exploit because uh you're you're breaking down one barrier of trust in an efficient campaign that that may prevent you and then from there you know who knows where you go once you once you're able to pawn someone um yeah and how do you get how do you get those um those credentials again just a wall to double click on so slack well you can you can find them in and you can find them in a number of number of different ways if you look at the groups like uh lapses and uh areas on there they they've been kind of breaching a lot of people at the start of this year i think it turned out a bunch of teenagers they were paying insiders at companies for credentials and for access so uh you know that's one one way you're an intern you've had a bad day you just lost your job and you have network access or get access you know a couple of thousand dollars might seem uh and then from there you can discover them slack webbed hooks and you see it is almost like a service but yeah it's like work clothes are particularly notorious to be on public github repositories because these are often projects that people are doing outside of their work so they want to be a a good a good employee they want to create a slack bot that's going to connect to your application and send everyone alert when it goes down or whatever it may be so you're working this on your own time and then you make it public so so they're they're notoriously widespread on public github repositories so if you're targeting a company let's say you want to target uh twilio i i'm not targeting twilio i could have picked any name that's just in my head you know a good strategy to do is to monitor twilio employees on their personal github repositories um are they leaking any any keys if you find a slack hook there's a high chance that it's a twilio slack hook or it's like a web hook or select key so you can find them in numerous numerous different different ways and then you know and once you do and you know when you look at keys you know aws key is is is is significant because it's obvious what you can do with it but a slack key could be more significant in the longer schemes because you might be able to get deeper into different systems but it's not perceived that way therefore security around it you're not going to be so careful of who you give that key who you give access to that key uh or who can generate that's incredible that you know that you know the adversaries can spend some time you know they have time to get it to come yeah they have time but they can also automate some of these problems oh yeah i guess monitor monitor the the kind of the slack uh users you know for that particular company and and eventually somebody will slip or eventually something like that would happen and then and then you write it's the instant trust i think that is is uh is key here and again pun intended yeah you know when associated with the um with the any kind of internal communication but i think that's why people should be aware that anytime they use a third-party platform you know uh you know highly recommended not to share you know secrets or not to have an instant um you know instant trust without verifying that the person is really who they say they are yeah exactly yeah yeah and uh and it's incredible you can still have all those uh checks and balance um you know i just uh heard a story the other day of a company um where somebody called uh basically somebody emailed the ceo asking to wire money to a different account for payments for for software services and um in the you know the system was verified so they called and when they called they said is that you they requested you know and they said yeah that's me go ahead it was it was a significant amount and what happened is um uh the reason why they still you know got hacked is that the the number on the signature got modified so they whoever they called right was the wrong person even though they did the right thing right wow from a verification perspective we had that exact scenario happen it didn't it didn't happen but we've we've had those you know emails going through jiren and i was like you know hey i'm in a conference can you please uh you know give me access to this or transfer money here yeah they can't they can't be convincing and again you know if it comes from ceo at gmail.com then it's probably not going to be uh you know if red flags are going to come but if it comes from an internal messaging system from that person then well already you kind of lowered lower that that that barrier to trust yeah absolutely so let me ask you this so you um you interact with developers quite a bit and and also uh you know at every seniority level as well interact with people that are responsible for development at companies large companies and so on um how aware are they um you know to this you know particular gap in security and you know like in terms of because again when i think about software development i think about these people that run really fast i mean talk about like you know they have to release and they have to get going in order for them to be competitive and so on so their mind is strictly in terms of where the you know it's strictly about like getting you know getting this stuff done and quickly so they can move ahead and but security is not necessarily something that's on top of their mind so what's your take again from from your interaction with these these folks you know how where are they you know when you start talking to them about the you know the kind of the the caveat of development and so on and uh you know how aware of the of this situation yeah yeah really interesting really interesting uh perspective there and there's kind of lots of areas to to run in this you know the but we're specifically talking about developers here i i i think what's important to realize and what application security professionals should realize too is that security is vast subject it's very complicated there's lots to know if you're dealing with it every day often it can seem complicated but it's it's it's difficult and application security engineers security engineers sizeos all the way through deal with and know how to do a lot of information it's not reasonable to expect a developer to know all of that however i'm a huge believer in shift left i think it's reasonable that there's certain areas that developers can own and can help to kind of be aware of that and so there's this disconnection with security teams i see a lot where security teams don't talk to development teams and it's friction everywhere because developers are creating problems for them and the security teams are often blocking or preventing developers from moving quickly to meet their goals so there's friction uh in that system so we need to kind of you know you need to to work with everyone to try and remove that system and and what's critical to remember here is every company is a software company it doesn't matter if you make paint or t-shirts or if you're building sophisticated cloud infrastructure you're a software company big pizza because you've got yeah you if you're big enough you've got you've got teams you've got there's a lot you know you're building sales at tools or you're building monitoring tools or your factory has proprietary so you know you're a software company so you need to you need to be aware of this um and so you know friction kind of comes from there particularly if it's not if development isn't ingrained into that into that culture so uh yeah software developers need to need to to move quickly um but there is a there is a there is a lack of kind of security um training out there apart from the ones that are really interested in it and secrets are a great example open source dependencies are a great example because these are things that developers deal with uh directly and what i've always found is you need to give them the tools um to be able to you know to be able to take security into their own hands it's it's not reasonable for them to know everything about security but if we focus on the areas that are in their control directly and give them tools that are going to help them that sit into their their workflow then that uh significantly helps and act like a critical piece here is that the tool that guardian makes for i'm just using a guardian because i you know i'm intimately acquainted with it but the tool that we give apsec professionals or security teams is not the same tool that we give developers right because it needs it needs to fit into their workflow it needs to be designed for them so if you're wanting to give developers tools it has to be tools for developers it can't be an appsec tool that may be able to do the job but doesn't fit in with that that workflow so you know that's kind of you know a critical element too if you want to stop security flows coming in from that development team you know purpose-built tools for developers that allow them to control security from their perspective and empower them and then they're going to be interested and demonstrations work you know don't just tell them they're bad because they leaked a secret or they injected a vulnerable framework or something else you know show them show them how you can exploit the secret because that will last in their memory forever uh there's some there's a great framework called beef um you know and you could you could it's a basically a you know a a malicious framework that you can use and you can do all kinds of stuff you can make an application to turn on someone's webcam you know do all this kind of stuff we should do an episode about that and showcase a couple couple things because that's that's cool stuff you know yeah you know you know definitely and and so if i was a you know software developer you know don't i want to be you know proud of saying okay my you know the way i i run you know and develop my code and i you know it's more secure is there like almost a badge of honor um right now because and to me it will think about a way to differentiate yourself you know there's a lot of software developers out there but you know if i can come back especially with some critical applications let's say i work for a company that does you know more of you know kind of a requires a high security maybe financials and so on don't i want to have that kind of clout associated with my work i look absolutely there's definitely developers out there that do it but we're also kind of we're talking in a kind of a form of currency right now of you know like you that that clout that you have is is currency but what's also currency for developers is building really fast applications building applications that use the minimum amount of memory or being able to build applications to do certain things so what's you know so from a developer's perspective you know what's what what are they thinking is more cool that you rebuilt a framework that uses 20 less uh physical memory uh to do the same job or you didn't blow up your last company you know so this is this is kind of the struggle but there's developers out there that love security and are super passionate about it you know i love it i find it fascinating um but i think it's just and and just what you see with application security professionals that also find it fascinating or security teams that that that phone is fascinating you know you sit there wondering why don't developers care about this and it's not that they don't care it's just that they're caring about other things um you know and this is mind sharing yeah there's so much you can care about exactly and again if you're you know if you're a developer as you mentioned you care about certain things you know like you know running faster quicker you know making it more you know the cooler aspect of software development is that yeah it's not necessarily because it's almost like if you and you know if if just thinking online and brainstorming here the problem is if if your code is secure it's not necessarily something that's evident there's no impact there's no like somebody like looking outside and say oh good job yeah yeah congratulations happening on the shoulder it's a great job on making this code secure it's just nothing happens you know things don't just blow up but if you make it faster cooler better you know there's somebody out there that pays attention yeah absolutely and there's there's a lot of there's a lot of things that you can do um in this you know giving giving no one wants you know as much as developers uh you know are interested in lots of different things also no one wants to be the guy that blows up the application right so give them tools to be able to do that so that they can care about it but it doesn't you know it doesn't affect them in their in their in their running life and build up security champions build up programs and you know it has to be in the in in the culture if you care about it if you're if you're a security team and you really care about security you need to you need to ingrain that into the culture to make that a currency um so you do reward people for building you know secure code or you know or different different different elements for that you know how many you know how long has it been since the the security incident you know we see that in the warehouse things you know all of these things get everyone on board with it because you know people care about what other people care about um and there's there's a lot for for developers to care about i certainly don't want to make it out like they don't care about security it's just that there's a lot to there's a lot to to consider yeah that's right and they all want to be i think the majority you know people and developers as well you know it's so you know they want to be good citizens and yeah and what people realize i think today is they realize the security is everyone a responsibility and it's in the collective um benefit for everyone because people don't realize that if a company gets hacked um you know there's a lot of impacted you know involved you know just the people that are directly uh maybe customers or employees that are directly impacted maybe potentially company closing down um there it would be like you know the ripple effect uh you know for the entire society depending on the size of the organization and who's impacted so it's in our collective best interest uh to you know create like secure you know source code and and making this a secure you know a secure society right so because they and you mentioned everything is software today so yeah you know do i want to order pizza and have my credential um you know expose or or the type you know i don't want anybody to know that i'm you know did i eat pineapple and and ham on my pizza no i don't so i i just try to keep all that secrets uh um you know secret yeah uh so so uh yeah so listen i we are running out of time for this first session um but i would love to to maybe dive uh deeper into some of the as you mentioned the um the ways you can expose code i think we touched upon some of that but i'd love to see some some examples of that as well and then in the upcoming episode i'd love to to bring uh you know conversation for example with the people responsible for for security maybe csos that are responsible for that and how they interact with the software development team and that interaction between you know business uh goals and and secure code uh maybe bringing some developers using the tools and people that are responsible for hands-on coding uh as well as everybody who uh maybe on the business side you know the people responsible for going to market quickly and you know for the company to be competitive and so on so love to to get all those as part of this podcast series yeah it's going to be great and uh what i'm so excited about doing this with you david is that you know we're going to get a broad range of things this isn't going to be about uh secrets every single time we're going to we're going to learn from from from lots of different people so i think that's a great reason to be able to tune in um you know and to participate and and and if you're if you're you know got something to share to to reach out yeah absolutely and then i would love to also um maybe you know there's always going to be uh you know some vulnerabilities and breaches and so on and the ones that are be related to you know specifically to you know code development and potentially uh you know expose secret or love to unravel those maybe discuss we'll come up with some you know cases uh that happened and uh potentially how it could have been stopped and so on but we we did uncover quite a few um uh you know potentially unknowns in this in this conversation so yeah i love to to uh to continue the uh this uh further so mckinsey thank you very much and looking forward to uh to future uh episodes absolutely thanks so much david and uh yeah we'll talk soon looking forward to seeing your uh vlogs as well i'm not sure if i'm ready to share that with the world yet i'm not escaping you're not escaping that okay because you know i always i find somebody's uh you know passion and i'll make i'll expose them right on this walk you're sharing all right okay take care talk to you soon see you next one