yourself hello everyone we should be live now welcome to this webinar uh really excited about this one it's going to be great this is the first time we're using this platform uh uh livestorm I forgot the name briefly livestorm is what we're using so welcome uh it's going to be really interesting topic to to first before we get into it and while we wait for everyone to join uh give me a give me a message in the chat to know that you can all hear me there we are everyone is uh is is tuning in so hopefully you can see me as well um uh great it's always a good start when using a brand new brand new platform is that there is uh no issues immediately but today's discussion we're going to be diving into solving the the secrets management puzzle uh so Secrets management is something that we're frankly not doing great at uh and we've assembled uh a panel to kind of go through some research that get Guardians recently just published and provide different perspectives so I'm going to be moderating the conversation then I'll do a little bit of a presentation just to start with we'll be short 10 minute presentation from me and then we'll get into the meat of what we're really discussing but just to introduce everyone that's going to be up hearing a little bit later on we have Mike Gary from one password so one password also offers Secrets management for developers we have James Governor from Red monk which is providing an analyst view someone that has a very good holistic uh understanding of what's happening in the world of developers and really understanding the issues and problems that they're facing and then we have Andre Perdue from bestseller who's a practitioner someone's working in devops engineering someone that's actually dealing with this uh on the grounds in a company so they're all going to bring their unique perspectives so really looking forward to it so before we get uh into it let me know where in the world that you are tuning in from I love to see that we have a global audience so in the chat let me know where you're tuning in from I'll start I'm coming in from you from Oslo in Norway I'm here for a conference uh briefly so let me know whereabouts uh in the world you guys are tuning in from Canada Manchester Boston Washington India Nairobi Ghana Nepal oh my basement good spot uh Colombia Germany Italy you know well Philippines wow we have some really cool ones uh really great see Palestine wow I think this is the first time we've had someone from Palestine so welcome Portugal slash Netherlands two places at once uh Paris turkey India Italy wow we are truly a global uh audience today uh it's really cool to see I love the reach that we have we strategically pick this time uh because it covers the most amount of time zones so I apologize it's very early in the morning or very late in the evening for you but this is uh research has gone into why we picked this exact time uh wherever you are in the world but just to uh let you know um in in the crack obviously you've all found the the chat button there's also a button next to it that says questions so you can actually ask questions throughout the webinar and by doing that by participating by asking questions by participating in the chat yeah it gets you in the draw to win some prizes from us we'll announce them next week on Monday we'll announce who the the winners are of that um but basically ask lots of questions participate in the chats and that's going to be the best way that we can uh share your enthusiasm and reward you with some gift cards or some packages or have some sort so please make sure you participate all right we have all the pleasantries out the way let's get stuck into it and I'm going to present kind of the the surrounding topic that we have so earlier on in the year you may be familiar we release a report it's called the state of secret sprawl um and the high level I'm not going to talk about this too much but you know we've had 10 million secrets that we're leaked on public places in GitHub so proving that there is really a truly massive issue uh in in this space so it leads you to the question of why why are these secrets and when I'm talking about secrets that you just in case you're not familiar I'm talking about API Keys credentials security certificates really the crown jewels of uh of our organizations and these are made to be used programmatically so machine to machine but the problem is that humans touch them and when humans touch something there's a chance for them to kind of leak out so why why are we historically in the evidence would suggest quite bad uh at managing the secrets so we went on a bit of a road trip around uh asking everyone that we could uh really what are their key issues why is this a struggle how are they doing and dealing with it in their organization um and we're happy to say that we managed to get 500 plus sizeos and Engineering leaders to answer the survey that we did so massive uh segment uh that we have and we've put it all together in a report called the voice of practitioners the state of secrets in appsec so that's what we're going to be discussing today we're going to be discussing some of the finding things from those survey we got a neighboring panel and basically we're going to dive into it and essentially challenge challenge it uh so it's going to be quite interesting so just to give you a very high level of of some of what we found in the report now I'm sure they'll they'll in the chat if you are there'll be someone that can post links to the report in there but I'm just going to go through just the high level of of really what we found uh in here so three out of four respondents declared that their organizations have experienced a secret leak so a secret leak can be quite significant this basically means that someone's been able to authenticate or could have been able to authenticate themselves into their systems or services so secret has leaked out of its control um so it's three out of four is a lot that's much more than half obviously uh so you know 75 percent it's it's showing that there is a real issue here and it's not an issue that's just hypothetical it's an issue that's actually fetching affecting people so you can kind of look at the breakdown of what we've had but only 25 actually answered no that they've never had a secret leak of some form uh with that so this is really quite concerning and it does continue to show the trend that what we've already kind of know from our other reports and other research is that these secrets are fairly hard to manage you know um and something that shocked me is that a quarter of the teams um over a quarter of the team still rely on manual code reviews to detect Secrets now manual code reviews are a terrible way to actually find these secrets so if you look at a scenario an engineering scenario with software engineering of how these Seekers would leak a common scenario might be that you have an engineer that's working on a feature they're working on a separate branch and get perhaps so they hardcore the credentials just to get it working then they remove those credentials later on because they're going to use environment variables or however they're supposed to manage their secrets a hundred commits later it goes to code review so that code Reviewer is typically going to look at the latest version that latest version is probably going to be free from secrets but is that code reviewer going to go past the last 100 commits to make sure that nothing sensitive is leaked probably not right it would be very surprising if someone actually did did that so manual card reviews is a terrible way of actually identifying where these secrets are leaking through not to mention the fact that they often leak through like debug logs or environment variables places that you probably wouldn't think to look so if you're debugging something that debug log might create a printout of your environment that will have environment variables in it which could be secrets so these types of things that maybe you're not thinking about that you wouldn't review manually so the fact that about 27 admitted that they're relying on manual reviews is showing a pretty scary state of where we're actually at we're not even implementing basic tooling to be able to detect these secrets so slightly over half the engineering teams use a dedicated Secrets manager system so this one really confused me uh what are you using if you're not using a Secrets manager uh obviously there's some answers in there but having a dedicated Secrets manager is is incredibly important it's the first step because it how are we meant to handle things correctly if we don't have a tool to be able to manage them in the first place it's no point being upset if a secret leaks when we don't have a place that we can actually securely put it that's well defined that has policies around it so people are sharing them uh on messages in clear text we can see here people are sharing them in code repositories so just because something's private doesn't mean it has adequate security to be able to store sensitive information like this so what you'll notice that a lot of adversaries are doing and the lapsis group is one that's recently done this is they will purchase session cookies often for places like company slack channels off the dark web so this is quite common to purchase these cookies cookies will bypass MFA as well so it's better than a credential but once they're in the slack Channel well then they're probably going to be able to uncover lots of different secrets that have been shared in there if a sequence manager is not being used this absolutely is probably how secrets are being shared and it's a really a problem problematic way of doing it to say the least and so this one here is interesting and it's going to lead into the next slide too is you know have you considered implementing any of the following in the coming 12 months so have you considered implementing new tools for sequence managers have you can considered implementing new measures to prevent secrets and only half the people you know are kind of saying yes we're ready to implement some some new tools some new systems in here uh which is it's it's just scary because uh we've already seen that we have a problem here and there's a lack of Tooling in place but there doesn't seem to be the appetite to be able to replace that and that's really what we've found uh overall is that 47 of the respondents that we have identified that hard-coded secrets are a key risk point in the supply chain so probably not that surprising this is absolutely uh very true but only 26 percent said they would actually invest in secret detection and Remediation 23 so it's showing this disconnect that we're we're facing uh right now is that we know it's a high risk yes 75 percent of people have even had experiences with with leakages but we're not willing to invest in this now this could be budgetary constraints this could be a fact that you just don't have people that can manage these projects uh but that's quite scary and quite concerning to kind of see that there's disconnect and it proves that we're kind of probably not going to improve on these numbers that we keep putting out as well so when we actually looked and asked people what application uh tools were you you invest in and there's a couple that come right at the top that you probably would imagine API security is very hot right now so number one said they're looking at investing in some API security testing um web application firewalls wafs is another big one Dynamic application security testing dast um and the list goes on and then way down the bottom you have secrets detection and Remediation and secrets management uh so I understand that managing Secrets may not be as sexy as you know having some cool Dynamic security testing devices and other other other tools that appeal but we're still not getting the fundamentals right and we really need to invest some some time into this so interesting results uh that have come out of this here so one other interesting thing that we had is we asked you know are they confident about their organization's ability to be able to prevent secrets from leaking and about half said that they were their confidence that they would be able to prevent it so that's probably the half that's investing in the tools that's probably the half that is kind of doing the research has those tools in place that understands what's happening but as we can see by these results is that you know we're really still a long way off from actually having the appetite to try and correct some of these problems and what's really interesting is that you know lead Secrets is a consistent problem it's continuing to be a problem um they use secrets are used in many different breaches perhaps maybe not as initial access but definitely bomb attackers once they make it onto your networks they will immediately start numerating through your network scanning information to try and find secrets to elevate their privileges because if they're reliant on a session cookie for their access then that can be shut off quickly so you want to make sure that you have multiple areas of access how do you do this Secrets but only half the organizations which we know are also not investing money into this are actually confident that they're they'll be able to prevent that if an attacker makes their way in or of a secret leak so uh it's really challenging insights that we get for you so that's kind of the high level of the report so what I want to do now is uh invite back on to the stage uh the guests that we have here so the part that I'm most excited about um so I'll do an introduction and if you guys can all turn your cameras on so we can see you hopefully everyone will be able to see you there I'll do a quick introduction for everyone and also invite them to maybe tell them a little bit because I have uh Mike Carey Carrie is from oh I just had an internet I think we're good so my carry is from one password um so one uh one password you may know is a as a password manager oh Mike do you just want to keep going and introduce yourself uh troubles I'm just going yeah I can continue um so I'm Mike I'm a senior product manager at one password as Mackenzie was saying one password offers a password management solution but we also um have developer tools and that sort of address a wide variety of use cases and I work specifically on the uh infrastructure Secrets team in our developer tools okay um I'll jump in uh I'm uh James Governor uh co-founder of company called red monk we're a research company uh we spend our time trying to understand the choices that developers engineers and practitioners are making uh we live in a world today when a technology decision making is is ever more uh from the bottom up and that's what we focus on at Red monk so yeah just trying to understand uh developers and the choices uh that they make um and uh Andre over to you yeah hi my name is Andrei Prado I'm a devops engineer at the bestseller as in Denmark we're a fashion design and retail company and the I.T part is responsible for the back the ID backbone of 20 or so different fashion brands so that's about me yeah I apologize I cut out for a brief minute of the Epic we've got the internet we're on a more stable internet so I apologize but thank you for taking uh for taking over um so thank you all for joining Mike uh um have you introduced your yourself or shall I did to do a yeah we're all good we're all caught up I shouldn't have said anything I could I like that was the perfect time for my internet to drop because you guys were able to take over so at least at least hey this is happening live this is what happens when we're live uh but yeah so the reason why we assembled this group is because it covers a broad range of people that have different perspectives so I'm I'm imagining that there's going to be some challenging ideas here that come up but we have people that are building the tools to be able to solve this problem uh Mike and also myself from your guardian and we have James who understands what's happening in the industry uh from an uh from an analyst point of view with the age of the ground that understands the troubles and then we have Andre who's got their their fingers in the mud it's actually dealing with this they can tell us that these conceptual problems that we talk about are you know how they affect in the real world so what I want to do is I just want to start off with a question for all of you uh so one of the key points that I meant out there is that three out of four respondents reported they experienced a leak so the first question that I want to have is does this surprise you and I think we'll start off with James James did this number surprise you that three out of four people had experienced some kind of leak of their credentials no not at all um you know we we I think as an industry we generally have not really done a very good job of security um I think part of the problem is is having these very it it has been a separate domain uh if we think about um other parts of the software development life cycle um and operations we've tended to try and bring um uh the the responsibilities together um and uh collapse some of those processes move away from things like waterfall security has been a bit of a holder hold out we've got the sort of separate team um that that is is in charge of security and I think too often I mean one of the things I always notice is just flying in um to San Francisco and you arrive and there's always still in 2023 there's always some someone selling you know like a firewall Appliance it's always like silver Barracuda or silver shark or silver something and it's this idea that you will buy this thing and then you will be secure and I think as an industry we keep on making the same mistakes um and we don't take it seriously uh you you see these wonderful presentations where they say oh reputational risk once you've had a breach means that you'll be out of business within nine months that's BS um I can point to most of the companies in the Fortune 500 and they probably had a breach at some point so I think just as a whole we've got a lot of work to do and it isn't surprising because we see these leaks every day um we see identities being leaked and it is something that we need to do a better job of frankly surprising absolutely not and Mike what what what do you uh bring to this if someone that's kind of building these tools were you surprised that uh that this is so prevalent still no um it's concerning but but not surprising I think um you know James mentioned that there is no Silver Bullet to this problem um but I think the problem is that there's there's still an awareness Gap when it comes to sort of the importance of managing secrets and then understanding what Solutions are available to actually help you do that on a day-to-day basis um you know if I'm being optimistic I'd probably assume that a lot of these respondents that said they weren't using dedicated Secrets managers are still leveraging some of the secure Secrets management within their infrastructure like maybe AWS Secrets manager or gcp Secrets manager but it probably doesn't go much further than that um and I think the reason that those tools get used is because of how tightly integrated they are with the infrastructure so I think the key to really sort of promoting this and educating users and getting wider adoption is is meeting the developers where they are um you know and and helping them manage Secrets throughout their entire software development life cycle um you know whether it's in the IDE for example we have like a vs code extension that can replace Secrets or in your CI CD pipelines um you know meeting them where they are so you can get that kind of adoption that you see with some of these other uh Solutions yeah I love that sorry to just jump in again and I'm I'll I won't speak for a moment after but yeah I think that's so right meeting the developers where they are and generally you know it's I say jokingly developers only want to be in like three places they want to be in GitHub uh they want to be in the editor and they want to be in slack for sharing uh gifts and stuff so you know uh integrating um your Solutions with them uh integrating your secrets into their workflows I think is really critical I think we are going to talk a bit more about developer experience in a minute so I'll I will stand down but I thought what Mike said was right on yeah well I want to bring Andre into this one and Andre I said in the presentation there's a lot of people that are relying on manual code reviews so from your perspective someone that's in there you know that's that's in the trenches let's say you know is this a figure is this something that you've seen too that we're where secrets are kind of only detected in manual places and and like would would this be effective yeah so I think uh there's two things here first um it depends on the um let's say maturity level a cmmi you would call it a level of the company the strength of the processes and the strength of the development teams and even then you're talking about humans we all make mistakes and it's easy to lose something and not see it as you said in the git history or uh where you have some older untouched code a few months ago we you know we found some Legacy code and some uh SVN repo and we said okay we have to bring this into our GitHub and then well let's say we found a lot of surprises in there but there was no pull request there was no this process you know wasn't available there so what do you do you you need more than that it's not enough yeah I got it now James I know you said you were you you're gonna stop talking for a minute but I'm gonna I'm gonna bring you back for for one for one more thing because I know that you understand the industry and as a whole and one of the things that the report really identified for me was a disconnect between uh what people are seeing as a risk and what they're willing to to basically implement or purchase or not even purchase because there's a lot of free stuff available open source stuff but just you know Implement invest you know invest in and introducing these tools so 47 they said they identified hard-coded Secrets as a risk 26 that said they would invest in sequence detection and Remediation why do you think there's such a disconnect at the moment with the risk that we clearly see and the willingness to be able to implement solutions for it I think it's just human nature we're we're really bad at risk management and we're really bad in investing in avoiding risk um we see this in in all forms of Maintenance you know technology or otherwise um you know people will um you know Drive their car when they know uh it has a fault and and then you know they're not surprised uh when they have to pull over to the side of the highway they they get angry but they never um they never got it serviced or you know me riding my bicycle um I'm like why is it Knocking this is strange and then you you you you wait you wait and you go no this is really bad and then you go and I you know and it's like oh no you know you're gonna need a new wheel um because you didn't do that before I think it's human nature I mean I you know climate change pretty sure it's a problem but invest to fix the problem oh no oh you know so yeah we live in a world of identifying risk and and and then honestly not doing anything about it until people will smoke until they have a heart attack and then they'll give up and unfortunately we sort of it seems like human nature is we have to go through these really uh traumatic events or need really you know very industry regulation that's one reason um that you'll invest in this stuff we should have in all Industries mandates for automated Secrets management that's obvious um so it tends to be either a near-death experience or it's going to have to be mandated from from outside and and yeah it's it's it's disappointing um but again it's it's I do think it's human nature that's not um it's not a technology problem uh it's just the way humans are it's the it's what they think they call the broken window effect where you a window gets broken the first day it annoys you you're very aware of it but if you leave it for a month you will eventually just stopped caring or stop noticing that it's broken um pass it on to you Andre as well now I know that you guys have you've implemented lots of tools you've implemented tools on managing secrets you have intermittent tools on detecting Secrets you've implemented policies around it but why but that's kind of a standout at the moment of what we're seeing so you know what are the what are the challenges that you see in actually being able to implement some of these changes what prevents organizations what prevents people in your position to to implement these changes so there's there's a lot to unpack there and I'm gonna try to keep it short so firstly you need the right motivation and usually as a both of you have been saying that comes after somebody gets uh gets it right some Secret Gets leaked some malicious infrastructure gets deployed some money is lost some reputation is lost unfortunately but when it gets down to actually making a change first you need a good solution so you need to have you need to Define uh the tools that you want to use and the way of using them and have it available for your developers then we do we kind of have a few steps first we teach we Empower we encourage and then reinforce that's kind of the the change pattern that we follow in our in our team so we teach them how to do it right we Empower them with the tools and the knowledge we encourage them and want to have some critical mass we will turn the switch and say now you can only do it the right way by via in this specific case we're using uh git Guardian uh pull request checks and pre-commit hooks and we're using scanners and we stop them from pushing directly to master no matter what etc etc right so yeah it's very interesting um but as I say you know like this is this is really on the bleeding edge of things of having these policies and and much deeper it's not something we're seeing um and James touched on something that we're going to come back to uh in terms of kind of talking about governance and stuff so I want to I want to bring up something that's kind of controversial I'm putting this slide in here because I disagree with it but I I I like the person that said it uh so during a during a quote one of the things that I said is while secret sprawl is a problem most companies experience it's not hard to solve so I disagree I think it's very hard to solve uh completely but I wanted to kind of get your perspectives on this so Mike I'll start with you uh in secret scroll uh and someone that creates a secret manager that that helps to combat this do you feel that secret sprawl uh is something that you know can be can be easily solved or at least solved um I honestly that that sort of sounds like famous last words um I'd be I'd be very worried to speak something like that because you're you're asking for trouble um but it's uh I think the reality is is secret sprawl is is an ever-evolving challenge um you know your organization nothing is static everything's growing um organizations grow and and even recently with you know the increase in remote uh workforces that's introduced a whole new set of problems um you no longer have the physical security of an office you don't have the security of a of an office Network everyone's on their personal networks each might have vulnerabilities um then digital Communications now I mean you mentioned people sharing secrets in slack um I'm sure that has increased over you know with remote workforces do you have screen sharing you could slip up and expose a secret through a screen staring session accidentally commit a secret um there's all all kinds of uh challenges and and they will continue to grow so I don't think it's it's never a problem that can be 100 solved it's a it's more of a of a strategy that needs to be implemented and continuously uh monitored um and I think it really sort of starts at at the organizational level and and on every CSO should have secrets management and secret sprawl uh front of Mind regardless of the size of their organization um because we all know you know a single secret leaks can just send shockwaves through an organization so it is super important um and it pains me to read this this quote yeah well I mean I I I I really like Brian and one of the things that I say I guess a little bit controversial too kind of along the same lines is that secret sprawl could be a solved problem in the terms that we have all the technology to solve it right like we we have the ability to have secrets managers we can share secrets securely we can store Secrets securely uh you know we can we we technically we can do it it should be a solved problem but it's not um so that's uh one area uh James do you have anything to say to that I mean with every problem in the world the the human factor is the Wild Card yeah we can solve it with technology but the humans are the ones who will make the mistakes the humans are the ones who are going to accidentally leak a secret so um that that's what I mean by it's an ongoing um challenge that you always have to be addressing because uh even if you have everything in place to technically uh remove the challenge of secret spiral there's still humans involved and they're the ones who are interfacing with these secrets so yeah absolutely I mean even these secrets they made to be used programmatically from machine to machine but humans still need to get their grabby hands on them and touch them and put them places they should intervene oh sorry yeah girlfriend no no no no I'm very pleased um I would say that it is somehow straightforward to do on a technical level but the people factor is also very very important and I think uh the technical side is just work you have to do you read up you study up you see what the technology is and you implement it you follow the guidelines but changing people's behavior can be a lot more difficult at the end of the day in my opinion and I have a funny story there we automated the DNS provisioning in our kubernetes clusters it's quite straightforward to do and then we said okay you do it like this and you don't have to get access to our DNS or do any of that and then somebody got the service account credentials from the cluster and we found it in a git repo so you know look it happens there's so we have uh uh the the website that we have again someone post it in the chat because I feel like I'm gonna get it wrong but it's uh uh shitty shitty secret management.dev or or secretstory.dev someone can post that in the chat for Greek Guardian there's a website that we have where people can share their funny stories about how credentials have leaked um and there's some there's some doozies on there I gotta say uh just before we move on uh James do you know do you does do you have a a kind of an in you know an opinion of this as well yeah I mean um I think the kubernetes example uh was was was absent on point we see this again and again um with the best will in the world and the best Automation in the world um you are you're still we're dealing again with humans and humans make shortcuts and uh humans uh make mistakes and and yeah um you know I think that that we're we're going to be in a world of of mitigate you're always in a world of mitigating risk you cannot eliminate risk um you know as a as as a parent you know my sort of my job um I think very often is risk management and you don't want to go too far um there have to be some freedoms there but clearly you need eyebrows and stuff occasionally our stuff will go wrong so um you know uh secret sprawl um is is is is is you know um again I I I don't I don't think we ever get 100 in anything when it comes to technology yeah definitely and uh just just in case everyone I I just want to say that there that this wasn't there to shame and you want to Doppler or Brian it's I just put this in there because it's controversial and uh uh you know an interesting talking point and you know I I'm sure if he was here he'd be able to defend himself but uh he's not so we'll move on so but why why why uh why is it harder why in my opinion so this is a great thing about being a moderator is that when it's my time to speak I can actually have slides that go with what I want to say so it's totally unfair but I'm gonna take the advantage but you know it's the reasons why I feel like this it's very hard to solve this problem is because uh it's really hard to catch them all without having any false positives everyone that works in security is affected by something called alarm fatigue so you know how do we capture all the secrets without kind of being alerted by too much lamps it's difficult secrets are everywhere so they're in the slacks we've talked about it they're sprawling around uh death keeps growing so when you don't have a proper Implement in place when you don't have secrets managers in place to to be able to solve this or detection then the problem gets harder and harder to the point where when we roll that git Guardian we'll find thousands of Secrets and there's almost so overwhelming that you don't even want to look at the problem anymore because how can a team of two people deal with 3 000 incidents uh you know and move forward so there's another issue of kind of why it kind of gets put off because once you know that you once you know the secrets of the year you have to do something about it if you don't know that there you don't um and then you know redeploying them revoking them going through the process it's not a matter of just finding a secret you have to you have to go through the steps of actually changing it so that's kind of what I kind of look underneath the service um but I do you know I do agree that the point that technologically it's solved on paper it's solved but we still can't get there so what what does timing Secrets brawl look like so uh it can be quite challenging so onto the top level you know you have we have two columns here unmanaged Secrets bad manage Secrets good uh so we won't hard code the secrets and clear text we don't rotating Secrets we're using long-lived credentials credentials that live on forever and we you know compare this to the opposite using volts automatically rotating them but it keeps going on um permission Scopes we've got the principle of least privileges making sure that we're not just uh creating admin credentials uh auditing etc etc etc so it's actually quite daunting to do it so how do we go from one side to the other so oh Andre I wanna I wanna start with with you because you've done this uh you've you know perhaps you weren't ever on the the left side but you've you've certainly gotten further and further of right what do you start how do you start dealing with this with this what what are you you know what could you prioritize stuff do tackle it all at once how do you actually go about solving this issue if someone's listening go okay I have an issue but what do I do how do I start so we we didn't have a a fresh start but we did have a start when my team got formed we were set to design and carry the company into the cloud and then we got a chance to look at what our company was already doing what besser was doing for secrets and we decided that's not gonna cut it that's gonna be useless for us it's not automatable it's not it's not what we need so we had the courage and we pushed a lot actually and we ended up succeeding and we deployed our own secret server where you're using a vault from hashicorp and then we had a long process of building a structure for the teams and environments and then defining how that should work and teaching everybody how to use it and what we expect from them and then also leverage this technology to do automated credentials Dynamic Secrets integrating with worker with identity you know secret less workloads and all that kind of stuff and it's still a very much ongoing process because the things we did in the beginning now they have really good Alternatives they are even better and we learn as we go as well so after that after we were happy and we were kind of in a good place we we looked at how do we clean up what we already have especially the ton of Legacy code that we brought in from the old Erp systems the old side projects the old consultancy that uh solutions that uh you know we purchased and we saw a lot of problems in there and we looked at the git Guardian and I can go into why but we thought it was the best one and we used that to see what problems we had afterwards uh remediate them one by one and then block Right Use the technology to make sure Secrets don't end up in our repositories and that people use the right uh the right solution that we made for them yeah that's kind of the story the one thing that you said there which is interesting is you said courage and I feel like you know like that's very true it does take courage to be able to go on to this about because it's a it's a big journey and one that certainly comes with a lot of challenges I I want to put this over to you Mike uh because it's someone that creates some of the tools for this um you know like where would you recommend people to start uh in starting to to deal with this project you know is it perhaps once they've rolled out one password or they've rolled out a sequence manager you know what's what's kind of the next steps what's the first steps and what can people do what would you recommend for people uh that you're interacting with yeah I mean from a password manager perspective I think what we're trying to do is just educate users on the use cases Beyond um sort of what they intuitively think a password manager is for which is just you know username and passwords and maybe uh totp tokens um so I think it's really our role to try to educate users on all of the other Secrets they're touching day to day that they might not even be thinking about and trying to encourage them uh to use those and then make the the process as seamless as possible that again we can start moving up the sdlc to start you know going from the individual developer and and their workstation uh to pipelines to um actual infrastructure um you know one way we did this recently was with a SSH key agent um so most developers don't really think about SSH keys they they have this private key sitting on on their machine that they create you know maybe once every time they get a new laptop and just leave it there and nobody thinks twice about it but once we built an SSH agent into one password so you could sign git commits with your fingerprint um it created such a frictionless uh solution that people started adopting it really quickly and now they don't want to go back to having to create SSH keys on the command line or manage them so I think I think by introducing features um you know and identifying things that um like we have ways of of prompting a user like hey this looks like a pass a sensitive credential you should store this in one password by doing that and sort of getting into their workflows you sort of chip away at it and move it up and I think the other aspect of that is you know not just starting at the developer but starting at the organization and educating the organization about the tools that are available and um you know what what they can do to help increase their security posture definitely it's very very Sound Advice James do you have anything to kind of add to what you're seeing in the industry of you know what would you recommend to people if someone coming to you and asking how do we how do we solve this problem where do we start yeah I mean I think that one of the things is obviously you want a good conversation between uh developers and um you know platform teams developers and security teams um and you know that's always it's always easier said than done um but a Frank conversation about Improvement and this may be a trigger on um you know people are a lot of organizations are currently modernizing their pipelines and thinking about what that should look like so that's a good time to have a discussion about hang on a second okay um we are going to you know upgrade from our 20 year old Jenkins systems and um sort of thinking about uh what what that will look like so while we're going through that process bring the developers in and ask them what they would like to use you know developers know about one password they're good examples Hey you know the the kind of experiences you had there could be valuable here are there any tools that you would like to see there well let's have a Frank discussion about this kubernetes roll out because frankly um the the default privileges are totally absurdly open we're going to need Secrets if we're going to have these micro services so let's have that conversation together and I mean that's I guess that's I don't know that sounds a bit sort of um uh a little bit of apple pie but I do think a Frank conversation between the different constituencies is going to be helpful it's not that developers don't care um it just hasn't necessarily been something they have been thinking about so involve both or or all three of those platform security and developers in a conversation about moving forward and making secrets for all in Secrets management a problem that the organization is going to fix together because I think that's how we've solved um well that's how we've got better at some of these things testing itself you know the shift testing left um you know these are things where you provide better tools um but also you listen to developers and you bring them closer to the platform teams it made me feel quite old there because you're like oh my 20 year old Jenkin thinks I remember seeing it feels like yesterday someone showed me Jenkins for the first time it was the most wild thing I've ever seen in my life so you could do what what automatically obviously now it's very old but I'm old enough that I remember talking to Kazuki when he just built it it was running on a laptop under his desk at Sun Microsystems it was called Hudson yeah I uh we're running out of time and I know I'm not going to be able to get through everything but I think that's okay because the conversation's great and is there there's a question that's coming to the chat um that that I think are really interesting and we can talk about this it's from Anonymous user 35 first name or last name I'm not sure the secrets and basically the question is are secrets only the responsibility of the security team how can Engineers actually help um so I I just thought maybe we quickly spend some time on this at an appropriate time um Andre you know how how can developers help the engineering team you're part of the Ops Team what would you or the devops team what would you kind of really like to see some of the developers or or take on to help this yeah all right yeah there we go um I was gonna I was saying that that is I think uh extremely hard and it's getting harder and harder because the cognitive load on developers is growing extremely uh and has been for the last few years especially since getting into the cloud all these tools all these Solutions all these pipelines how they all chain together what are they supposed to be used for it's it's very difficult um so I don't blame developers so much but there needs to be at the end of the day a simple you know Common Sense stuff that we have seen is if you commit committed your password if it's in git then you should considerately it's not enough that you amended your commit or change the history you know report that to the security or um ask if you don't know right just say hey uh a devops team how do I do I have this scenario this crazy scenario I don't know how to do it can you guys help me out right that's what we're here for to to help the Developers yeah very cool I'd love to hear other people's opinion but I think we're gonna have to move on because we've got to uh we've got a lot to cover but uh you know there's there's some interesting areas of of why this is such a challenge too looking at it um and what Andre was just saying about you know working with developers is the developer experience of of Secrets managers is quite challenging at the moment as he said especially that the fact that you know they've got so much on their mind if it's not seamless if what you're trying to get them to do isn't seamless in their developer experience um then it's so difficult and we're going to talk a little bit more about that seamless too because uh the one password sequence manager has some great tools that Mike talked about about uh in in vs code that makes that seamless security training for Debs this one's interesting uh you know what what's you know should we be doing more security training for devs or or do they have enough on their plates I'm curious to know James what do you what do you think of this one here yeah I I think as Andre says they've got a lot on their plates um there there are we keep on creating new responsibilities and saying oh well this is the responsibility of the developer um but but definitely just training in general um you know we do expect developers to be sort of like you know lifelong Learners or whatever the phrase you want to learn uses but then we don't give them the time and or the budget to do that um you know if you are an organization that expects to involve your developers in a in a conversation about improving your security posture um but then you are not um uh offering them training if you are not uh offering them a budget for online education if you're not allowing them to go to that uh conference they want to go to um about the area um if you are not funding them to run a meet-up about the thing in your company then then you're not going to get good outcomes so I think too often organizations haven't been great about this I think they've got a lot better and it's good to see sort of uh you know Enterprises beginning to understand this but yeah you've you've I mean I think one of the the the the there's questions is it's about alignment of incentives we'll say a bit more about how do you you know get developers to do this but yeah absolutely more security training fine but in what format do they want to do it do they have a budget do they have the time to do it because if they don't it's not going to happen and if I mean yeah what sorry what is the purpose of this training because um uh a lot of companies are probably all the big Corps have this mandatory cyber security training stuff but that's that's not made for developers that's made for the lawyers to so all the developers are just gonna be like like this playing next or whatever skipping to it that's that's not going to do anything yeah and they're getting clever at making you do it too like yeah security training is you can't you can't yeah it has to be like high volume it has to be on your screen you can't do anything else when you're watching it they really make you suffer through that one uh but yeah it's really interesting I want to get onto something we've got down here in pose regulations but I want to talk about uh regulations uh kind of in total so we've got nist guidelines are coming out we've also got the the cisa checklist now these are starting to introduce secrets into their guidelines what is your opinion on having more regulation more policies and more guidelines that perhaps is even kind of has some legal consequences about adding Secrets is that good is it bad will it help or will it not help uh so Michael I'll maybe start with you you know what's what's your opinion on uh on guidelines policy standards uh that involve secrets yeah I definitely think it's um a good place to start like by having compliance in place it at least um sort of forces the the top of the organization to start thinking about this um and what it can do is it can lead to policies and processes to to handle secrets that may not even be covered by uh by the compliance regulations but you get that secondary effect of having those policies in place um I think on the flip side of that though one problem I see with compliance is often companies will sacrifice security to meet compliance if if the choice is between being compliant or implementing the most secure solution they're going to choose compliant because compliance since that has legal uh you know repercussions if you don't follow it so again I think that's where the service providers come in to offer Solutions uh that allow companies to achieve compliance while still having a great user experience and being able to actually take action on these things when they need to while maintaining their compliance James from a holistic point of view you know like you're you've got your head to the ground you don't have an opinion from one organization compliance standards yes or no what are your thoughts it makes people do stuff I mean you know I I think that um as I've said people really do need external stimuli in order to change their behaviors and look I'm not saying that all regulation is good um but I'm definitely not of the belief that all regulation is bad and uh encouraging um better behaviors and certainly look the regulated Industries are not going to become unregulated all of a sudden we're not going to see I mean you know yeah we're not going to see banks in Europe suddenly do away with regulations so we might as well have some useful parts of the regulations uh you know useful controls um in these areas that will make breaches and security uh problems less likely so I do think that's one of the positive one of the the the levers or levers if I'm speaking in a British accent that is um very applicable um in this area so um yeah I mean it's going to be part of the mix it does sometimes it's it's theater there's no doubt um but uh it it it it and you know there are interesting ones there you know in the insurance industry you know why why do people um have fire alarms in their buildings it's because it gives them a lower cost of insurance that's another sort of it's the these Financial leavers regulatory levers it's just part of of how mature Industries and organizations get things done and yeah I do think I do think it should be part of the mix okay and Andre just quickly as as someone if if regulations came in I mean this is going to affect you you know what's your opinion on it is it kind of like let us let us just do our job and do it well and if people don't do that so be it or would you be in support of having you know more restrictions or compliance it really depends so um right now I work in a fashion company so you can get an idea like a gut feeling of the level of regulations but I also used to work in a CMI level five company which is uh you know has the ability to do U.S government and Military contracts and health care and stuff like that and I got to see both sides of this coin and I can tell you that uh the legal and security uh parenthesis audit Department can take care of any regulation possible Without Really changing anything discernible for the everyday developer so you you can write your procedures and your uh the way you do things in your company in such a way that nothing gets done practically and uh you know you keep running as usual so to that I would say it really depends on the organizations and um the management to understand uh what actually needs to be done and push the right changes through or not I guess uh depending on what makes sense and for the developer for the developers uh usually is always a matter of sweetening the carrot enough and lengthening the stick enough uh I've improve my experience that's that's what works the best they're like well let's talk about developers now then uh we're running out of time so we're gonna have to be quick but uh you know one of the things that I like that you know one password is doing is is be able to make it easy and integrate into the tools with it is a problem with it's one of the problems why we've ended up in this situation is that we we haven't focused enough on developer experience um now Mike you're going to be you're going to be biased I'll come to you last but James what's your what's your opinion on uh you know on this do we need do tools and vendors like a guardian like us do we need to focus more on developer experience will that help in getting developers to use these these always always I mean developers are lazy um if you can make the right thing that if you can make the right thing the easy thing um I mean all humans are lazy it's not developers particularly I mean marketing marketing people can be lazy too but you know developers just want to get from A to B in the easiest way they possibly can and so from that perspective yeah make it easy for them um and they're much more likely to do it that's absolutely natural convenience is the killer application um no doubt um you know I mean I think you've got uh a great quote from um my colleague Rachel Stevens coming up in the deck hopefully we'll uh we won't have time to get to that but but yeah uh developer experience if you want I mean Andre talked about the cognitive overhead we need to make it easy for them developers shouldn't have to worry about Secrets because the platform is built in such a way that a lot of it is done for them they have to understand the they need to understand the issues but yeah they should not be happy in the glom together things you know move secrets from place to place be dealing with platforms where they have to do that there are apis here um uh integrated with the secrets management platforms and yeah the the the the it should be a discussion and we should definitely be making developers lives easier and if we do they're much more likely to adopt the the the the better working methods and the platforms so Mike have you seen better adoption since you guys have really focused on us so for those that don't know this little gift that's on my screen is showing uh Integrations that one password Secrets manager um uh because remember it's more than just a password manager but one participant actually you know integrates into vs code have you seen better adoption have when you've released these types of features has it increased the use of it yeah absolutely we've seen really really great adoption um the the thing about about developers um you know it's a hard group to to Market to you can't use traditional marketing techniques so so the growth ends up being quite organic you know through through Twitter and things like that so that's why it's it's really so important to focus on a good developer experience um because if you make something uh that is I guess the other point to say is you cannot interrupt a developer's workflow um you get one chance if you add friction to their workflow it's over um and they might never look at you again so if you really make it a seamless experience that that sort of creates that wow factor for them they'll go and and share that around um they'll evangelize inside of their their teams to to share this this new tool they found regardless of what you're working on not just Secrets any any developer product so I really think it's super important to um have that tight developer experience and it goes beyond the feature really it's documentation it's uh support uh just making sure developers can get what they need when they need it um is sort of part of the whole developer experience package absolutely so we're we're almost we're almost at the time so we have uh you know we we have the quote that James was talking about here but you know and and I think that this this describes well what we're talking about if we're asking developers to be increasingly responsible for building secure apps we have to make it as frictionless as possible for them to do so we need platforms and software baked in security defaults we need to embed uh principles of lease privileges guard rails and uh I mean you could you could read it and I love this and the reason why I love this is because I don't know why but you know it's it's slowed down a little bit but when you look at vulnerabilities like we we just got bombarded with you know companies having data breaches because they had misconfigured Amazon S3 buckets it would just seem to be everywhere why is it so easy to misconfigure an S3 bucket it should be difficult it should be difficult to make something insecure and it you know any and I think that's what we're uh what we're all talking about but we do have to uh we do have the the to wrap up um a little bit now because we're running out of time and I know people have anything but um you know I just want like to to to put some final words in there because again on the moderator I can I I have the power so I can plug what I want uh but you know really consider investing in your your your secrets uh programs it's not just Secrets management it also needs to have factors into identifying when sequence leak it needs to have factors in how you deal and rotate Secrets uh with this what you actually what you do and then an analytics to see where where they're coming from so I mean there's lots of tools available I won't talk about keep Guardian because obviously I'm biased but it is the best and uh that's what Andre said you know and he's clearly very smart so so so definitely uh uh you know try and check that out uh but I would just like to say uh thank you uh to all of the to all of the guests that we had in here Andre James and Mike it's been a really fantastic discussion I've really enjoyed it um and uh so thank you all I'm sure everyone else enjoyed it we had a lot of people in here uh we had over 30 questions invite the guardian team's been answering them behind the scenes so lots of activity so I think that's a good indication of what you guys were saying was really resonating with people so thank you all uh for joining us and uh we'll let you know who the winners are of the prizes um and with that I wish you all a good night