so what what does timing Secrets brawl look like so uh it can be quite challenging so on this the top level you know you have we have two columns here unmanaged Secrets bad manage Secrets good uh so we won't hard code the teachers in clear text we don't rotating Secrets we're using long-lived credentials credentials that live on forever and we you know compare this to the opposite using volts automatically rotating them but it keeps going on um permission Scopes we've got the principle of least privileges making sure that we're not just uh creating admin credentials uh auditing etc etc etc so it's actually quite daunting to do it so how do we go from one side to the other so oh Andre I wanna I wanna start with with you because you've done this uh you've you know perhaps you weren't ever on the the left side but you've you've certainly gotten further and further of right what do you start how do you start dealing with this with this what what are you you know what could you prioritize stuff do you tackle it all at once how do you actually go about solving this issue if someone's listening go okay I have an issue but what do I do how do I start so we we didn't have a a fresh start but we did have a start when my team got formed we were set to design and carry the company into the cloud and then we got a chance to look at what our company was already doing what besser was doing for secrets and we decided that's not gonna cut it that's going to be useless for us it's not automatable it's not it's not what we need so we had the courage and we pushed a lot actually and we ended up succeeding and we deployed our own secret server where you're using a vault from hashicorp and then we had a long process of building a structure for the teams and environments and then defining how that should work and teaching everybody how to use it and what we expect from them and then also leverage this technology to do automated credentials Dynamic Secrets integrating with worker with identity you know secret less workloads and all that kind of stuff and it's still a very much ongoing process because the things we did in the beginning now they have really good Alternatives they are even better and we learn as we go as well so after that after we were happy and we were kind of in a good place we we looked at how do we clean up what we already have especially the ton of Legacy code that we brought in from the old Erp systems the old side projects the old consultancy that uh solutions that uh you know we purchased and we saw a lot of problems in there and we looked at the git Guardian and I can go into why but we thought it was the best one and we used that to see what problems we had afterwards uh remediate them one by one and then block Right Use the technology to make sure Secrets don't end up in our repositories and that people use the right uh the right solution that we made for them yeah that's kind of the story the one thing that you said there which is interesting is you said courage I I feel like you know like that's very true it does take courage to be able to go on to this about because it's a it's a big journey and one that certainly comes with a lot of challenges I I want to put this over to you Mike uh because it's someone that creates some of the tools for this um you know like where would you recommend people to start uh in starting to to deal with this project you know is it perhaps once they've rolled out one password or they've rolled out a sequence manager you know what's what's kind of the next steps what's the first steps and what can people do what would you recommend for people uh that you're interacting with yeah I mean from a password manager perspective I think what we're trying to do is just educate users on the use cases Beyond um sort of what they intuitively think a password manager is for which is just you know username and passwords and maybe uh totp tokens um so I think it's really our role to try to educate users on all of the other Secrets they're touching day to day that they might not even be thinking about and trying to encourage them uh to use those and then make the the process as seamless as possible that again we can start moving up the sdlc to start you know going from the individual developer and and their workstation uh to pipelines to um actual infrastructure um you know one way we did this recently was with a SSH key agent um so most developers don't really think about SSH keys they they have this private key sitting on on their machine that they create you know maybe once every time they get a new laptop and just leave it there and nobody thinks twice about it but once we built an SSH agent into one password so you could sign git commits with your fingerprint um it created such a frictionless uh solution that people started adopting it really quickly and now they don't want to go back to having to create SSH keys on the command line or manage them so I think I think by introducing features um you know and identifying things that um like we have ways of of prompting a user like hey this looks like a pass a sensitive credential you should store this in one password by doing that and sort of getting into their workflows you sort of chip away at it and move it up and I think the other aspect of that is you know not just starting at the developer but starting at the organization and educating the organization about the tools that are available and um you know what what they can do to help increase their security posture definitely it's very very Sound Advice James do you have anything to kind of add to what you're seeing in the industry of you know what would you recommend to people if someone coming to you and asking how do we how do we solve this problem where do we start yeah I mean I think that one of the things is obviously you want a good conversation between uh developers and um you know platform teams developers and security teams um and you know that's always it's always easier said than done um but a Frank conversation about Improvement and this may be a trigger on um you know people are a lot of organizations are currently modernizing their pipelines and thinking about what that should look like so that's a good time to have a discussion about hang on a second okay um we are going to you know upgrade from our 20 year old Jenkins systems and um sort of thinking about uh what what that will look like so while we're going through that process bring the developers in and ask them what they would like to use you know developers know about one password they're good examples Hey you know the the kind of experiences you had there could be valuable here are there any tools that you would like to see there well let's have a Frank discussion about this kubernetes roll out because frankly um the the default privileges are totally absurdly open we're going to need Secrets if we're going to have these micro services so let's have that conversation together and I mean that's I guess that's I don't know that sounds a bit sort of um uh a little bit of apple pie but I do think a Frank conversation between the different constituencies is going to be helpful it's not that developers don't care um it just hasn't necessarily been something that they have been thinking about so involve both or or all three of those platform security and developers in a conversation about moving forward and making secrets for all in Secrets management a problem that the organization is going to fix together because I think that's how we've solved um well that's how we've got better at some of these things testing itself you know the shift testing left um you know these are things where you provide better tools um but also you listen to developers and you bring them closer to the platform teams it made me feel quite old there because you're like oh my 20 year old Jenkin thinks I remember seeing it feels like yesterday someone showed me Jenkins for the first time it was the most wild thing I've ever seen in my life so you could do what what automatically obviously now it's very old but I'm old enough that I remember talking to Kazuki when he just built it it was running on a laptop under his desk at Sun Microsystems it was called Hudson yeah I uh we're running out of time and I know I'm not going to be able to get through everything but I think that's okay because the conversation's great and is there there's a question that's coming to the chat um that that I think are really interesting and we can talk about this it's from a lot of messages of 35 first name or last name I'm not sure the secrets and basically the question is are secrets only the responsibility of the security team how can Engineers actually help um so I I just thought maybe we quickly spend some time on this at an appropriate time um Andre you know how how can developers help the engineering team you're part of the Ops Team what would you or the devops team what would you kind of really like to see some of the developers or or take on to to help this yeah all right yeah there we go um I was gonna I was saying that that is I think uh extremely hard and it's getting harder and harder because the cognitive load on developers is growing extremely uh and has been for the last few years especially since getting into the cloud all these tools all these Solutions all these pipelines how they all chain together what are they supposed to be used for it's it's very difficult um so I don't blame developers so much but there needs to be at the end of the day a simple you know Common Sense stuff that we have seen is if you commit committed your password if it's in git then you should considerately it's not enough that you amended your commit or changed the history you know report that to the security or um ask if you don't know right just say hey uh a devops team how do I do I have this scenario this crazy scenario I don't know how to do it can you guys help me out right that's what we're here for to to help the Developers [Music] foreign