The Main Thread

Episode 09 Transcript

Is your platform secure.
How do you know,
come to think of it?
How do you even know all the machines,
networks,
software and data that make up your platform in the first place?
How do you go about answering these questions?
Let's dive into how a staff engineer can leverage their relationships across teams A I tools and a complete asset inventory to keep their company systems locked tight against external threats.
Today's guest,
Amir Ali Shahid Pour is head of security data platform at Twitch and he'll walk us through what he's learned after 17 years of industry experience.
Let's get into it.
Hi,
everybody and welcome to the main thread.
My name is Alex Geyser and I'm here with my co hosts,
Ryan Ogilvie and we're here with Amir Shain,
head of data security platform at Twitch.
Hi,
as you mentioned,
my name is Amir Ali Shain Pour.
I go by Amir,
I started my career in industry in 2006 as a tech support in a software development company.
I taught myself a little bit of programming and SQL and eventually I transitioned to development department of the same company I worked there for about four or five years.
And that was the time that I knew I wanted to pursue my career and get the CS degree.
And I came to the United States,
I started to go to school and eventually got my CS degree from UC Santa Cruz.
I also,
at the same time,
I worked as a foolish like developer for about like four or five years.
And after I got my CS degree,
I started to work in Silicon Valley mainly focus around data.
I,
I've been working as data engineer in data platform teams,
data infrastructure engineer and things like that because I'm like very uh fascinated by data and what you can do with it.
And currently what I do,
I run a team named security data platform uh within the security organization to each that we provide data and analytics capabilities uh for security organization.
Cool,
why don't you tell us a little bit about how you got into the security side of things?
Not everybody we've talked to has made the transition from full stack developer over onto um security.
I mean,
sometimes people go the other way,
but I've always been interested in security and I'm curious like,
how do you get into that field?
Yeah,
I'm also just really impressed I just from from tech support in 2006 to being the head of a very important team at a large company uh is an impressive trajectory.
So that's pretty cool.
And I'd love to know to Alex's point,
what kind of things you pitched your star to basically that you were you were pursuing as you went along that journey and how you got to,
to being a security expert.
Yeah,
definitely.
So first of all,
I don't think that I would say I am the security expert.
I am like more expert on the data side and like right now what I do,
I provide like data and analytics capability that supports security.
So I'm getting more and more familiar with the security side.
Uh But I always been interested in security and I had a friend who used to work in Tu each and he referred me basically to tuition and he told me that there is like a uh open position uh for the South for engineer actually in the one of the security team within to each and I started there,
I moved to,
to each for ALA which was a data catalog company and that's how I got into the security.
And after that,
I transitioned from being IC to the management and I formed this team within the security organization because I saw that this is a need,
like data is something that is like very relevant to not only security,
obviously a lot of different parts of the business but also for security.
And that's how I got into security.
So would you say that there's much difference in the approach?
Actually there is a lot of common things that it is still there regardless of like what you're supporting,
is it like security or any other things?
But also security has its own parts that like,
I had to like,
learn a lot and like,
better understand what are these things in security and how you can support them and like have a mental model that you can like relate that.
Oh,
maybe we can use this thing to like support a specific thing.
So it's a lot of common things etl in the data collection of the data aggregating the data,
analyzing the data,
that type of things are there.
So you mentioned learning and I'm curious,
the security space evolves so quickly.
It seems like every day there is a new attack vector that becomes public and how do you go about keeping yourself and your systems up to date on what is out there,
what the threats are and what you need to be making sure that you're securing against.
I try my best to read different blogs,
uh talk to people in the industry.
I have friends that they are in the security and I talk to them a lot to learn from them and understand what's going on in their organization and uh basically learning from other people because as you mentioned is like very fast paced and evolving space.
Yeah.
So I'm curious like when you wanna push up for security priority,
so you find something that is a serious critical change,
but it,
and it affects the whole company.
How do you go about doing that,
making,
making other,
all the other teams adopt.
So that is a critical change to do that.
I think the first thing that when you want to touch something that's like going to affect the entire company,
especially in bigger companies.
The first thing is like communication and getting alignment.
I think this is very important to communicate with leadership,
engage the team lead P MST P MS,
that type of thing.
And then I think it's very important to understand what is the other team's priority and what is their product?
What's their value prop and that type of thing.
Because one of the thing is like,
you need to highlight why it's important for them to do so correct and then understanding their product and their value prop.
It allows you to like reframe this importance that is like basically the in the context of their product.
And also I think as far as like telling other people like other leaders that why this this is important to do something,
maybe it's like we can uh quantify the risk,
maybe it's important to maintain the trust with customers and like that type of thing,
upholding the brand reputation.
They have to know why this is important and why they have to work on something.
And then I think being data like use the data from the past to,
to be able to say that OK,
this is like something that in the past it's very similar to this thing,
we didn't do it.
And this is the consequence for that that allows us to like really communicate better and get a better buying from the leadership.
And then also,
I think when it comes to planning and prioritization,
it's good to have the data by having a good data and understanding of the organizations and business units.
I think we can add options for other teams that hey,
we think you have to do something but because we understand your infrastructure or business very well,
we think that it's ok to do something else.
Like that's what I mean by add options.
If you do this,
we,
we buy some time basically.
And finally,
I think it's about providing tools and resources to other teams.
When we,
we are asking them to do something,
it's important to have impact with other teams as a security or we should try our best to automate as much as possible before we go to them and say,
hey,
we did like 50 60% of the work and this is like some amount of the work that is like left over for you to it.
And again,
by understanding the business very well,
maybe we can shift resources around and say,
OK,
you guys,
you can't do this right now.
There is another or we can have,
we can like help you.
And by,
I feel,
I feel by following these type of things,
it's easier for us to like get things to other people,
road maps and get things done,
which is a priority for security.
That makes sense.
I was um I,
I was thinking that as you talked that,
you know,
getting ahead of it,
doing a lot of the work yourself for as much as you can before you approach a team is so important.
And sometimes that work that doing the work that you need to do involves sometimes making unpopular decisions,
like deciding that you can't simply cover the entire universe of packages or processes that might be out there.
And the in order for you to do your work to say,
hey,
we're almost there,
we just need you to do this last bit is to say,
you know what,
we're actually no longer supporting this way of doing things.
So we're not going to support this particular third party platform or,
or,
or whatever it is.
You have to say,
listen,
we,
we are going to do the work to make sure that we understand how to secure a small number of things.
And then you have to go to teams that are using those other things that they love and say,
listen,
I know you like this,
but we do not as a company have the resources to support that and this other thing and the tough decisions been made,
it's this other thing,
how do you go about having those conversations when the work that you did means to basically take some options away from people?
Yeah.
Yeah,
that's actually a tough conversation for sure.
And these are the type of conversation that you get pushed back and it's tough to like,
really convince people.
But 11 thing is like,
again,
I guess like having empathy,
really understanding that like,
ok,
if they are having push back,
why is that?
And providing alternatives?
Like basically we need to say,
OK,
you cannot use something but we thought about it.
There are like alternatives that like we can help you to like shift from using this to the other things as opposed to saying that OK,
you should not like use this at all.
And we are not going to do anything like we need to like understand other people,
help them,
do our due diligence and like find alternative and hopefully maybe even like help them automate things if it's possible to like replace things and that I think will be helpful.
I think that's a good point,
especially what you were saying earlier about II I think of it as like showing your work in terms of like demonstrating how much you've already done that.
You're not just like mandating that another team does something without um giving any consideration on your part as to what they are required to do.
But then also demonstrating that you have done your due diligence and that this is truly important that,
oh look,
we've actually done,
like you said,
50% of the work we've um demonstrated that we're actually got skin in the game too that we're not just asking you to do something.
And I feel like security especially has the because it's,
they have to interact with so many different teams.
There's that tendency to just say like this is what you need to do and do it.
And as,
and the other teams don't always get to see like how much work went into that decision to deprecate something or to like or to upgrade something.
And that's something that I never thought about before.
I think that's really,
really a good way to get buy in from other teams to demonstrate that you,
you know,
that you did a lot of work before you ask them to do something.
Yeah,
I think that definitely will help.
So when you're doing your work,
because you're obviously working with data and getting a analyzing data,
you yourself actually have a product that you provide for the rest of the company.
And how important is it for you to plan ahead for scale and the growth of the company so that you can continue having an efficient product that provides the information that they need for security.
Very important.
Actually,
I think uh like one of the product is that support a lot of things and securities is like having the reliable asset inventory.
Did you want to have this and make sure that this is up to date and reliable and trustworthy that at any time in security it might be,
you need to use this inventory,
so have to plan for it,
make sure that it is up and running.
It doesn't have like bog and you basically anticipate everything and making sure that you can use it at any time and scale it with the company.
And so from where you sit,
how do you know that what teams are building and what they're using to build it that it's in the inventory?
How do you know that,
that it's that there's compliance.
How do you track that?
So I think it's with a lot of automation and having the uh process at the very early on that,
like when you are in your organization,
you're trying to build a new service at that time,
you need to have like a understanding and visibility that like OK,
something new is coming,
pull the data for from that time in inventory.
What's the infrastructure that they are using?
What's the software that they are using?
What versions they are using?
Like everything?
What's the relationship between that service and other thing?
And uh basically having the extra context about each of them?
Like what is the purpose of this service?
Does it have customer data or not?
Like that type of thing is very important and it's like you need to have processes in place that like,
hey,
there is a new design review coming up.
That means that like there is someone is thinking about like bringing up a new service.
So depending on the maturity level of the company,
it's difficult for uh all the company to have like in this level of maturity.
But like if you can like put the processes in everywhere,
then you are able to get all the data for everything that comes within the organization.
Yeah,
it's always tough.
I think for a company to decide when is the right time to put that level of process in place.
Obviously,
if you're at a start up and you put that level of process in place,
there's not enough people to build the purpose of the company's existence.
But if you let it go on too long and you get so large,
it is incredibly difficult to rein in everything that's going on and get it back under a manageable state.
Absolutely.
You're right.
And I guess like,
again,
I'm a fan of automation,
like really have uh automation in place to like scrape your environment.
Like OK,
we find a new,
let's say account,
we find a new service,
we find a new end point and that type of thing and then that's like going,
going to trigger something that like it's either other jobs to go and scrape more detail and get like more data about something or if it's like a smaller company,
maybe,
like,
just like,
ok,
now we understand that something is happening,
let us like,
go do some manual work to get something.
But uh it's about like really having a good asset inventory that is like,
hopefully is like complete and up to date.
So I guess speaking of process and speaking of good systems to be put in place,
what sort of systems do need to be in place to handle a crisis when it comes up?
Like we've talked a lot about the kind of the day to day.
But what about the the unusual event where log for is the one that comes to mind?
But there's a million examples.
Yeah,
absolutely.
This is a good question.
And I think there would be a framework that I think it would be good to have in place to be able to be basically prepared to address this type of uh vulnerabilities like like L for sure.
So I think there are like five pillars to this framework.
The first one is prepare,
the second one is identify,
the third one is analyze,
fourth one is communicate and then finally threat you need to prepare,
which means that you need to have policies and uh basically training,
set the expectation and everything should be communicated with the entire company.
So you prepare people that's number one.
And then as part of this prepare,
you need to have that completely trustworthy and up to date asset inventory,
you need to know everything about like what do you have in your organization?
Because then when we go to the second pillar,
which is to identify,
you cannot like really understand what's the impact of that vulnerability to your organization if you're,
you don't have a good inventory.
So that is like very important.
And then in the identify you first,
you want to hopefully very soon understand that.
OK,
there is a vulnerability coming which is like basically threat.
You need to make sure that you have a donation and processes in place that you,
you can like identify that.
OK?
Something is like out there,
let me identify the affected system right away.
And that is like where your asset inventory comes in place.
And actually here,
I think it's important to know that it could be the case that like your environment,
your organization is not vulnerable to this vulnerability that is there because you just don't have anything that is affecting.
And for you,
that's the end of the story.
But let's say if it is the case and you have affected system,
then you want to analyze uh that exploit and like really understand,
OK,
how does it work?
Is it like any uh conditions there that is like I have to be aware of because that type of thing is going to allow you to prioritize.
And again,
this is going back to that having like that uh inventory with the extra context that hey,
these are the all the systems that they are affected.
These are the one that they are like tier zero services.
These are the one that they have customer data,
that type of extra context,
it allows you to like prioritize correctly and then you want to communicate you want.
That's the time that you,
you start cutting the tickets for owners,
you create dashboards that you want to basically show that.
Ok,
this is like the impact that services.
This is how the be done is working.
And finally three,
which means that owners start to really act on things and remove this vulnerability.
And I think there is another thing that is not part of the framework per se but having the post incident analysis,
it's extremely important like it's not only for security,
any incident that happened.
I I think we should always review the process that we have find area for enhancement uh lesson learned action items and that type of thing I feel by having this type of like rigor and like system in place,
we can just like improve and basically response and address any vulnerability.
I have to chime in there for the the uh post incident analysis.
I feel it is so important for companies to treat these as blameless post mortems because if people are defensive,
when they show up for these things,
the opportunity to learn and to improve is just cut off at the knees.
And I think there are a lot of large companies who famously use blameless postmortems to great effect.
And a lot of other large companies who take the other approach.
And I think it is a real hindrance to a culture of ownership at the company.
People don't want to be responsible for things if they're gonna get yelled at,
if something happens.
Yep.
It's what we've talked about 100 times on this podcast already.
Trust is the most important thing.
Building trust.
It's like you talked about building trust with people so that they don't think that you're just without cause adding extra work and,
and trust so that when something does go wrong,
people trust that they're not gonna get blamed for what could be an innocent mistake or even just a regular old mistake because otherwise you can never know what led up to it because it's gonna be too busy covering their tracks,
figuring out how to shift the blame to somebody else.
It's,
I mean,
it's a cliche but it's,
but it's a cliche for a reason.
It's because I think there's truth in it.
I want to talk a little bit about the analyze pillar that you mentioned.
We have now more than ever before in history.
Access to a tremendous amount of A I and machine learning platforms that can crunch a whole lot of data and give us a lot of valuable insights into that large amount of data,
but they may or may not entirely be trustworthy or reliable.
So how do you balance the amount of ML that you use to analyze large amounts of data versus manually going through and talking with teams?
And can you leverage both those things when you're analyzing?
Yeah,
I definitely think it,
you need like a hybrid model that this should be automated and you use ML and that type of systems as well as like the manual check of like these exploits,
like doing research and things like that to really do this analyze and they are,
they should both be in place.
I don't think at least I know a company that like we can say that OK,
they just do this by ML and they are 100% confident.
So we are getting better and better,
but the manual process should be there.
Yeah,
I mean,
I agree.
What is something that you have seen in all of your experience that successful organizations do in regards to security?
Um And then a follow up to that is what's something that unsuccessful uh organizations do poorly.
Maybe this is like uh because I'm biased to like having the data but feel the companies that they don't invest on data collection and like being data driven.
Those are the ones that they are like in problem when something happens and they cannot like really even say,
what do we have,
what system are affected and things like that?
That is like issue that I,
I have seen that and I heard about their companies like that and that's like the biggest problem.
I think the companies that I've seen that they are like mature and they are uh they can handle things very well.
Those are the one that they are investing in that type of thing,
making,
making sure that they have like a very good asset management programs and asset inventory and they can like leverage data for their day to day operations,
prioritization,
making decisions like uh things like that,
I guess related to that is as your career has progressed,
how is your understanding of like security's role and importance and how to do all of these things that we've talked about?
How has that changed as you're,
you've gone from,
you know,
starting out,
you know,
in 2006 all the way till now,
like,
how,
how has that changed it?
It definitely changed a lot.
And I feel like school site,
that's I whatever I learned in a school,
like really working in different companies and different products and things like that.
That was like something that really helped me grow.
And I really learned as I moved from company to company by looking at like what software they are using,
how they do the build process,
how they ship their product and how they basically collaborate with each other.
A lot of different things that like,
I feel like this helped me grow and there are a lot of things that I learned and I'm still like growing and learning and yeah,
it's been,
it's been a lot awesome.
I think that's a great time to move ahead for picks and plugs.
So this is the time in the show where we just take a few minutes to tell our listeners anything about things that we're interested in,
things that we're reading lately.
If it's tech related,
great or if it's a TV show you like whatever,
uh it's anything you want.
Um Do you want to start Lair?
Sure.
Yeah,
absolutely.
So one thing I'm always like,
really into like,
uh business and like,
I want to see like how uh people start business and how the start ups are emerging and becoming successful,
why some of them they are not and things like that.
I'm recently reading a book named The Art of Start 2.0 by,
by Kawasaki.
And this is about like,
basically,
uh how to start a business,
how to avoid the pitfall,
uh and starting a company who to work with and hire and how to pitch and things like that,
that I think this is a very good company.
And also it has like some story about like why this company failed,
for example,
which I think that is also important to know that like,
not only like what is good and what it works,
but also what is not working and,
like,
try to not do the same thing.
Is that something you're interested in someday in your career,
in an indeterminate future,
starting a business?
Yeah.
Absolutely.
Absolutely.
Awesome.
All right,
Alex,
what you got?
I've got,
I'm,
I've restarted the Mythical Man Month Management book.
I started a while ago.
Uh,
and then I gave up on it because I didn't,
I don't know,
I found some things in there that I wasn't,
that didn't think that were that applicable anymore.
But I've started it again,
I think with fresher eyes,
it's given me a lot of things to think about and it's,
at the very least it's a,
uh,
it's a classic.
So I figured I'd work my way through it all the way through and see if my opinion of it changes.
It's a classic for a reason.
So,
and I've been reading a lot of like the classic books recently.
Uh,
and then I have a youtube video that I have not fully watched because it only came out a few hours ago.
But I was,
but I had another one and I like this one so much more.
I kicked the other one out.
It's uh Get Hidden Gems from uh N DC conferences at Copenhagen at F 2023.
It came out just a few hours ago and I watched,
um I was watching it before we started the show and I'd already learned like three or four interesting things about G before I,
even before I,
even before I even started.
So,
like,
just a few minutes in,
I already found it helpful enough to,
to recommend.
So as soon as we're done here,
I'm going to put it on and finish it,
but it's,
it's pretty cool.
Anything that,
like actually can teach you something new about a tool that you use every day is pretty great in my book and get,
is the first thing whenever somebody asks me,
you like,
how do I get into software engineering?
The first thing I say is uh how good are you at?
Get,
you don't show up and not know how to get.
Right.
Exactly.
And our first answer is the answer is usually uh get where,
get what.
And I'm like,
OK,
start there.
That's good.
I like that.
I like that.
I actually,
um I,
I started a course um on Udemy,
I'll have to look at the name of it because I can't remember it right now.
But it was about like advanced command line,
you know,
because I,
where,
where I'm working right now,
so much of the command line has been built into GIS,
you know,
like we have like built in,
you know,
we,
well,
first we use material instead of GIT but,
but it's,
you know,
a lot of it is there's a G I to like rebase all your commits or rest stack them or organize them,
all that kind of stuff.
And it's super fast,
but I feel like my command line skills have gotten a little rusty.
So I wanted to get back in there and get act like my earlier days in my software engineering career where I was a command line with.
So I'll get back in there.
We'll post that too.
It was good.
My pick today.
Uh This actually ties in really well with some of the things that Amir was talking about,
about communicating.
So this is a media article I found by Elizabeth Eyre and it's called,
don't ask forgiveness,
radiate intent.
So it's a lighthearted rebuttal to Grace Hopper's famous quote about it's far better to ask for forgiveness than to ask for permission.
And while Elizabeth understands the spirit in which that was given uh in a,
at a time when a lot of people felt disempowered to make decisions and move ahead with things,
she says it's actually better instead of asking for permission after you do something,
be really,
really vocal and clear that you are about to do something.
You're not asking for permission.
You're just saying,
hey,
this is what I'm doing,
especially if it is potentially risky.
If,
if you are gonna say deprecate a package or delete an entire service that you think no one's using so that you don't have to maintain it anymore.
You want to be very,
very clear that that is coming and give everyone the opportunity to pipe up uh rather than a and this is in context or in contrast to having a design review where you say,
you know,
I think I might want to do this and then people just jump on to it because they think they need to say something.
Instead you've done your design,
you're very clear,
this is what's happening and then that gives people the opportunity if they really see a problem to pipe up and say,
whoa,
whoa,
whoa,
whoa,
whoa.
Before you do that,
we need to think about this.
That's gonna get broken anyway.
It's a great,
it's,
it's a pretty short read,
but I really like the perspective and it's a great,
it was a,
it was a great take on how to communicate effectively within a company when you are going to make a decision that might be either unpopular or worse risky.
So that's mine.
And so that'll wrap us up.
This has been a great conversation.
The me with these always go much faster than we expect them to.
So we really appreciate you being here.
Nice.
Yeah,
same here.
I had a great conversation and thank you guys for having me here.
Yeah,
I,
I'm,
I'm excited to go back to work and tell everyone about our guest and how much he really thinks it's important,
what I work on.
Yeah.
Being in the asset inventory space is pretty good.
All right.
Thanks again for being here.
Really appreciate your time and listeners.
We will catch you on the next one.
Thanks for listening to the main thread.
As always the views expressed by the hosts and guests of the main thread are our own and do not reflect those of our respective places of employment.
Are you enjoying the podcast?
Do you have a topic you'd like to hear,
discussed in a future episode?
Well,
please reach out to us with any suggestions or feedback.
We are at the main thread on all major social media platforms including Facebook,
Twitter,
Instagram,
and of course threads.
If you prefer,
you can send us an email at main thread podcast at gmail dot com.
For links to resources mentioned in today's episode or to download a transcript of the podcast.
Please refer to our show notes that's all for today's episode.
Thanks again for listening and we'll catch you next time on the main thread.