Lessons from a Decade at Amazon

I worked at Amazon, but this is my own opinion.

I learned more in one year at Amazon than I did in my seven years at Novell. A lot of what I learned in the last ten years at Amazon is weird esoteric stuff about Linux distributions, AWS services, open source licenses, and a bit of law. But I also learn some life (or at least work) lessons.

Dig in Before You Ask

You don't know everything you need to know to do your job. You can ask a colleague and they'll probably be able to tell you, but instead of knowing how to find the answer, you'll only know the answer. Before you ask a colleague, consider if you might be able to answer your question yourself in about the same time as it will take you to ask, wait for them to make time to help you, and find the answer themselves (because they don't know either, but they know where to look). Every time I stop to at least try to find answers on my own I'm glad I did.

There Are Things You Don't Know

Even when you're good at seeing the big picture you won't have all the information. Conversations are happening above and to the side that you aren't part of. Things you find in legacy systems and think "What the actual heck?" (shock-value profanity is lowbrow and folksy language is charming) - those things often have astoundingly good backstories. Sometimes they don't, but it's worth uncovering the story before you burn the whole thing to the ground. So often someone comes to you with something that you think definitely means they're an absolute moron but when you dig just a bit deeper it turns out the problem is complex and nuanced and actually they're doing an excellent job of managing that complexity.

There's also a fruitful side-benefit of working to disconfirm your beliefs: If you lead by asking questions instead of by giving answers you'll help everyone arrive at the answer you already have in mind with a sense of ownership and new understanding that will help them be right without asking you next time. Unless you're wrong.

Always Be Teaching

Automating everything isn't the only way to scale. Another way is to teach everyone else to do your job (especially if your job is to automate everything). I don't know what "force multiply" is supposed to mean (I know what force is… I know how to multiply…), but I suspect it's somewhere in this direction. Give a man a fish, yada yada. The point is that every time you give someone the answer they need, if you can give them some of the background on why, or how you found that answer, or some insight into how you think about it, you forestall their asking you the same question again later. Could be because they hate listening to your longwinded answers, but at least they went away with more than they bargained for.

Jokes aside, what I've found trying to be a teacher at Amazon is that people become interested in the weird things you care about, because actually it is interesting but they didn't know that until you showed them.

Do What Is Right

A powerful old hymn has the chorus, "Do what is right; let the consequence follow." One of my core beliefs is that outcomes do not change the rightness or wrongness of actions - A choice is right if you chose the right based on the best information you had at the time, regardless of how it panned out for you or anyone else in the long run. What's best for you isn't always what's right! You are welcome to believe differently of course.

Many times I was faced with a choice between doing what was best for my customers and doing what was best for my career. You'd think those should line up, but the simple fact is you're not going to get promoted for fighting fires, and if there are fires that need fighting that usually means your customers are suffering in some way. The right choice for me was always to do the right thing for my customers or for my team even if that work wouldn't help me get promoted.

Similarly, I am almost always faced with the choice between what is best for my career and what is best for my family. I will do what's best for my customers right up until we flip from "my job supports my family" to "my family supports my job" and then I'm out. Every time I get a new manager (except once, see People Actually Cry at Their Desks?) I make sure they know how firmly I will hold that line, see Own Your Own Destiny.

Don't Blame People, Blame Systems

Something I've loved about Amazon is our COE process (correction of errors). It's a post-mortem, not unique to Amazon, where we try to dig down to what really went wrong that lead to a customer-impacting incident. What I love about this process is that it doesn't stop at "so-and-so did something dumb" - it blazes past that to "why could they do that?" The reason is that "you did something dumb" is about good intentions, and we know that mechanisms are what make a good system. Hope is not a strategy.

Okay, so that's a lovely idea, but why love it? Because it means that as long as I don't do something terrible on purpose, I'm not going to get fired for my mistakes. I've sometimes heard people joke that you have to cause a Sev-1 (highest severity issue - large customer impact) to get promoted. That's obviously not true but it speaks both to the idea that you're not getting fired for a terrible mistake, and to the opportunity to shine by owning up to your mistakes and figuring out how to change the system so it can't happen again.

A Career Is a Series of Jobs

I learned this from a college professor when I asked if I should take the job as a SysAdmin for a small airplane parts manufacturer that was willing to pay me what I wanted to be paid, or take the job at Novell as a Build Engineer being paid less than I wanted to do something that wasn't quite what I had in mind at the time. His point was that taking the job at Novell was likely to lead to more opportunities than the job at Cogswell Cogs. Novell might be a step to something else, and it was. Not only that but the SysAdmin gig wasn't going to ever pay me more unless I went into management, but Novell had an engineering track that went a little higher and Amazon has an engineering track that goes all the way up to VP and beyond with zero people management along the way.

Lateral moves at Amazon are easy. Tired of distributed systems? Go grab a job building drones, or enabling EC2 to deploy Mac Minis. Whatever suits your fancy in the tech industry, we're probably doing that and probably hiring. Be sure to ask what their ticket load is - I once investigated a team who got tens of Sev-2 (high severity - work until it's fixed) tickets a day, many of which were not actionable. I'm sure they're in a better place by now.

Own Your Own Destiny

Think of Amazon not as one huge company, but as a bunch of startups all under one VC. Of course there are ways in which Amazon really does behave like one huge company, but the notion that it's not helps you get comfortable with some of our peculiarities. Part of the philosophy behind this is that every org needs to fully own their own destiny. We don't blame someone else for our failures. If Amazon WorkSpaces has a problem that's rooted in Amazon EC2, we don't sit back and say "well, that's EC2's problem" and we definitely don't tell our customers "hey, it's EC2's problem." Instead we own it fully - we find a way to make WorkSpaces work well regardless of existing issues in EC2. We recognize that even if a specific EC2 issue is a big deal for us, it could rank lower for EC2 (or lower for now) compared to some other pain their other customers are seeing. We strive to help them solve our issue while also solving our own.

Now extend this to yourself. It matters that Amazon is a huge company with culture and policies that impact your work-life harmony, but your power to massively change culture and policy is limited. So instead of railing against the machine you recognize that you own you, fully.

Every once in a while you need to do the right thing for your customers and stay late to resolve a Sev-2, but unless your manager is completely broken you'll get comped that time and you can average about 40 hours per week. There's no mechanism at Amazon to help you here. There's no Leadership Principle for Nurtures Work-Life Harmony - there isn't even one for Manages Expectations. The closest you'll get is Frugality but you have to squint or stand on your head to see it. But because you own you (Ownership is an LP!), you can manage expectations and it's critical that you do so. Burnout is real and it hurts. I've burned out at least 3 times in my decade at Amazon. I've even cried at my desk a few times (but did not disclose this to the New York Times).

Put Your Pain Where Your Money Is

There will always be a question of who should own something. My favorite way to think about this problem is that the team that should own solving a problem is the team that's going to feel the pain if the problem isn't solved. At Amazon the team that builds a service also owns keeping the service running and fixing bugs. Most developers are part of on-call rotations. This aligns ownership and pain so that problems that need to get fixed get fixed by the people most able to fix them. The code you wrote on Monday and deployed to production starting on Wednesday gets you paged on Thursday and you roll back so that you can spend Friday fixing it and then have a weekend (best case).

This breaks down when your customer are feeling pain and they don't have a way to communicate that pain to you. It also breaks down if your customer service team has a great workaround for the problem and isn't telling you how many calls they're getting about it.

But there was something I wanted to say here about pain and money and I've lost that thought. TODO

FAQ (Fortunately Anticipated Questions)

Novell still exists?

Yes, and Netware is still a (I assume shrinking) cash cow. At the time I was with them they were focused on SUSE Linux, various directory-related products, and a variety of other things. We all talked about when we get laid off, not if, and people saved their vacation time to maximize their severance. Interns had private offices in the Provo campus and entry-level engineers like me had window offices (regrettably south-facing). Whole building were vacant and parking was plentiful. They got bought by a company that makes a terminal emulator, you know, for airlines or whatever, and the entire Mono Project was let go because the executives didn't believe Mobile was really going anywhere.

It was a good job, and it lead to better things. See A Career Is a Series of Jobs.

Wait. Why have you stayed at Amazon if you've burned out three times?

Because the grass is not greener on the other side. My tolerance for risk is quite low. I have no reason to suspect that working for some other huge company or for some little startup is going to be better for me, and working for a mid-size company would lack much of what I love about my work. Also, I tried twice, but the first time something way more important came up, and the second time a trusted friend was starting to build an interesting product here and asked me to help.

Whoa! People actually cry at their desks there? That's monstrous!

Yes, and sorta yes, but I don't see that as Amazon's fault (at least not primarily). Remember this is a huge company. Even just statistically you're guaranteed to find the occasional terrible manager or the occasional maddeningly conflicting objectives. The latter especially if you really do obsess over customers.

My first time was when I had a meeting with a team in Cape Town about an issue important to my team, and I was blindsided by my Principal totally not having my back. See There Are Things You Don't Know.

My second time was when my weirdly and temporarily broken management chain was laser focused on hitting a goal to release something by re:Invent that we were definitely not going to hit without incurring massive amounts of technical debt. I had backbone. I disagreed. I refused to commit and found a different job (see Why have you stayed at Amazon?). The team both missed their goal by several months and incurred more technical debt than planned. No customers were harmed - these things happen. The worst of that management chain "left Amazon to pursue other opportunities" and others moved on to other things.

My third time was when I was pulled onto a problem on another team and found that I was pretty sure they were going about it completely the wrong way, and got pulled into an emergency meeting with the GM and other leaders to justify my assertion that they should miss their deadlines and re-architect that piece. I was told by one of the directors that everything I thought I knew they had considered months ago and I was relatively new to the problem. It turns out he was trying to give me an opportunity to save face by saying "yes, you're right - I'm new to this problem and don't know all the details" but I felt like I was being told I hadn't done my homework and was embarrassing myself in front of everyone who had a say in when I next got promoted (sidenote: I didn't get promoted, but not because of that). See There Are Things You Don't Know again.

How can you have a family and work at Amazon?

If you don't find ways to draw some boundaries for yourself, and find the right work-life harmony, no one's gonna do it for you. (CEO Andy Jassy, in the April 2022 company all-hands)

"no one's going to do it for you."

"you could work 20 hours a day if you wanted to, but that's not sustainable."

"it's very possible here at Amazon".

"You can find ways to get that harmony, if you take those measures yourself."

Yes, you can post about your own experiences as an Amazon employee. Please remember not to include confidential information.