Software Engineering at Google: Lessons Learned from Programming Over Time

by Titus Winters, Tom Manshreck, Hyrum Wright

Today, software engineers need to know not only how to program effectively but also how to develop proper engineering practices to make their codebase sustainable and healthy. This book emphasizes this difference between programming and software engineering.

How can software engineers manage a living codebase that evolves and responds to changing requirements and demands over the length of its life? Based on their experience at Google, software engineers Titus Winters and Hyrum Wright, along with technical writer Tom Manshreck, present a candid and insightful look at how some of the world's leading practitioners construct and maintain software. This book covers Google's unique engineering culture, processes, and tools and how these aspects contribute to the effectiveness of an engineering organization.

You'll explore three fundamental principles that software organizations should keep in mind when designing, architecting, writing, and maintaining code:


How time affects the sustainability of software and how to make your code resilient over time
How scale affects the viability of software practices within an engineering organization
What trade-offs a typical engineer needs to make when evaluating design and development decisions

Genres: Programming, Technology, Software, Computer Science, Technical, Engineering, Nonfiction

602 pages, ebook

First published January 1, 2020

Reviews

[Sebastian Gebski · Rating 5 out of 5]

I liked it a lot.

If you take a look at the TOC, you may get the impression that this book is some sort of manual to software engineering. It is not. Yes covers all the most important aspects of SE, but it doesn't present you all the basic knowledge from the group up - what it does is presenting Google's PoV & unique challenges and how this organization has solved them.

This has quite significant implications - their problems may be very far from your problems. Their solutions may not only be unreasonable in your case but even not doable at all. But this doesn't make the book obsolete or uninteresting - quite the contrary.

The value of SE@G is in:
* presenting what the engineering (problem solver's) mindset is about
* revealing some interesting details - how do companies of such scale do deal with growth & inertia
* showing that you should approach problems with pragmatism, not zealotry (because the real-life is all about trade-offs & picking the battles wisely)

I've really enjoyed the read. The further, the better. Maybe it was not super-revealing, but I LOVE sharing warstory-like experience & that's what this book has provided.

[Angad Nadkarni · Rating 3 out of 5]

While this book provides a very interesting insight into the problems faced at Google-scale in an organized manner never attempted before, I choose to rate it as a 3 because it isn't (in my opinion) indispensable.

The first half is rather weak, not for lack of appreciation for the managerial skill that building good software takes, but because the book provides the same old cliches that most in software would have come across without providing substantially more motivation. Most of the takeaways are a few bullet-points at most.

My other problem with the book is its (self-announced) myopia to Google. To most novice engineers who haven't themselves run into some of the problems described in the book, the motivations will continue to remain unclear and solutions further obfuscated within the Google-context/parlance. It is entirely possible I was reading it too lazily, but I personally find a barrage of case-studies of unrelated products rather tiring, especially when the content isn't organized in a way where the takeaways are clear.

The second half picks up pace well as it is technical, and clearly closer to the author's day at Google than the former. I found some of the paradigms employed at Google very interesting as to stumble upon them online in one singular place is unlikely, so this book is your best bet.

In summary:

> If in your journey as a software engineer, you are yet to acknowledge software as a living, breathing thing, you will find little to no value for yourself in this book.

> If you are at a point where you are seeing some scaling issues with your team or engineering process, this book is a must-read because it can provide for a few new paradigms.

> If you are in a large traditional business, you will continue to remain in awe of Google and leave this book without too many actionables.

> If you are in a large tech business, you likely don't need this book because your bootcamp covers it; however, it'll be a fun read!

Think of it like a very interesting conversation for hours with a Google engineer. He's smart and really experienced, but also incapable of any imagination outside of the Google box.

[Ioana Lily Balas · Rating 4 out of 5]

Google is probably the most representative company in software engineering today, and also one of the most stable ones when it comes to organisations that recognise and respect the craft. They've been going steadily for over 20 years, and have been pioneers when it comes to strategies on how to build programmes sustainably and reliably. Naturally, I was keen to learn from their experiences as well, and here are some things I picked up:

- Cost encompasses finances, resources, personnel and transactions (what does it cost to take action);
- In creative fields, finances are usually not the limiting factor;
- Behavioural principles: humility, respect and trust;
- The social game is not about politics, it's about creating relationships to get things done;
- Not failing means you're not being innovative enough;
- Testing on the Toilet and Learning on the Loo - love a catchy name;
- 1-2% of Google engineers are readability reviewers for programming languages and they operate as volunteers;
- Don't build for everyone, build with everyone;
- Creatives need nurturing, time and space to think and create;
- Hope is not a strategy - build for failure handling;
- Goals/Signals/Metrics to guide metrics creation;
- Tests begin to lose value at 1% flakiness;
- Hyrum's Law: the customer usage of your API speaks the truth, not your intended use;
- Use third party APIs as seams for tests;
- Deprecation warnings should have actionability and relevance - don't really on the customer to deprecate it for you;
- UI is considered responsive if latencies are below 200ms;
- You can avoid becoming dependent on your public cloud solution via using open-source, build tooling on top of it, or using a hybrid (even between 2 publics or a public and a private).

Overall, great read, especially the chapters on Code Reviews, Dependency Management and CI/CD. Easy to read, yet thorough and with plenty of examples of issues that Google dealt with, both in technical depth as well as business depth. Highly recommended and one where I will keep looking at my annotations!

[Justin · Rating 2 out of 5]

The book is mostly theoretical and Lacks practical implementation details. The theory is also is scatter shot and lacks depth. To me the extreme brevity of each chapter makes this feel like a series of blog posts

[Prasanna · Rating 4 out of 5]

Pretty interesting look into what Google's software engineering looks like. A huge chunk of the book can be described into few principles
* Hyrum's rule: If you expose anything in your API someone's going to use it. Both internal and external ones
* Beyonce rule: If you liked it enough, should've put a CI test on it. ie, it's not infrastructure change's fault if CI didn't catch it.
* Make it easy and build tooling for large scale changes because languages change, systems update
* Upgrade early, upgrade often
Another thing this book does well is to go over the build system Bazel. Having used it in the past for some open source projects I had familiarity with it but this book provides a pretty concise explanation of what an artifact based build system looks like. Unfortunately for Go, it still doesn't quite fit very nicely. But the idea that you can remotely execute your build stages and map it directly into your filesystem is very tempting.
Each chapter has a TL;DR at the end which is so great!

[Muhammet Orazov · Rating 5 out of 5]

It is nice book!

Most of the parts may not apply to small team or companies, but most of the practices are solid if you want to achieve long-term software engineering scalability.

One of the main takeaways for me is the Hyrum's Law. (Hyrum Wright is one of the authors)

Hyrum's Law: with a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

Authors argue that even if we test 100% for conformance to strict specified contract, the effective user contract applies to all visible behaviors. It's unlikely that unit tests alone test for all visible behaviors that are not specified in the public API.

We can see this law in everyday software engineering.

[Bernardo · Rating 4 out of 5]

If you have been in the industry for awhile you probably have seen and/or applied many of the concepts explained in this book. Having said that it's still a well written book that covers a wide range of topics. Worth it especially if you have some experience but it's still relatively new to the industry.

[Bartosz · Rating 4 out of 5]

I am positively surprised by this book. It covers a variety of topics: everything from team building and leadership to unit testing. It's also much more in-depth than I'd expect from a book that tries to cover this much ground.

Overall, it taught me some new rules and solidified the ones I was already familiar with.

4/5

[Joel Bastos · Rating 4 out of 5]

It's an enjoyable book but not groundbreaking by any means.

Although the perspective of engineering best practices is from the point of view of Google, most of the book can be helpful to any organisation as it refrains from tieing itself to a specific Google proprietary process/technology, with a few exceptions, of course.

I really liked the clear separation between software development and software engineering, which many companies struggle to grasp.

We can find several solutions for problems most senior engineers working in relative big companies have faced at one time or another, but juniors probably won't have the proper context to understand the "why" of such solutions being created in the first place.

I loved the Beyonce rule, which states, "If you liked it, you should've put a test on it".

[Mauricio Chirino · Rating 4 out of 5]

Entertaining read

It provides some solid guidelines for scaling teams and improving code quality overall. However there were just so many times the author simply said “in google we use X tool for this” and the tool is private so it made me think in those instances that they were simply internal manuscripts that somehow found the public light in this book.

[Elie De Brauwer · Rating 5 out of 5]

I really enjoyed this book, it shows a different viewpoint on software engineering, especially the aspect of doing software engineering cost effective at scale. This in contrast of daily practice where aspects are often overcomplicated without the aspect of scale. But the amount of common sense in this book is extremely high.

[Susan · Rating 4 out of 5]

It's interesting to read this book as a Google engineer because it makes me more appreciative of the tools I've been using on a daily basis and stop taking them for granted.

[Fatih Arslan · Rating 5 out of 5]

Must read for any Software engineer. It doesn't matter if you work for a small or big company, there are many valuable lessons and tips&tricks to make you a better Software Engineer.

[David Souza · Rating 5 out of 5]

This is definitely a book you need to spend some time to really learn from. This book is useful for both people that work at Google and people who don’t.

I did 3 internships at Google as a Software Engineer and there were a lot of things I didn’t understand about how things work in the company: processes, tools, behaviors, etc. I wish I read this book before. It prepares you very well so you don’t start completely from zero if you land a job at Google, or as I said, makes you understand the reason of why things are they way they are if you’re already working at Google.

And it’s also very useful for people who don’t work or plan to work at Google, it teaches you a lot of valuable stuff that you can easily use elsewhere.

Of course there are some quite complex topics that you might not need, but buying this book will end up as an excellent decision. You’ll learn plenty of useful things reading this.

[Leo · Rating 4 out of 5]

A really good one, highly recommend.

Book concentrates on Software Engineering, not on Programming - it rarely discusses code itself, but challenges in supporting large engineering environment, where code may live for 10 or more years and has to be modified by many different teams.

Many of the discussed problems (especially second part of the book, that concentrates more on tooling) would not be applicable to most companies, but some views and "modes of thinking" would still be useful. First part of the book is probably a bit more "generalizable", discussing approaches to testing, code reviews, documentation etc.

Really recommended, if you want to "peek behind the curtain" a bit and see how big companies work, and how their scale poses unique challenges.

[Boyan · Rating 4 out of 5]

This book gives a good balance of presenting both the practice of software engineering and the rationale behind it for many of the techniques used at Google.

Be warned that many of those techniques (eg a large single repository) are not straightforward or necessarily the right path for most organizations -- this isn't a how-to guide. But many of the lessons learned over the years can be applicable to many others, such as when a large change is required (such as a major upgrade of a library or language) to focus on a small expert team doing it rather than imposing it on everyone else. Thinking about sustainable development (rather than just maintainable) is also a great concept -- what is the life time of this code, and can the code be maintained/extended/morph over time to support what is needed?

This is why the book is useful -- the rationale can give you the understanding for the why, and the fact that Google is practicing it (on such a massive scale) shows that it can be important and useful. The how you apply it will likely be different, as each organization has its own history, culture, and skillset.

[Willian Molinari · Rating 4 out of 5]

I'm migrating all my reviews to my blog, you can read the full review (with my highlights and comments at https://pothix.com/softwareatgoogle

It was a great book to read. It’s not the kind of book you read and start applying stuff at your current job. We all must remember we’re not Google, and many things mentioned there are great, but only applies to their context.

With that said, I quite enjoyed the book. I was curious about Google’s structure, and it’s mind-blowing the amount of problems specific to that scale. I even learned a few things that are still worth applying in non-Google companies.

[SY77 · Rating 3 out of 5]

Pretty good overall. There's a lot of decent advice, both for managers and for engineers. A lot of this advice is probably going to be pretty obvious though, since it starts from the very beginnings of Google in late 90s. The level of detail across chapters is rather inconsistent — for example the later chapters about testing I found rather tedious. But it's good if you can deal with that.

So overall I would give this book four stars, but there's a thing that I have to take off a star for. It's the "Hyrum's Law". One of the book's authors came up with a "law" (it's actually a rather sensible observation) and then you're going to read about this guy's law in every chapter, multiple times. Really, it comes up so often that they didn't have to bother with referring to it by name every time to make me roll my eyes. So... A star off for my sore eye-rolling muscles.

[Gustav Messany-Oberwandling · Rating 3 out of 5]

The 2/3 of the book is quite general. I don't see much difference to the software company where I work. Some details are interesting, but the solutions are related to the HUGE monolithic repo. I'm glad I read it to the end, where concepts like "large-scale change" or their approaches to dependency management are explained. They inspired me.

My impression:
- Too much: "We are great and smart" (but don't talk about that other companies are basically doing the same thing).
- Not a critical word about their problems. I would be more interested in the impact of hierarchies and defined power structures, for example.
- Most of the time too general to really get insights and reusable approaches (some parts are different like Unit Testing and I liked them more)
- Rarely concrete numbers, data, facts and rules of thumb. Like "for changes that require more than 500 edits, it's usually easier to learn our change creation tool" - I can work with that.

And most of all, it's nice to see companies allow a glimpse (albeit most likely biased) into their culture and processes. It makes me, at least, feel like humanity is working together a bit. So thanks for that! :)

[TK · Rating 3 out of 5]

It was an interesting read about how software engineering works at Google. There are some things that are particular to Google itself (problems and scale) but most of the topics mentioned I already saw as a process or best practice working in the industry.

It's an interesting resource for newcomers and people who didn't have the chance to work in companies that are solving difficult technical and scalability problems.

I liked that it's a fast-paced book. Divided into topics and easy to digest. Some of the topics, like LSC, are very Google's (scalability) problem, and maybe it's not that useful for most of the companies in the world. But the overview on culture, tools, and processes were all interesting to me.

[Kirill · Rating 5 out of 5]

Good though-provoking ideas and valuable experience sharing on how to scale most critical engineering principles.

I liked a lot ideas about scaling cross-team collaboration, motivation when to measure (and when not!) engineering productivity as well as precious advices for managers. Technical chapters contain many well known basics (unit testing, CI, CD) but still good to read, most importantly to learn what kind of tooling can be applied in a fast growing environment.

[Oleg Stotsky · Rating 3 out of 5]

It's hard for me to rate this because experienced software engineers will know 90% of the informations that this book presents, as most of these things are already industry standards.
For junior developers on the other hand this book has too little practical advice to be considered useful in my opinion.
I guess this book could bring a lot of value in 2000-2010 when most of the knowledge and techniques described were not as wide spread.

[Mohammad Tanviruzzaman · Rating 4 out of 5]

The book is written by build/release/tool people, so the focus is on that part of software engineering that makes the coders more productive - how could the coders build, test, review code better? It does not talk about things like design patterns for example. The code for which the engineering ideas are targeted is assumed to live long, like for decades. The book is divided into three parts: (1) Culture (of learning) (2) Process (strong code review and testing) and (3) Tools (critique, piper, rosie, borg... - names are cool). The book talks about so many ideas that it is hard to imagine that other software houses have done all of those or they may not have done it in the same way which leaves food for thought. For example, put docs in the same repository side-by-side with code, both using the same workflows (as an extension why not have unit tests for the code snippets in the docs as well?) Code review is a turn-by-turn game, making it easy to find whose turn it is and what s/he is supposed to do should help a lot. The idea of monorepo is interesting (I cannot say I fully understand all its greatness yet). Feature flags let features be less coupled with binaries. The code is read optimized and the Readability process is crucial. Hyrum's law - that with a large enough userbase of an API a seemingly innocuous change breaks someone - is a recurring theme in the book. Compute infra should behave and be treated like cattle not pets. Code is liability, useful features are assets - being able to provide useful features with 0 lines of code should be ideal. In the last chapter, the authors talk about compute as a service and the approach there is to start with the manual installation or fix on a single machine and grow from there naturally. The book as a whole, if followed the pattern from the last chapter, would have been way more awesome.

[Ibrahim Tasyurt · Rating 4 out of 5]

Overall I enjoyed the book with great takes; however the chapters focusing too much on Google internal tools were a bit boring to me.

Good:
- Great takes on Software Engineering, Culture and Leadership : a concise version of best practices with some color from Google
- Practical aspects on documentation, testing, style guides and code reviews
- I particularly liked "Compute as a Service" chapter.

Bad:
- Some chapters are very much based on the mono repo approach of Google; and it is not easy for a Software Engineer to relate unless you work there. For instance, the Code Search tool; which has been expanded in 30~pages: It is very much Google specific and probably wouldn't be built that way if Google wasn't maintaining a gigantic monolith. So, reading this chapter is a bit hard to relate and adds little value to the user especially they are familiar with other alternatives (e.g: Github). Similarly a bit too much on Bazel or Critique did not keep me engaged.

[Scott Pearson · Rating 5 out of 5]

Relatively little is known about how to organize/manage software projects so that they come to a successful, on-time resolution that lasts the test of experience. This is the field of software engineering, and over the last two decades, Google has mastered this art. They share their hard-wrought wisdom in this book.

Many developers, like me, wish they could undertake several internships at leading companies like Apple, Google, Microsoft, or Facebook. They could learn the tricks of the trade from what made these entities so successful. Concerning Google, software developers now can just consult this book. It provides an in-depth look into the state of the art at this engineering company. Each chapter is written by company experts and covers 25 timely topics ranging from code testing to dependency management, from continual integration to cloud services.

Google is one of only a few companies that have broached these issues in depth ever. They do not claim that their answers will solve all problems for all time. Rather, they encourage readers to learn from their well-reasoned thoughts and understand their own problems in that light. This book is relevant to the start-up as well as the corporate developer. For us computer programmers, this book is fodder and inspiration for continually producing better software.

I find the quality of writing particularly fresh. Instead of hiding behind older and well-established verbiage, they provide newly thought-out terminology in newly explained reasonings. For a technical book, this work is extremely engaging. The reader rarely if ever gets the sense that the authors are merely regurgitating rehashed theory.

This book is particularly relevant to software developers and managers of software efforts. It gives a lingua franca to the software development effort and provides abstracted concepts that will help companies move development forward. All the way to the end, I stayed engaged, and I predict many others will, too. Kudos to Google for giving back to the industry in this way!

[Nikos Pothitos · Rating 5 out of 5]

Disclaimer: I read only the three chapters by Ben and Fitz (online for free). The five stars go to them because I'm glad that they extended here their marvelous Debugging Teams textbook that helped me to understand what I did well and wrong in my career. The only downside I can find is the repeated praise to Google as a corporation, while it's a common secret that its culture deteriorates over time.

[Goce Bonev · Rating 3 out of 5]

Depending on where you are as a software developer, and the scale and environment in which you work, this book might be too high level if you are just starting out or too basic offering no actionable information and even nothing new. I had the feeling that some of the chapters were a bit dry, repetitive, and even cliche-ish, and others covered subjects barely scratching the surface. There were good bits here and there but overall I was a bit disappointed with what was inside. Maybe the name Google in the title made me expect more.

[César Suárez · Rating 4 out of 5]

No soy muy fan de los libros de programación tochos, porque generalmente están repletos de relleno, pero en este caso se hace una mirada tan completa sobre todo lo que rodea a la programación que bien merece la pena. Flojea cuando se mete en las herramientas internas de Google (reconozco que me las he terminado viendo en diagonal), pero es difícil no encontrar algo interesante en cada uno de sus capítulos.

No hace falta leerlo en orden ni mucho menos, también funciona como libro de referencia o para picotear sólo los capítulos que más interesen.

[Pawel Waclawczyk · Rating 4 out of 5]

It was a long journey for me to read this book.

I enjoyed a number of parts, especially the ones of leadership and management, as well as more technical chapters about testing. Not all essays of tools were that fascinating to me at the time of reading. Yet, I love the "engineering" approach maintained through this book. Also, the ability a look closer at a big company like Google and see their day-to-day challenges and solutions was quite a pleasure and good learning time.

[Daniel Gomez Rico · Rating 5 out of 5]

It is fun, full of facts and practic ideas about every software development company flows, and even talks about técnical related stuff like branching models, versioning and how qualify improves the company.

One thing I really liked is that they never advise "don't think about quality in your project", somehow they give ideas and a macro perspective to think around an evolving quality and to think about where to target it.

It's a good idea to save it and try no to read it all at once, it talks about a lot of different things and can make you feel overwhelmed.