Diving head first into software architecture

Episode transcript

Rebecca Parsons: Hello, everyone. My name is Rebecca Parsons, and I am one of your recurring hosts for the Thoughtworks Technology Podcast. I am joined today by four individuals, a little more people than we normally have. We are going to be talking about a book that is just coming out called Head First Architecture. The authors are Neal Ford, who is one of our co-hosts.

Neal Ford: Hello, everyone. You'll recognize my voice for sure.

Rebecca: Mark Richards.

Mark Richards: Hi, everyone.

Rebecca: Raju Gandhi.

Raju Gandhi: Hey, thank you for having me.

Rebecca: Yes. Sarah Grey.

Sarah Grey: Thanks for having me.

Rebecca: My understanding is that this Head First series has a very particular format and approach to materials. Why don't we start with what makes a Head First book a Head First book?

Sarah: As a development editor at O'Reilly, I'll take that one. Head First is a special series. O'Reilly publishes a lot of different series with different formats. This one tends to be, first of all, much bigger, and second, designed in a very different way. It's meant to engage the entire brain. There's art, there are hands-on exercises, there are bits of story, there are jokes, there are annotations in the margins, all sorts of things in order to engage every kind of learner, whether you're a visual learner, even a kinesthetic learner — you can pull the code samples off of a repository and play with them yourself. This isn't a passive reading experience, it's a very active one. 

During my time at O'Reilly, I've actually found it to be by far the best way for me personally to learn coding skills. Art seems to be the only thing that makes them stick. We did a lot of custom art for this particular book in a very comic book sort of style. I think people will really enjoy it.

Neal: I like to say this is almost like a comic book and a technical book got together and merged and had a baby. That's basically a Head First book.

Sarah: Yes, exactly.

Mark: I prefer to use the term graphic novel, actually, because it felt like I was writing one with this book.

Raju: I have to say that, and this is a story I've told Mark and Neal before, my wife's uncle, who spent a lot of time in the education space, looked at my previous book, Head First Git. As he was sort of flipping through it, he saw a section, he saw an exercise, he saw quizzes, he saw sort of a narrative through the chapter. He looked at me and said, "This looks like a fun textbook." That's when it dawned on me that that's essentially what O'Reilly is going for, is making a textbook, but much more approachable and fun, rather than just being prosy and dry.

Neal: One of the interesting challenges for us, as Sarah said, it's really designed. The canonical book is Head First Design Patterns because it takes a very visual approach to that. Most of the books in the Head First series are like Raju's previous book, Head First Git or Head First Python. They're very concrete, very coding-centric, very small, discrete exercises. That was one of the big challenges we faced in this book, because software architecture is pretty abstract. How do you have an exercise every three pages about software architecture when you need to build up so much context? That was an interesting challenge for us, who've written about software architecture in different media, but this one feels completely different.

Mark: I think the other thing I noticed after-- what was it, Raju and Neal? Two years, actually, I think, right, Sarah? Two years of writing this book. The other thing that I actually noticed is a big difference as an author for the Head First books was the actual style of writing. Neal and I, of course, wrote the Fundamentals of Software Architecture together, as well as Software Architecture: The Hard Parts. We embarked on this project as initially, and Neal always joked about this, "Oh, this will be filler work for us."

This was a major project. Part of that was the difference in the style of writing. Each consolidating topics, each page effectively became a work of art. Although there are , it was a different style of writing than in our other books. The use of diagrams and figures and characters asking questions. The bottom line is I really felt as an author was more connecting with the reader through the Head First books than in our other books.

I felt through Neal our other two books that we're really telling people about all these things, telling the audience, telling the reader about this. For some reason, as I was writing some of the chapters in Head First, it became like I was actually with the reader and holding their hand and showing them things and following through and communicating with them. That was a really-- it was an interesting aspect of the book.

Neal: It's a mistake to think that there's just this big giant pile of artwork and you sort of sift through it and use it any way you want to. My analogy to this is, this is much more like writing a sonnet in poetry. Sonnets have a very strict structure, has a certain number of syllables, number of lines, and rhyming scheme. Head First is exactly that way. You have to learn all these things.

There are these specific characters like Skeptical Girl, but she can only be used in specific situations and specific transitions and not too much, but not too little. Finding the sweet spot for what we sort of affectionately slash, derisively called the little doodads of Head First, little characters and the iconography and the graphics and all that stuff. Figuring out where to use that when it's appropriate, what it's there for and making it part of the flow of the story is a very interesting challenge.

Sarah: Absolutely. This is a series that's been around for about 30 years now. The pedagogy has been developed in this really specific way. It was very much a challenge to not only master that, but then to integrate it with a much more abstract concept like software architecture than we usually do on these books. I think the result is really accessible as well as fun.

Neal: I'll give you an example of where we cheated on the Head First style. In most Head First books, you can't have any examples that extend past one chapter, but there's just no way you can make everything in software architecture completely self-contained. We created a couple of what we call sidecar chapters, which are basically taking a step away from the sort of educational stuff we're doing. It's just solving problems using all the steps that we built up the chapters before that.

We have a couple of those which are basically just purely implementation chapters. They're called do-it-yourself chapters. They're very much designed to reinforce the sort of abstract steps we've been leading up to but glue it back together to make it a little more holistic, because you can't write a book about architecture where every chapter cannot connect to the others. It's just too tough.

Rebecca: You all have been enthusing about this format. Have you just given up on sales of the other two books? Can you talk a little bit about how the books relate to each other?

Neal: Absolutely. I'll start and then let the other two authors chime in. We have not given up on the other two books. Mark and I also do online classes and videos, too. I actually don't like learning about technical subjects through video, but I know a lot of people do. I actually don't care much for the Head First style when I'm trying to learn something, unless it's something extremely concrete like Git.

I sort of facetiously refer to this as the ADD guide to technology, because there are all these interruptions and sidebars. I actually prefer just regular prose, as dense as you can get it, but I realize that that's not true for everybody. Mark and I are very conscious of trying to basically write stuff in every medium we can find, because people consume and learn things in different ways. This helps facilitate that. That's just one of the many differences. I'll let the other two dive in some more.

Mark: I think, as Neal is fond of saying, if you draw a Venn diagram of our Fundamentals of Software Architecture, and also then the Head First Software Architecture, they don't exactly meet. There are things that we were able to introduce after-- what was that, Neal? Four years. Four years since our Fundamentals of Software Architecture has been released. Things that, in hindsight, we wish we would have added to the Fundamentals book, but also things that we wanted to add but didn't have room for.

We had the opportunity, for example, specifically one of the chapters Raju worked on, was something we didn't even introduce within, and I'll let Raju talk about that one, but things that-- entire chapters that weren't even in the Fundamentals book. There is quite an overlap, but also I think part of the primary difference, originally, Neal and I thought, "The reason this is probably a filler work is because we've already written the Fundamentals. We just need to lift that into a different format." [laughs] Neal, you still owe me for that.

[Laughter]

Because nothing could be further from the truth. That was our original intention until Raju and Sarah started teaching us the Head First style, and like, "Oh, no, no, no, no." There's a lot of differences in this. Part of it is just the way we went about describing and teaching a particular concept. You may read about characteristics, architecture characteristics, for example, in our Fundamentals book, and you turn to the chapter on architecture characteristics in the Head First book, and it's entirely different teachings. It's that style that I gained appreciation for as writing the Head First book. This has transcended into a lot of my training classes that I teach, both online and also in person, in just ways I explain some really abstract concepts.

Neal: I can give you a concrete example of what Mark is talking about. One of the things that we purposely did not talk about in the Fundamentals of Software Architecture book was the difference between design and architecture, because it's such a sticky subject, and it's so nuanced, and we purposely just did not-- we just avoided that in the Fundamentals book. Because of the target audience for this book, we felt compelled that we really should address that in this book.

In the first chapter of this book, we dive into the real difference between architecture versus design. I think because we wrote this book a few years ago, we keep getting asked this question over and over, and we've come up with examples and some thoughts about what is the real difference here. I think the interim time between the books helped in some insights that we wrote about in this one.

Raju: One thing that I didn't mention when you asked about the differences between Head First format and other formats is there's a constraint on how you think about teaching a concept, and that is, typically, you have a page to introduce a concept. There is no overflow. You can't just keep writing prose, and if it flows into the next page, just put a page break and go to the next one. You have a page to describe a concept. More importantly, you can only describe one concept per page. In other words, you can't have two different concepts simultaneously introduced in the book.

I think that really, as someone who had consumed both of Mark and Neal's books previous to this, and joining them on this project, I think teething that apart, forcing us to think about how can we introduce each topic separately, building on what we already have, really distilled and clarified many ideas. To Mark's point, we had to come up with analogies, examples to explain ideas. I think that's one really big difference.

The other one, which is we introduced more material and skipped some. We didn't want an exact overlap between the two. Then to Mark's point, we had the opportunity to dive much deeper, especially into things like the four dimensions of describing architecture. We have a chapter dedicated to each dimension, which lays the foundation before we get into the architectural styles. That wasn't something that was in Mark and Neal's book. Not explicitly. It was there hidden in the subtext, but it wasn't there explicitly.

Sarah: I think this book also provides a great jumping-off point. If you're interested in software architecture and you're feeling like you don't necessarily have an entry point, this is that entry point. Then you go on to the Fundamentals book from there.

Neal: Yes, I think this is a perfect on-ramp, particularly if you're not quite have enough experience to understand the Fundamentals book completely, because in terms of what they both cover. We've been talking about where they don't overlap, but where they do overlap are things like discussion about architecture characteristics, but in a much longer, more drawn out way than we did, much many more examples than we did in the Fundamentals book. We cover a lot fewer architecture styles because we just don't have the room. This is a very expansive way of writing a book.

The three parts of the Fundamentals book, part one was sort of intro and characteristics, part two was styles, and part three were the soft skills. We basically covered chunks of part one and about half of part two, and then little bits of part three in the Head First book. The parts in part three, like ADRs that we only covered lightly in the Fundamentals book, we cover really extensively in this book. There's actually quite a bit of differentiation and a more streamlined approach to the things that show up in both books.

Mark: As a matter of fact, I would say, having written both, that the books are very-- when I say the books, the Fundamentals of Software Architecture and architecture, the Head First Software Architecture, are very complimentary to one another. In other words, different audiences may have different ways of being able to learn about a particular concept. However, maybe reading a part of the chapter on characteristics, which is a good example, and then going over to the Fundamentals book and reading the corresponding chapters that have to do with characteristics, both would actually complement each other and vice versa. Reading one of the chapters of Fundamentals and scratching your head and saying, "Yes, you know what? I just don't get it." Going over to the Head First book, seeing if we cover that piece, reading and going, "Now I see exactly what you were talking about." I think in a lot of ways, the books are very complimentary to each other as opposed to overlapping material.

Rebecca: Oh, I would think it would be challenging to say the least after having laid out conceptually and put in the effort of how you're organizing the material for the Fundamentals book and then say, "Okay, let's toss that out the window and start over." What was your approach for deciding which of the chunks to include and not include? Were there principles in how you would approach something in the Head First book to sort of re-imagine it in that way?

Obviously, I'm inferring that this is much more example-driven. We certainly saw this with the evolutionary architecture book, particularly around fitness functions as it really helps people to have concrete examples to think about. What was the philosophy and approach to taking this body of knowledge that you had characterized in one way and now characterizing it in a very different way?

Mark: One of the starting points was the fact that all of us, including Sarah, are in different geographical areas here within the United States. One of the first things we did was to learn more about the style. That's where Raju really helped us at the beginning, not only with some of the architecture stuff and the content, but really Raju and Sarah just teaching us the Head First style. That was absolutely necessary to come up with our narrative arc for the book.

Because certain topics lend themselves, I think, to the Head First style and Neal and I recognized other things we wrote about aren't going to. Part of it, Rebecca, that I think was the core crux of our narrative arc was the realization when we put a spreadsheet together of, if we're talking about one page per topic and exercises every two to three pages, this is going to be about a 15,000-page book. That realization, we just took a machete to our Fundamentals book and said, "Yes, we could probably maybe cover a quarterof it, maybe."

That was one of the drivers to really pick out a solid narrative arc about really the topic of what is architecture and pulling out the relevant pieces from a Head First style that would lend themselves towards visualizations and metaphors and analogies and exercises. That really was for me the thing that started to drive really the topics that we were going to cover or did cover in the book.

Neal: It turns out that Mark and I are on like this 15 year refinement cycle for material because when we first started talking about, the first conversation we had about a book about software architecture, we're like, "There's no way you could write that down. The topic is just too big. It's too multifaceted. There's too many nuances." What we did by doing training classes and conference talks, et cetera, is filter that down to the Fundamentals of Software Architecture book.

Then we had to look at that corpus and then say, "Okay, now can we refine it and filter it down even more?" This is a question probably for Mark and Raju. Is it humanly possible to write a book about software architecture that filters any of this material less? I think this may be the smallest possible chunk of IP that would completely describe software architecture. I don't think we can refine it anymore. Authors think about the horror of Mark and I sat down with a spreadsheet and tried to come up with the titles of 450 pages because every page has a unique title indicating the topic.

Every outline that you do for a book, we ended up throwing almost all those things away as we started writing it. It was really useful up front to get a-- because we were afraid, to Mark's point, that we're just going to start writing this thing. Then, at 1,000 pages in, it's like, "When are we going to be done with this thing?" Getting our heads around it and starting that sifting and refining process, which continued as we were writing the chapters. Every chapter I wrote, I bumped into at least a couple of topics where, "Oh, we're going to talk about this, but that's going to be eight pages. I don't have eight pages of budget that I can talk about that, for the setup and the transition and the exercises and all that." That also, to your original question, Rebecca, the format itself helps filter things down to a certain core.

Raju: I'm certain that most of us, or at least the three authors, have repressed the memories of many moments in this project, because I cannot keep count of how many false starts we had. Neal just alluded to this, he and Mark sat down in New York and hashed out this Excel spreadsheet. Then we figured out, as we started thinking through what we wanted in the book, we're going to dispense with a lot of that. Even with writing chapters, I still remember writing the chapter about the two laws of software architecture.

I wrote the whole chapter, and then we have the editorial crew, who now lead O'Reilly's Head First efforts, look at the chapter, and absolutely trash it. As part of the commentary that they left behind, we finally realized that the whole chapter doesn't have to be a sort of mirror for what's already in the Fundamentals book. It helped us focus, sort of zoom in and, say, just focus on the two laws of software architecture. We trashed the entire chapter and rewrote it from the ground up. That actually helped us focus the next two chapters of the book.

I think we had a lot of false starts. I know Mark's gone down this road where he's written 16 pages, and then realize, oh, no, this is a dead end. You can't just delete three paragraphs and refocus. You have to go back to the beginning of the chapter and rewrite the entire narrative. Because there's a storyline, there's an arc, there are characters, and there are constraints that come in. I think, from a challenge perspective, and this happened to me when Head First Git-- I can't remember how many chapters I had to throw away, just because I hit that dead end, and there's no way to recover. I think that's probably one of the biggest challenges of Head First. It's super focused, but if you don't aim correctly from page 1, you're just going to go to back page 1 and try all over again. There's no recouping in the middle of a chapter.

Sarah: We did a lot of troubleshooting different metaphors, playing with them, trying to see if they could stick. I think one of the other challenges, too, was bringing it down to that filter level where people who are new to the field are going to understand. With three authors who are super geniuses and spend their whole day immersed in software architecture, my role was to ask the obvious questions that a reader might ask.

Mark: Sarah, you ended up being, at least in my opinion, our hero in the book, because Neal and I just started-- all of us, of course, Raju, you had your Head First Git. Poor Neal and I are just like, "This is right." Sarah and also Raju were both very instrumental in saying, "Yes, no, let's start over." It was a steep learning curve. I think that was one of the biggest challenges was that first couple of chapters each were really a learning process. At one point, it was such a steep learning curve. Sarah was there every day.

Raju was helping us understand the dynamics. That first chapter took so long to write. I sat back and I said, "Oh, there's no way I can do this." It was like a hockey stick. It started out slow and it's like, "oh, I get it now." It got better. Fortunately, Sarah was there to say, "No, no, no, no. You can't use that character twice in a spread. Raju would say, "No, no, you can't say that. You can't use an annotation there like that," and, "Oh, boy, why not?" It's like, "You can't." The constraints that were placed on us were good, solid constraints, but they were still fairly frustrating constraints.

Neal: A lot of the learning curve, too, had to do with tooling. Mark and I prefer to write in pure text and use Git as a back end, focus purely on the pros. In Head First, this is done in a graphic design tool. Every page is a completely unique layout. The pros really literally does flow around the graphics and vice versa. You can't just change the pros without changing the graphics and vice versa. That makes it very intense. I would estimate that of the 450-ish pages that made it in the book, we probably generated 800 pages worth of material and then deleted the 350 that were left over because I know at least the first two chapters I did basically rewrote from the ground up twice before we finally found the sweet spot.

Mark: Neal, and to your point, the microservices chapter, which I tackled, how do you explain and describe microservices using graphics style in a Head First book? That was case in point, about close to 30 or 35 pages, and realized, no. As a matter of fact, it was Raju who said, "No, you're going from the top down. You need to go from the bottom up. This is Head First. You're showing everything and then trying to dive in like little spikes. It just doesn't work that way."

I'm like, "It looks good to me." It's like, "No, no. Scrap. 35 pages. Just go on." I said, "Fine. New head, new pages. Let's start over." Raju, you were right. [laughs] It was just something that I didn't observe and Raju did. That no, you get to start out small and build the triangle as a broad base as opposed to the other way around, which is how I was trying to describe it. Yes, I would say it's probably a fairly accurate number of the number of pages that we ended up saying, "No, try that again." [laughs]

Neal: Fortunately, InDesign does not track that because I would be afraid to look at that number. It would be shockingly depressing. It just melted away into the ether.

Raju: Just to close that thread really quickly, InDesign is a horrible collaborative medium. We were literally acquiring locks on files. If Neal's working on Chapter 4, nobody else work on Chapter 4. Everybody else work on something else. Then it was texting and semaphore locks and all kinds of shenanigans ensued just because the tooling does not support collaboration so much.

Sarah: Can't tell you how many times I've texted Raju saying, "Help, I think I did something wrong in the repo."

Rebecca: Given all of these challenges that seem unique to the Head First style-- Just writing a book comes with its own set of challenges, but then you add all of these constraints that you were talking about. Raju has obviously decided he's okay doing a second Head First because he just did. Neal and Mark, what about you two? Did you feel like this was a worthwhile decision exercise for the material that you were trying to create? Given the amount of trouble it was, would you do it again?

[00:30:42] Neal: I've been asked this question recently and my stock answer to that has become, maybe, but not next, and not soon [laughter] because it is a fascinating challenge, especially given the subject matter that Mark and I have talked about and written about and created videos about and podcasts about so much. The challenge of this book had very little to do with the software architecture stuff. It was how do you get it and massage it into this format, which is interesting, but it's interesting in both good and bad ways. I think I completely understand the pedagogy behind the Head First style.

I think that most of what they're doing is right, but some of it is a little too constrained, in my opinion. I think there are places where I would loosen up some of the strictures a little bit, but that's probably going to be true for any author. We'll find things that they would prefer to push against. That aside, I certainly have acquired the skills now, but I'm not looking forward to leaping into another one soon. I'm actually looking quite forward to settling into a chair with Emacs and writing some nice long page after page after page of prose with no graphics whatsoever anywhere in sight for a while.

Rebecca: That sounds like my kind of book. [laughter] Lots of prose and no diagrams.

Mark: I think I'd have to parrot Neal and say, maybe, but not now. It's a huge time commitment. Raju had given both Neal and I a metric of number of pages per day. Actually, it was number of days per page. I'm like, "No, no, no. Are you kidding?" I wrote an entire chapter on European flights back home. "No, no, Raju, you're wrong." No, Raju, again, was right. That metric was scary because I'd looked at that. However, however much frustrated I did get in terms of the time commitment and the rework and stuff, I really, again, did just gain that appreciation for the different way of describing certain concepts and really grew to like it.

As a matter of fact, you will start to-- probably a lot of you listening have already seen annotations slowly appearing in some of our classes and online trainings and talks and conference sessions. I started to adopt a lot of the techniques and the style of teaching and I decided I rather like these little call-out annotations in my slides. Yes, that's a good example of where I started to adopt some of the Head First style in just my daily practice of writing and working. Not right away though. The frustrating piece is all this skill that I gained, not only in InDesign, but also the techniques. I don't want it to go away, but definitely want a break. [laughter]

Raju: I actually signed on with Mark. Mark and Neal called me up and said, "We would like to work with you on Head First Software Architecture." I signed on that project before Head First Git was over. I definitely am a glutton for punishment. [laughter] I literally went back to back with two Head First books. There is, depending on where my head sits in maybe a month or two months from now, the chance of a third one collaborating with somebody else on a Head First book but I'm going to really think through that one before I sign on.

Neal: I think Raju is in danger of, or will benefit from becoming the Head First whisperer because he understands. He's encyclopedic on the Head First style.

It's actually worth saying, I think, the order of names on the Head First Software Architecture with Raju's name comes first because we didn't want people to think that he's just there because he's the Head First whisperer. He brought a lot of subject matter to this book as well.

If his name is the last name of the book, everybody's just going to think, "Oh, he's just there because he knows all the Head First stuff," which he does and he's encyclopedic, but he's also a good subject matter expert. I think that's going to constrain Raju. He's going to want to write about stuff that he knows about, not just as the Head First whisperer. We're quickly going to find out the series of topics that Raju feels comfortable enough to expound about. We'll turn it to Head First books.

Sarah: Fortunately, there are lots of those.

Neal: It'd be a real shame for Raju to lose. He really does now have an encyclopedic, "Oh, yeah, you can do that. That was in the Head First Agile book on page, like the 200s or something." It's just really amazing how much he has incorporated that. He's become an encyclopedia for Head First.

Rebecca: I will admit when Sarah was first introducing the concept of Head First book, and she said they tend to be larger and I recalled just how many pages were in the Fundamentals of Software Architecture, I was a little nervous, I will say, but the 400s, [chuckle] that seems doable. I would like to thank all of you, Neal, Mark, Raju, and Sarah, for talking with me today about the Head First Software Architecture book and the Head First series, the Head First approach, as well as how these books relate to the other software architecture books that both Neal and Mark have collaborated on in the past. Thank you all.

Raju: Thank you.

Sarah: Thank you.

Raju: It's a privilege to be here and huge fan of the Thoughtworks podcast. I'm glad to be on one episode, at least.

Mark: It's good good to be back, Rebecca. Thank you.

Neal: Yes. Thanks for having us. I will now get up and walk to the other side of the room where the hosts sit. Thanks, everyone.

Rebecca: Thanks, everybody. Bye-bye.