The state of API design

Podcast Transcript

Mike Mason:

Hello everyone, and welcome to another Thoughtworks podcast. My name is Mike Mason and with me here my cohost is Zhamak Dehghani. Hi Zhamak.

Zhamak Dehghani:

Hi Mike.

Mike Mason:

And we have Brandon Byars with us who's a tech principal from Thoughtworks and also James Gregory, who's joining us all the way from Australia today. Brandon, would you introduce yourself for the purposes of the podcast today?

Brandon Byars:

Sure and thanks Mike. So I have been a long time tech principal at Thoughtworks actually 10 years, which is nice in terms of getting sabbatical here shortly. I've led some fairly large accounts and in recent years I helped bootstrap the digital platform strategy offering along with Ryan Moran and Paul. In that role, I've played an API, SME across a couple of accounts in North America, a platform architect role once or twice and I've done a fair amount of gnarly API work in telcos, retail, marketing science, and airlines. And right now I'm working with oil and gas company here in Houston trying to find a way out of SAP. And they seem to think API is their three letter silver bullet for that. I have a book coming out this summer on testing microservices and maintain the Mountebank open source project.

Mike Mason:

Great, thanks Brandon. And James, would you introduce yourself?

James Gregory:

So I'm a developer and tech lead out of Sydney. As a tech lead, I tend to be one of the first people to drop onto projects when we're kicking them off, especially one of the first technical people and that means that I am often involved in all of those early design decisions and discussions about what our systems are going to look like, what the infrastructure is going to be like, and how our APIs are going to be designed and integrate. I've worked a lot with small to medium sized organizations like startups and SAS providers looking to expose APIs to the public and then plenty of B2B or internal technology companies looking to build up their capabilities and often APIs former integral part of that. So I've had a fair amount of experience of seeing these projects created and then I try to stick around for as long as possible to see at the impact of our decisions and try to learn from what we do and see how we can do it better next time or improve what we're doing as we're going.

Mike Mason:

Thanks James. And so today we're going to be talking about API design, REST and so on and so forth. So Zhamak, would you tell us a bit more about the topic and I think it was something that you specifically wanted to talk about?

Zhamak Dehghani:

Yeah, absolutely. And especially with James and Brandon because they have hands-on experience, field experience, taking organizations from kind of more old style or legacy style of service design to kind of more modern API design. What I want to cover is the state of API design. What we have learned over the last few years, we had very overexcited reaction to RESTful APIs after SOAP API design. And we've learned a whole bunch of lessons. But I feel the industry is shifting and moving in different directions. We see the emergence of GraphQL in certain communities, the reemergence of RPC style with GRPC and some folks don't like any sort of point-to-point API calls and they are in favor of more event-driven API. So I would like to pick Brandon and James' brain and hear from their experience, what their experience developing APIs over the last few years and what the evolution been for them on the ground. Brandon, do you want to get started and tell us a little bit about your journey in terms of API design?

Brandon Byars:

Sure, and I would definitely count myself in that early overexcited crowd with REST. I think one of the tendencies we have as an industry is to find some new idea contrasted to an old one that we can all throw darts at and push it probably further than where it's actually useful. I certainly did that with REST, I think a number of other people did as well. I started off in sort of a lightweight fashion within a hospital designing some REST APIs and then along with Mike went to a telco up in Canada where there was a very large legacy, a replatforming project and some previous Thoughtworkers who preceded me on the account. Jim Weber in particular had set a vision of web-based integration for doing that and we were really at the forefront of pushing REST into a very complicated organization. So it was very exciting time to try to shift thinking out of the ESB and traditional integration approaches towards more of a web-based one.

Brandon Byars:

We went very heavy down Atom as our integration platform trying to use hypermedia and I think we had some battle scars coming off of that experience. That at least help pull me back towards realizing where the boundaries of pragmatic REST and then where you get into a bit more theory where we as an industry, the tooling and the thinking really haven't advanced to a state where at least in my experience, where we found a lot of value out of taking some of the hypermedia conversations forward. Where I've landed is realizing that APIs are one of the reasons that we disliked SOAP and we disliked WSDL was because everything was about machine to machine communication. If you wanted to generate client libraries for SOAP you would use machines to generate those client libraries based on some of the specs that you had.

Brandon Byars:

And what felt good in the early days about REST was that it let us humans see the contract and it removed layers of abstraction so it simplified it. As we moved further and further down the path towards hypermedia we started to make some of the same mistakes we made in the SOAP days of optimizing from machine to machine state transitions. At least my learning and obviously it's not universal, is that it's good to revert back a little bit and to focus on the human readability and focus on the developer experience which means compromising on some REST purity. And so I've taken that approach in recent years and I feel like I've had more success sort of dialing back some of the Richardson Maturity Model to say hypermedia is great as a documentation for developers. Hypermedia as the engine of application state is still a little theoretical in the environments I've been in.

Mike Mason:

So Brandon, you were talking about pushing REST a little bit too far and you mentioned hypermedia specifically. Some of the thinking behind REST is that because you're making extensive use of hypermedia that an API becomes discoverable rather than needing extra documentation to go along with it. Is that what you're talking about when you say you're trying to push REST a little bit too far and it not working out?

Brandon Byars:

Yeah, and even more than that. The idea of hypermedia is the engine of application state is actually making these state transitions within the API discoverable and automatically navigable by an application without necessarily the human having to manage it. And it's a great idea and the web normally works that way except you always need additional intelligence to figure out how to navigate the state transitions in the web. So when a web page sticks up a form and it has a data that you can enter that's great, that's your state transition when you click submit. But you still as the human need to know the intelligence of what to fill into each field. And that's the hard part but eventually what you end up with is turtles all the way down because what we keep trying to do in the REST world is allow the computer to know more and more of that intelligence in the human and ls.

Brandon Byars:

And so we end up with these very complicated hyper media formats things like Collection+JSON or Siren. There's some lightweight ones like how in JSON API but the people who are pushing the envelope in hypermedia, Mike Amundsen is one that's well known are really trying to push as much of the intelligence in the machines as opposed to the humans. And what I believe ends up happening is you have APIs that are no longer easy to read for humans. And my experience is that it's better to enable developer productivity by reducing some of the hypermedia by expecting that developers will manage the state transitions by playing around with APIs and sort of figuring them out.

Mike Mason:

James, what's your thinking on REST and the journey of us using that? Is it something that you have come to find useful and found the right kind of balancing point in terms of using REST speeches versus maybe a shoeing some of them in preference for simplicity?

James Gregory:

I think it's a really interesting and difficult problem for us to solve. So part of my journey has been starting over in the SOAP world and slowly evolving to RESTful APIs and there was a sweet spot a few years ago I found where we were doing a lot of machine to machine or service to service integration where REST seemed to fit fairly naturally. It was reasonably discoverable book obviously as we got into Tareq it became harder. The thing that I've found challenging over the past few years has actually been a reintroduction of several old constraints that we used to have that we've thought we got rid of, which was around performance and things like the number of connections that you need, the lights and see for your servers and the overhead of processing responses and GE zipping and GE unzipping things.

James Gregory:

And this has come about because of things like mobile phones obviously they're not new, but we're still seeing a lot of resource constraints happening in mobile phones and in single page applications that don't lend themselves to really elegantly designed pure REST APIs that require numerous requests to serve as to navigate about and interpret the responses. When you're using a mobile phone that has bad connectivity and only half of your requests come through and you've got to deal with that. That's been a really interesting kind of shift at to deal with in my projects. That's the purity of the API really doesn't matter that much if nobody can use it because it's too results intensive or is too difficult to run for a mobile app. So yeah, I found a shift back towards some of the more RPC way of working to be quite useful in that you can have clients applications request exactly what they need in as few requests as possible and that's really challenging some of our thinking around how we design RESTful APIs.

Zhamak Dehghani:

Can you share some of the technologies that support that style of communications? They should be in the layer that you mentioned in the mobile client facing layer.

James Gregory:

So, to build these kind of APIs that are much more client friendly than kind of developer role or API friendly. Firstly, we tried it with REST. We've done some macrobiotics to try to describe what the client is actually presenting to the user as some kind of sophisticated virtual results. Like if you're looking at your profile page, maybe a profile as a resource so you can get everything you need in one request. Maybe a dashboard is a resource or a report, but it was always kind of stretching the metaphor a little bit. So beyond that we struggled with that, it didn't really work very well it was a bit inelegant. And then we found approaches like GraphQL, like you mentioned at the start that I really know complicated being evolutions of what we're doing still operate over HTTP. They still use JSON and all the other things that we're used to with RESTful APIs.

James Gregory:

But they allow you to send singular queries to a server that return you the data that you need in a shape that the client needs with just few requests and as few transformations as possible. So I've been doing a lot of work with GraphQL over the past year or so and it's interesting. It's solving a lot of the challenges that we've had with these larger REST APIs. I don't want to sell it as a silver bullet because I think people get the perception that GraphQL is going to replace REST, that it's going to replace RESTful APIs and we're never going to talk about REST again. And I don't think that to be true. It's a complimentary technology for a very specific problem with some real benefits for things like mobile applications and single page apps in web browsers.

Mike Mason:

So James, you're talking about GraphQL here and I think we should probably explain it a little bit more in case people haven't come across it already. It's basically a mechanism for the client of a REST API to specify a specific query that they want the server to run and the way that they want the server to produce output. Is that right?

James Gregory:

Yeah, that's correct. Again, it's nothing that is revolutionary. We could certainly implement this kind of stuff and probably have before in our REST APIs. But the key thing is the client specifying exactly the data it needs in the shape that it needs, sending that to your API and having the API respect that and respond in the shape that the client wants them. And that actually allows the client to do away with a fair amount of code for manipulating responses from APIs into the structure that it needs. So at its cole Graphic UL is sending a JSON document to the server and getting one back in the same shape but populated with the data that the server knows about.

Mike Mason:

So you could have two different clients, maybe one that wants a richer set of data from service and one that is happier with maybe a lighter weight set of data and they can use the same rest endpoint and change their GraphQL queries to automatically get back less data and maybe structured in a slightly different way.

James Gregory:

Absolutely, yeah. And that's one of the things that we've seen a lot with single page applications and mobile apps that are consuming that mulled traditional REST APIs that you build these general purpose end points for retrieving a list of resources or retrieving a single results. And because you've got multiple consumers of those things you end up setting up the super set of everybody's requirements out of those end points to the conclusion of like these mobile apps receiving way more data than they care about because there is just as general purpose results available to them. And that's really where GraphQL sets out to try to alleviate is how do you send just what the client needs to that particular client.

Mike Mason:

And I guess that also gets you away from needing to use something like a full API translation mechanism, Apogee and various others do that kind of thing. And I'm always slightly suspicious because... Well I'm waiting for everything to be the next ESB. So I worry about API translation for the same kind of reason. Moving on a little bit, we've been mentioning RPC right and I think one of the benefits of REST was getting people to think about resources and operations on resources rather than just methods executing a remote method. But we've sort of been dancing around the fact that there is a bit of a resurgence in RPC in general. Brandon, do you want to talk a little bit about that resurgence and why you think that's happening?

Brandon Byars:

One of the quite difficult tensions that I certainly don't have the answer for and it's a common question in clients around APIs is how do you balance that tension between reusability and fit for purpose. The more you optimize for reusability the slower you deliver. The more you optimize for fit for purpose the more the explosion of APIs within ecosystem. And when we started down the REST journey, I think there were a lot of really early successes that we could point to because there was a lot of public API support for REST, duel PayPaL and we have some really good example that came later with the GitHub and Stripe and that felt very good. So that helped build the momentum for REST APIs. But what was easy to forget is that those were public APIs that are necessarily general purpose.

Brandon Byars:

When we adopt that mentality inside the enterprise and we try to aim for the same level of reusability, sometimes we get stuck on the concrete. I recall at a conference in London years back top developer agent Adrian Cockroft who is the Netflix cloud architect and he talked about what developers really wanted and of course Netflix work with tremendous speed and what developers really wanted was RPC because it let them get something done quickly. And Netflix case actually even having some client libraries for that allowed them to change out the transport mechanism for throughput and foreseeability transparently to clients.

Brandon Byars:

So they built tooling to enable agility and allowed some of the explosion of services to happen when microservices became more popular and you have other examples of that places like Huber, that started to build more momentum back towards RPC. I think we're a pendulum swinging, I expect it is probably going to swing a little bit too far but that seems to be what has happened is organizations adopting and mentality of I can get something done much quicker if I have more fit for purpose and don't have to deal with some of the slow down and trying to get the modeling just right that we adopted when we tried to move towards a REST medium with.

Zhamak Dehghani:

I want to cover the full scope and then come back into compare kind of different styles that we talked about. And for that, I don't want to leave event-based communication out. So I wonder what James, what your experience with even driven style of communication or even Dubin style of API design as opposed to REST or GRPC which is more point to point. You know who you're talking to and where does his requests and responses going as opposed to event-based broadcasting the events and letting the consumers to decide what to do with it. Can you talk a little bit about your experiences? Where do you think that model fits and how it compares?

James Gregory:

Yeah, I think it's interesting evolution of the techniques that we've developed over the past several years. What we're seeing with like event-based architectures is things like we are still doing the modeling of our domain. We are still spending a lot of time thinking about how to represent on domain and how it ties together and where our contexts are and all of that domain design that we really started to do with REST we're doing our events now. For the same reasons that these things are going to be long lived pass between different systems. We don't really know who the consumers are, we need a bit of rigidity around how things are structured and how we expect them to be used. But it is a very interesting shift for us. Like we are still doing all of that design but not in a way to actually expose it necessarily through an API, through a service like a REST API.

James Gregory:

So I've done a fair amount of stuff about where we have systems that publish events of change. They go onto a queue or they go into a stream of some description and then over services integrate in that approach using that from deprecating legacy systems. So integrating systems using events by publishing data when things change. And in fact we've developed several services doing [inaudible 00:21:27]. That actually have no publicly consumable interface in the form of like a REST API or a or an RPC interface. These things purely consume events, react to them and publish over events. And I think it's quite interesting to think how RESTful APIs or API design fits into this wealth where systems may not even have a public interface that can be read or interacted with. They are purely responsive publishing events back and forth on streams.

Mike Mason:

I think what's interesting as well James is that you're touching on the fact that architecture and integration design are heavily intertwined and you might have a system where pieces of the system could get information from each other via REST APIs in a synchronous manner and you might have events flying around and you might have a mixture of them. I think we've also touched on microservices as well briefly mentioned in the podcast and it's interesting to me how all are these things relate to and need to be considered in conjunction with each other.

James Gregory:

Yeah, absolutely. I've often thought like as we go towards smaller and smaller services, microservices thinking about how we publish that data out. What is a resource in regards to microservice? How much data is that service actually holding and what are the representations of it? And the effort that we put into designing RESTful interfaces, how much of that is applicable to these individual little services that may only actually be exposing a single resource and how does that work when we talk hypermedia and versioning and linking between resources, it's all a bit of a shift that doesn't quite fit as neatly as it used to.

Zhamak Dehghani:

I think it would be interesting to go through some kind of hypothetical scenarios where we would use different style of APIs. Brandon you mentioned you work in large clients, large enterprise organizations. If I was an enterprise coming to you and said we had kind of old styles SOAP integration models and it's very hard for our teams to innovate and use these services, discover them and we want to start exposing our internal capabilities to the rest of the organizations. So we want to have some sort of an organizational wide API strategy. What would be your recommendation? Where shall we get started in terms of the choice of API and it's sign of those?

Brandon Byars:

So I think before getting to an answer to that. There's some broader questions that that informed the strategy. I think you probably want to start with understanding what the real business capabilities are and again there's some tension in to how much fidelity you want to put in that modeling versus how much agility and you want to show some progress to the users. But a lot of times the capability model proceeds to decision as to whether you want to move towards more of a traditional RESTful approach to improve the problem that they're directly speaking to, of discoverability of ease of use versus a more inventive model. Because James said it's the same domain modeling that you tend to do but the event theme model may have the additional benefit of getting you closer towards democratizing data across the enterprise which is a recurring theme at least in my consulting that I've run across is organizations are starting to realize that they are losing money and opportunities due to their inability to stitch data together to gain insights.

Brandon Byars:

And while they've done the traditional data analytics path of data warehouses and data lakes, they don't seem to be making as much progress as they want. And I think the interesting aspect of venting is that when that is part of the problem space that the organization is facing, if you start down the path of giving still an application programming interface, then it still meets the definition of API. But it also provides a way of separating data sources and data syncs in a way that you can solve some analytical concerns as well. So I like to think of this as multiple dimensions. The format of how you expose the capability, sometimes you expose it multiple ways because they're solving multiple organization problems but I think appreciating that you need to first think about how you think about the business capabilities within the organization.

Zhamak Dehghani:

Yeah, so business capability design kind of has the foundation. And then depending on the migration strategy, as you mentioned you might have different sides of integration in place at the same time. James, I know you worked at different startups and using APIs for exposing their kind of offerings and capabilities. If I was a startup coming to you and I said I want to have build an API business, monetize APIs, exposing the services that I provide, where do you think I should start? Should I use GraphQL or something else?

James Gregory:

I think that's a good question. It depends on what you're trying to achieve and how much of an understanding you have about your offerings at this point. Like talking a little bit to what Brandon said about how much fidelity is there in your model of the Bowman, do you actually know what your offering is. And technologies again like GraphQL lend themselves well to change quite easily especially if you're developing again mobile apps or heavy user interaction based things. You can get a lot of quick evolution of your interactions without a lot of rework needed on your APIs by using things like GraphQL which can lend itself really well.

James Gregory:

We are seeing companies like GitHub and I believe Twitter as well have started exposing some GraphQL APIs, but I don't think it will ever be the only thing you offer, I think it might be the place you start, but as time moves on and you want to move into something that's perhaps a bit more stable when you understand what your offerings are. I would imagine we will see a lot of organizations offering up REST APIs with complimentary services like GraphQL on top. I don't think we're ever going to be in a space where we have only one way of representing our data to our customers.

Zhamak Dehghani:

I kind of want to let wrap up with your thoughts on what's next in terms of where do you think the industry will move? Where do you think we need more tooling developed? Personally, what the journey for you is going to be in the next few months building APIs and exposing APIs. Brandon do you want to kind of share where your focus is going to be and share your thoughts?

Brandon Byars:

Maybe dime into weeds quite a bit, because of my affiliation with Mountebank, I end up in a lot of conversations with organizations trying to move protocols that were not designed to be exposed externally to more modern approach and how they break it up. And so I'm involved in a lot of conversations around test approaches that help facilitate some of the migrations and that got me to thinking there's a common debate in the API ecosystem around contract first development. A lot of the tool vendors want you to author your design specs and their tools which I don't necessarily agree with but in theory I understand the intention of writing the spec for like your swagger or your ramble first before you write the code. There's a way of sort of productizing the API. A lot of Thoughtworks teams I've been on prefer the other way which is code first and then I'll have some annotations that I can generate a spec with.

Brandon Byars:

And in theory that guarantees consistency. But in practice it doesn't because the annotations oftentimes are a little bit more than comments, especially with some of the more advanced aspects of the spec. And it's a traditional synchronization problem which we've seen in multiple other contexts in the industry. And it feels like neither of those approaches are perfect. And the thing I've liked to poke at is the way that we as Thoughtworks have approached those types of consistency and secret edition problems in the past is through testing. And so I wrote a little proof of concept that would run your tasks against an API and automatically generate a spec on the backside of it. It's imperfect and I don't think the idea is quite right, but it feels like we have prematurely framed this decision as binary and I'd like to sort of push on that to see what we can do to solve the consistency problem without getting locked into the two approaches that are currently being talked about.

Zhamak Dehghani:

Fantastic, we'll make sure we include the links to the tooling you mentioned Mountebank and what you're working on right now. Is this open source as well, but the two that you just mentioned, the proof of concept?

Brandon Byars:

Well I think you're elevating it a bit more than I would by calling it open source, but it is publicly hosted on GitHub.

Zhamak Dehghani:

All right. How about you James? What would be your focus and what's going to be next in your opinion on evolution of API Stein?

James Gregory:

Well next for me I think is, well actually it will be a continuation of what I'm doing right now, which is how do we improve developer experience and develop a tooling around building and consuming APIs and right now a lot of this is focusing on GraphQL mainly because we see building rest API APIs, a lot of effort goes into producing all the different things that we need to safely consume APIs, things like JSON scheme is and types in all JavaScript around what is exposed from our APIs and documentation, Brandon mentioned things like Swagger does a lot of manual labor goes into producing REST APIs that when you flip over to something like GraphQL, it's quite refreshing to not have to necessarily build a lot of that because it's kind of built into the tool, there's a scheme a defined, it's repeating a lot of what we saw before with things like WSDL and SOAP but hopefully with a little bit of stuff that we've learned since then from rest.

James Gregory:

So I'm interested to see how that space evolves. Whether GraphQL is the future or whether it's just another stepping stone, but also how we can port some of that back to our REST APIs, how we can produce better documentation without having to handwrite it and keep it in sync without code. How do we better validate our inputs and our outputs again without having to worry about things drifting or being different to the tests that we're writing and just all of that developer experience there's a lot of energy doing all of that and I think much of it is perhaps unnecessary. And if we can find a better way to do some of this.

Zhamak Dehghani:

It seems like regardless of which direction we go with developer experience and testing and documentation, is that kind of at the center of evolution.

Mike Mason:

Indeed and I think we've run out of time. So I would like to thank everyone for their time. My cohost here Zhamak Dehghani as well Brandon Byars, James Gregory joining us today talking about API design. My name is Mike Mason and thank you for listening.