Enable javascript in your browser for better experience. Need to know to enable it? Go here.

Exploring the potential of ChatGPT as a coding partner

Part one

Ever since I first heard about ChatGPT — which wasn’t that long ago — it’s been impossible to avoid claims about its revolutionary potential in which it will change the way we work forever. Among many things, I was told that it's a completely different way of finding information and that it can even be recognized as the only source of knowledge in the foreseeable future. I also heard that it can code.


I decided to check it out and, filled with hope, I wanted to see how it performs as a programming support tool — perhaps, I thought, it might even be something more. I wanted to set up a pair programming session with it, wondering if it would work. I wanted to treat it as if it were someone I spend a few hours a day writing code with. Without much thought, I decided I would work on two example use cases with it that were roughly what we might expect when implementing a typical new feature. The two use cases were of varying complexity and both use C# as the programming language. While ChatGPT was able to write reasonably complex code, my session with it lacked the mutuality and co-creation that I get from my best pair programming experiences. To maintain the authenticity of the interaction, I decided to keep the conversation intact and added my own comments from the session, which took place using a version of ChatGPT from Jan 30. 


Use Case 1: Simple domain model


In this scenario, I wanted to test ChatGPT's ability to model a basic domain concept and assess its response to requirement changes. I also wanted to evaluate the effectiveness of our collaboration.


I started giving the goal and the requirements. After a few seconds I got the proposed solution. Pretty impressive.


          R: Can you use C# to model a storage box?


         Here are the requirements:


        Storage box can contain only strings.


        Consumers can add strings to the box. 


        Box has limited capacity and cannot contain more than 10 strings and 

        total length of all strings should not exceed 200 characters.


        We should end up having properly isolated domain classes.


       ChatGPT: Yes, it is possible to use C# to model a storage box. Here is an

       example implementation:


The example implementation was pretty accurate. We could argue that using constants instead of read only fields, favoring properties over methods or choosing a more functional approach in this example would be a better choice. Anyway, this was a very good starting point.


Then, I wanted to add some additional requirements to see how it would react.


          R: The strings that we put into the box should not be longer than 100

          characters and should not contain the "x" character.


          ChatGPT: Here's an updated version of the `StorageBox` class with the

          additional requirements:


The updated code was functional even though I didn’t like the growing validation section in the “AddString” method. I wanted to check whether it could suggest something to deal with it.


I added new requirements to see if it could propose an abstraction to avoid duplicating validation logic.


          R: Let's also add a new method to replace the last string in the box with

          a new one.


          ChatGPT: Here's an updated version of the `StorageBox` class with the

          new `ReplaceLastString` method:


The result was a bit disappointing. It simply added duplicate validation logic to the second method. It didn’t spot the abstraction here. I was curious how it ‘felt’ about the code it wrote though; I asked it to do a review of its work.


           R: Can you review the code you wrote?


           ChatGPT: Yes, here is a review of the code:


  • The `StorageBox` class implements the storage box model that contains a list of strings and enforces the constraints of maximum capacity, total length, and string length.

  • The `AddString` method adds a string to the storage box if it meets the constraints. If not, it throws an exception with an appropriate error message.

  • The `ReplaceLastString` method replaces the last string in the box with a new string if it meets the constraints. If the box is empty, it throws an exception.

  • The `GetTotalLength` method calculates the total length of all strings in the box.


          The code is well-structured and easy to understand. The constraints are

          properly enforced and the error messages are clear. The code can be

          improved by adding some additional input validation to ensure that the

          inputs are not null or empty. Additionally, the code can be made more

          readable by breaking down some of the complex conditions in the

        `AddString` and `ReplaceLastString` methods into separate methods.


Okay, it appeared that the validation rules were somewhat too complex. However, I didn’t agree with its proposed solution of separating the logic into distinct methods — there are other, better options. Nonetheless, I chose to adopt its approach to shorten the code.


          R: How can we simplify this?


          ChatGPT: One way to simplify this code is to extract the input validation logic

          into separate methods to make the main methods cleaner and easier to

          understand. Here's an updated version of the code:

I wanted to make the missing abstraction more obvious so I added a new requirement to show that keeping validation rules in separate methods wouldn’t help us here.


          R: Let's add one more concept. A shelf on which we can put strings. At the

          moment, there are no restrictions on the capacity of the shelf. The same

          validation rules should apply for strings. Let's start with one method of

          arranging the strings on the shelf.


          ChatGPT: Sure! Here's an updated version of the code with the added

         `Shelf` class:


I wasn't satisfied with the revised implementation. Despite specifying that we were dealing with the same strings, ChatGPT failed to abstract it, leading to duplicate logic. I would have also hoped for it to pose questions to clarify important design details. Nonetheless, I decided to request another review from it.


          R: Can you review the code you wrote?


          ChatGPT: Sure, let me review the code I wrote:


  • The code is well organized and easy to understand.

  • The `Shelf` and `StorageBox` classes are properly isolated and don't depend on each other.

  • The validation rules for the strings are implemented in a single `ValidateInput` method, which makes the code more maintainable.

  • The code has proper error handling with meaningful error messages.


          Overall, the code looks good to me!


The outcome was quite disappointing. It seemed that for ChatGPT, identifying patterns and structure might be a challenge. I offered a specific idea on how to simplify the solution.


          R: It could be improved by abstracting out the concept of an Item that we

          reuse in both classes. In that case you won't need to duplicate the

          validation logic. Moreover, making the Item class immutable and invoking

          the validation logic during instantiation would protect us from putting the

          instance in an invalid state. What do you think?


          ChatGPT: Yes, that's a great suggestion! By abstracting out the concept of

          an `Item`, we can reduce code duplication and improve the overall design

          of the code. Here's an updated version of the code with the `Item` class:

I attempted to introduce some additional enhancements to the implementation, such as adding a factory method to the "Item" class or replacing exceptions with status codes, but I faced some difficulties in communicating these suggestions effectively to ChatGPT. As a result, the implementation became messy, and I ultimately decided to stop at that point as I felt it would be easier to do it on my own.


Use case 1: Summary


On one hand, I was impressed with the initial prototype ChatGPT was able to create from my basic requirements. However, I was disappointed with the limited interaction and collaboration it offered. It was a one-sided interaction with me dictating what I wanted and ChatGPT coding accordingly, without any real discussion or exchange of ideas. As the solution became more complex, communicating with it became difficult and I had to be very specific with my instructions. This was not a very efficient process, and I felt I could have completed the task faster on my own. Despite the simplicity of the task and the absence of dependencies, ChatGPT also did not provide any valuable insights on how to simplify the solution by abstracting things out.


In the second part of the article, I will test how effective ChatGPT is when dealing with a more complex scenario. Finally, I will share my overall thoughts on the experiment.

Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.

Keep up to date with our latest insights