ChatGPT Can Write Code. But Can It Engineer Software?

2022 will be known as the year AI

Growing up as someone who wanted to make games but couldn't draw, I fantasized about having a machine that could take my words and turn them into pictures. It's hard to believe that such a program was still a pipe dream less than a year ago. Now, we have three of them.

Like many, I've spent hours this past week glued to the screen, trying out ChatGPT. Seeing what it can do. Watching for news about its IQ (83, according to an online test) and its SAT scores (1020).

Some are trying it for work. Others, for entertainment. Nils has been experimenting hard: he has discovered it can act like a terminal, and is using it to solve Advent of Code. He's even created a new Twitter account for ChatGPT-generated stories.

There's a lot to be said about this. What can we build with it? Will AIs keep getting better as fast as they have this year? What does it mean for our jobs? Someone let it control his computer — at what point does it become deathly dangerous?

But that's not what this newsletter is about. Here, we care about one thing only: can it design good software? Let's ask it!

It knows about using more precise types:

And it even immediately spotted a bug in a 7-line program that the vast majority of programmers can't see in 2 minutes. (We use it in the course, so no spoilers.)

Outside of that though, it's fared quite poorly. It knows using strings to represent arbitrary E-mails is bad, but when I ask it to improve a User class that contains such a field, it generates a bunch of validators. I tried with several prompts to get it to refactor the code ((request.startDate.getTime() - (new Date().getTime())) / (1000 * 3600 * 24) < 3. It split the code into a bunch of well-named variables, which only makes the code longer. With some handholding, it can do better, but never comes close to my preferred solution of Time.until(request.startDate.getTime()) < Time.days(3).

On the other hand, its knowledge of design patterns looks strong:

Overall, ChatGPT's skills are just a reflection of the Internet. Not very good.

Nonetheless, based on what I've seen so far, I will recommend running future code reviews through ChatGPT. The Internet still knows a lot of things.

Now let's go fill the Internet with training data of better software design.