I tried to use AI to help with software development tasks back when it was ChatGPT was released. I didn't find much use for it then. Here's another look at it.

A few weeks ago I had a success with using a chatbot to write some code to extract a few things from a website using Puppeteer, which I'd never used before so I wasn't familiar with its API or setup boilerplate. Essentially, I just gave the chatbot an HTML fragment and told it to generate some code to extract a certain piece of data from the fragment. It did what I asked and gave me some boilerplate I was easily able to edit into a full solution to the problem I had. I was pleased with that.

But that's really just a slightly optimised copy and paste workflow from the getting started guide. It's not a hard problem. In terms of solving real problems it's still not proving useful to me in a professional sense. A lot of my job recently is tracking down obscure concurrency bugs. These are hard problems that require a lot of effort to figure out. A typical flow for this is to read logs, read code, read more logs, put in more logging statements, set up test cases to run repeatedly until one fails, then read more logs and hope I can figure out 1. what's going on and 2. how to change it so it works better. Chatbots are useless for this, for a number of reasons.

But I do occasionally try to use chatbots when I'm looking at errors (usually from third party code) that don't make sense to me. So far, I am yet to find a situation where it has actually helped. I'm sure it could help with easier problems, but I don't need help with those. Or so I thought (on both counts). Recently I had a temporary failure of brain function and gave it an easy one, which it also flubbed.

I had a piece of code that looked like this (classes renamed):

var manufacturer = session.Get(manufacturerId);
...
var car = session.Query().FirstOrDefault(c => c.Manufacturer != null && 
    c.Manufacturer.Id == manufacturer.Id);

This was giving me a NullReferenceException propagating up from the ORM on the second query, wrapped with the message Evaluation failure on null.Id. The reason is pretty obvious - manufacturer was null - but my brain wasn't working and I was fixating on the c.Manufacturer != null && c.Manufacturer.Id part.

So I gave it to a chatbot to tell me why I was seeing this error. Here's what it told me:

The error occurs because c.Manufacturer can be null, and you are trying to access c.Manufacturer.Id, which results in null.Id, causing NHibernate to fail during query evaluation.

Why is this happening? c.Manufacturer might be null: When NHibernate evaluates the expression, it does not short-circuit the logical condition c.Manufacturer != null && c.Manufacturer.Id == manufacturer.Id in the same way C# does at runtime. Instead, NHibernate tries to translate the entire lambda expression into SQL. Since c.Manufacturer is sometimes null, it attempts to access c.Manufacturer.Id, leading to the error "Evaluation failure on null.Id".

Solution Use safe null-checking by restructuring your query:

Option 1: Use a Separate Query Condition

var car = session.Query()
               .Where(c => x.Manufacturer != null)
               .FirstOrDefault(c => c.Manufacturer.Id == manufacturer.Id);

This ensures that only non-null Manufacturer values are evaluated for Id.

It also gave two other code snippets that were equivalent to the first.

There are two problems with this answer:

1. It has failed to identify the real problem, that manufacturer might be null.
2. Everything it has said here is incorrect. It is not at all true that x != null && x.Y == something is an unsafe way to access x.Y, and rewriting it to use Where(predicate1).FirstOrDefault(predicate2) is no more or less safe than .FirstOrDefault(predicate1 && predicate2).

After telling it that this code still produced the same error we went around in circles and did not get closer to a resolution.

I see people online saying that chatbots are revolutionary and solving real programming tasks on their own. Beyond very basic tasks, I'm not seeing it.