I have spent a lot of time getting frustrated with LLMs.

One minute they can look incredibly smart. They can help design an architecture, explain code I have never seen before, suggest approaches I had not considered and generate a decent amount of useful work very quickly.

Then five minutes later they do something that makes you think: how can something this clever be this bloody stupid?

The more I have worked with them, the more I think that is the wrong question.

The real problem is that I have spent my career working with deterministic systems, then started expecting a probabilistic system to behave the same way.

And I was also forgetting something even more important.

I knew things the model did not.

A 16:9 split infographic comparing the rich project context an experienced developer carries in their head with the much smaller set of information supplied to an LLM. The gap between the two is labelled Missing context can lead to a different decision, reinforcing that some apparently poor AI decisions come from information the model was never given.
Experienced developers often judge an LLM using context they already know but never actually supplied to the model.
Text version of the visual

Developers may implicitly know architecture decisions, business constraints, coding standards, fragile dependencies, rejected approaches, customer intent and the real definition of done. An LLM may receive only a prompt, ticket, selected files and partial state. That information gap can materially affect the decision it produces, although missing context is not the only reason an LLM can be wrong and more context does not guarantee correctness.

Developers carry more parameters than we realise

When an experienced developer looks at a system, they are rarely looking only at the code in front of them.

You might also know:

  • why an architectural decision was made six months ago;
  • which part of the system is about to be replaced;
  • which business rule is weird but absolutely cannot change;
  • which dependency is fragile;
  • what your team's coding standards are;
  • which approach you already tried and rejected;
  • what the customer actually meant when they wrote a vague requirement;
  • and what "done" really means in your environment.

A lot of that knowledge is not even written down.

It is just sitting in your head.

Then we give an LLM a ticket, a few files and a prompt and get annoyed when it does not make the same decision we would have made.

From the model's point of view, some of the parameters are simply missing.

That does not mean the answer it gives is good. It does mean we should be careful about calling the model stupid when we have not given it the same information we are using to judge it.

Deterministic and probabilistic are different tools

A normal program is built around explicit rules.

At a very simple level:

known inputs
+ explicit rules
+ known state
= predictable result

That is what developers are used to.

LLMs are different. They are probabilistic systems. They generate an answer based on the information available to them and the patterns they have learned.

That makes them extremely useful when the solution space is open.

Things like:

  • architecture and design exploration;
  • explaining unfamiliar code;
  • generating content;
  • drafting specifications;
  • finding gaps in an idea;
  • comparing implementation approaches;
  • critiquing a design;
  • and creating candidate solutions.

I have started thinking of them as very powerful design engines.

That is where they can be brilliant.

But I do not want to use probability for work that already has a stable answer.

Stop using an LLM where a script will do

Imagine I want to know whether every object in a repository follows a naming rule.

I could ask an LLM to inspect them and tell me what it thinks.

Or I could write a deterministic check.

for item in objects:
    assert follows_naming_rule(item)

Now I have changed the nature of the problem.

Instead of:

I think these comply.

I can get:

47 checked
47 passed
0 failed

That is a much better engineering boundary.

The LLM can still help me design the rule, understand the problem, create the first version of the checker or explain failures.

But once the rule is known and repeatable, I would rather move that work into something deterministic.

The question I started asking instead

For a while I kept asking how to make the AI smarter.

I think the better questions are:

What am I asking the LLM to do that should never have been probabilistic in the first place?

And:

What information do I know that the LLM does not?

Those two questions changed how I started building with AI.

Instead of trying to make the model responsible for everything, I started thinking about the engineering system around the model.

What should be deterministic?

What needs durable state?

What should be verified independently?

What context should the model receive before it starts work?

What should be remembered so I do not have to explain the same thing again tomorrow?

And what should happen when the model gets something wrong?

LLMs are not the engineering system

This is probably the biggest change in how I think about them now.

The LLM should not be the entire engineering system.

It is one part of it.

Use the LLM where reasoning, design and generation are valuable.

Use deterministic tools where the answer can be checked exactly.

Give the model the context it actually needs instead of assuming it somehow knows what is in your head.

And then verify the result rather than trusting the confidence of the answer.

That sounds obvious written down.

It was not obvious to me when I started.

I had to get annoyed with these systems a lot before I properly understood what I was asking them to do.

That distinction between probabilistic reasoning and deterministic engineering has become the foundation for almost everything I have been building since.

And it leads to the next question I had to answer:

If a task can be made deterministic, how do you decide when to stop asking the LLM and build the tool instead?