AI triage of issue reports: what closes itself and what doesn't

In our CRM there is a button that never goes away, whatever screen you happen to be on: it's for reporting a problem. We wanted it always visible for a mundane reason. If you have to go looking for where to report something before you can report it, the report never arrives. People at work have a negotiation, a viewing or a document to send on their minds, not our software. The button has to cost one click and nothing more.

What happens after the click, on the other hand, is the part we have thought about most. Every report lands in a table, and from there an automated AI triage kicks in. Here we explain how it works and, above all, one rule we have decided never to call into question.

What the triage does when a report comes in

The moment a report is created, the triage does three things.

1. It writes a first diagnosis. This isn't gospel: it's a readable hypothesis about what might have gone wrong, set down next to the description from whoever raised the report. It's there for the person who picks it up next, so they don't start from scratch.
2. It classifies the problem. Giving each report a type makes it possible to tell at a glance, for instance, an operation that needs redoing from a genuine software defect.
3. It looks for similar reports that have already been resolved. This is the most useful step. A CRM used every day tends to trip people up in the same places, and sometimes someone reports something that has already happened and already been sorted.

When a report closes itself

If the triage finds a precedent, it looks at how it was resolved. Every remedy can be marked as reusable or not. Reusable means it can be applied again, exactly as it was: an operation to repeat, for example.

In that case the report closes itself, applying the same remedy. Whoever raised it isn't left waiting on something we already know about, and we don't waste time re-diagnosing a problem that already has an answer.

Put like that, it sounds like the easy part. In fact it's the part that most needs fencing in, because a system that closes things by itself is convenient right up until the day it closes the wrong one.

The rule that isn't up for negotiation

We have set two constraints, and we treat them as non-negotiable.

The first: every closure records the remedy. No report is closed without leaving a trace of what was done. "Resolved" isn't enough. If the remedy is missing, the next triage has nothing to recognise, and the system's memory empties out precisely where it would be needed.

The second: a remedy that is a code fix is never reusable. Never, not even when it would seem convenient.

The reasoning is this. An operation to repeat is, by its nature, something that may be needed again: redoing it is the cure. A code fix, by contrast, ought to put the problem to bed once and for all. If the same fault reappears after the code has been fixed, we are not looking at a known case: we are looking at a regression. Something we had fixed has broken again, or the fix wasn't the right one.

If that remedy were marked reusable, the triage would do the most reasonable thing from its point of view: recognise the precedent, see that it had been resolved and close the new report. Silently. The log would say everything was fine and the defect would stay in the software, with the added problem of looking as though it had already been dealt with. It's the kind of error that makes no noise, and that is exactly why it's the most expensive.

That's why a regression has to stay visible. A report that resembles a code fix doesn't close itself: it stays open, with the precedent alongside it, and someone has to look at it.

Who touches the code, and how

We make code fixes in a working session with Claude, following a written procedure. The procedure must be read before touching the reports, not afterwards and not halfway through.

This isn't red tape. The procedure is where we have written down precisely the rules in this article: how a report is closed, what goes into the remedy, when a remedy may be marked reusable and when it may not. A session that starts without reading it risks doing the most natural thing, namely fixing the problem and closing it, perhaps marking the remedy as reusable because "it worked". That is exactly the behaviour we want to rule out.

Reading it first means that whoever works on the code, person or model, starts from the same contract: the fix gets made, the remedy gets written, and the door to automatic closure for that kind of fault stays shut.

What we take away

Three things, which hold true beyond our CRM too.

• Automation suits things that repeat by nature. An operation to redo is an excellent candidate for automatic closure. A software defect is not: if it comes back, that's news.
• Memory is only worth something if it's written down. The triage recognises precedents because every closure leaves a remedy behind. Without that written remedy, the search for similar cases turns up nothing useful.
• The rules that matter belong where the work happens. A procedure to read before you start costs a few minutes and removes the most likely mistake, which isn't a technical one but closing things too hastily.

The button is still there, on every screen. What we have learnt to look after is what happens next, and above all what must never happen on its own.