When the evidence runs out, don’t ask AI to guess

One of the most dangerous things about generative AI is also one of its most impressive capabilities. It can keep going. Give it incomplete information and it can often produce a coherent explanation anyway. That is useful when drafting. It is much less useful when investigating how an organisation actually works. I encountered this while helping a manufacturing business examine several important processes. We used ChatGPT to analyse process documentation, structure interviews and compare what was supposed to happen with what employees actually did. It was useful. But in several places we reached a point where the organisation simply did not possess enough evidence to know what was happening.

The wrong response would have been: “ChatGPT, what do you think the cause is?” A much better question was: What evidence would we need to distinguish between the possible explanations? That changed the role of AI. Instead of being asked to fill the gap with a plausible answer, it helped us design the next part of the investigation. What should we observe? What data should we collect? How many examples might we examine? What would support or contradict a particular hypothesis? Small studies were then carried out and the results fed back into the analysis.

This is a useful discipline well beyond process improvement. AI can help us reason about evidence. It can identify inconsistencies. It can generate hypotheses. It can suggest questions. But a hypothesis generated fluently remains a hypothesis. Organisations need to resist confusing the quality of the explanation with the quality of the evidence supporting it. So when AI reaches the edge of what you know, do not ask it to make the uncertainty disappear. Make the uncertainty explicit. Ask: What would we need to find out before we could answer this properly? Then go and obtain the evidence. That is a much more powerful use of AI than asking it to guess convincingly.