The 4 things I try to keep front of mind when using AI assistants
The web is full of long lists of things you should remember when using AI assistants, often with contrived acronyms and colourful PDFs so dense with tips and tricks you need a magnifier to read them.
Improved models and the advent of reasoning modes/models have made a lot of the ‘prompt engineering’ guidance (e.g. ‘ask it to think step-by-step’) less essential. Instead of trying to craft the perfect prompt, you can increasingly just start explaining what you’re trying to achieve and the AI assistant will ask for any additional info it needs to help.
That said, there are four things which I always try to keep front of mind when using an AI assistant:
1.) It makes stuff up (so fact check)
Although some newer reasoning models appear to be more prone to hallucination than older, non-reasoning models, hallucination rates appear to be generally trending downward, thanks to techniques like RAG.
There’s a risk that’s going to make us less vigilant.
Hallucinations are an inevitable feature of next-token prediction machines, so we need to keep verifying AI assistants’ outputs whenever factual accuracy matters.
I’m terminally optimistic, but I hope that the fundamental unreliability of AI assistants might prompt us to be more diligent in assessing the veracity of everything we read, see and hear.
Some of the critique of AI hallucination gives the impression that the pre-AI internet was a bastion of factually accurate information, which is not quite how I remember it…
2.) It reflects biases (so mitigate)
All humans have biases which inform the things we think, say, write and photograph, often unconsciously. Generative AI, trained largely on human-written text and audio/visual media, have internalised those biases in their model weights. Bias is not going away (in humans or machines). The notion of a bias-free AI is absurd (see Executive Order 14179, which states “we must develop AI systems that are free from ideological bias” and last week’s America’s AI Action Plan which suggests “the government only contracts with frontier large language model (LLM) developers who ensure that their systems are objective and free from top-down ideological bias”).
In the real-world, AI assistants are advising women to ask for lower salaries than men with identical qualifications and producing less empathic responses to Black and Asian Reddit posters seeking mental health support.
Being mindful of the inevitability of bias in the outputs of generative AI models should be priority number one. Attempting to minimise and mitigate biases should be priority number two.
How you prompt a model can go some way to mitigating bias (simply adding the word ‘diverse’ to a prompt can help).
Your choice of AI assistant is another variable (I favour one which doesn’t check what Elon Musk thinks before responding).
However, it ultimately comes down to ensuring the unfiltered output of an AI assistant isn’t being used for high stakes decisions and is always treated as a contributor with biases which need to be mitigated by the experience and judgement of real people following robust processes (as those people inevitably also have biases).
3.) It needs context (so provide)
AI assistants are (mercifully) not mind readers. Whilst their persistent memory is increasing (see #5 of my AI predictions for 2025), they’re still not always great at working out what information is pertinent to the task at hand.
When demoing deep research functionality during a recent presentation, I asked the audience for a candidate research topic. The prompt we arrived at (“plan a holiday to lake garda, taking mum who is 70, include italian cooking course and a boat, 5 star hotel. mum's paying”) resulted in a much richer response than simply “plan a holiday to lake garda” would have elicited.
Thankfully, newer models are getting better at asking for more context, rather than just busking it with what they’ve got.
Still, providing the most relevant context in the first place can save some unnecessary back and forth.
4.) It’s trained to please (so pinch of salt)
The system prompts of most AI assistants instruct them to be helpful and RLHF tends to reward agreeable responses.
The result is AI assistants which bias towards telling you what your prompt indicates you want to hear.
ChatGPT had to roll back an update that dialled this up too much.
Keeping this inclination in mind is important, especially when seeking input on high-stakes decisions.
Partial mitigations (not silver bullets) include using open-ended, non-leading prompts (e.g. ‘Is this better or worse?’) and asking it to lay out pros and cons.
I was demoing ChatGPT’s vision mode during a recent presentation. I asked it whether the very obviously creased shirt I was wearing looked like it had been ironed. Its response? ‘Your shirt looks pretty good! I don’t see any wrinkles, so if you ironed it, you did a great job.’ It’s possible this was a failure of ChatGPT’s computer vision but I think it’s more likely it was telling me what it had inferred I wanted to hear.
I’ve previously shared how I use AI as a writing assistant, asking it for feedback on drafts. I find this really helpful but constantly have to remind myself of AI assistants’ people-pleasing tendencies. After showering me with praise for a draft of this blog post, I asked Claude whether it was just telling me what I wanted to hear. Its response? “Ha! You got me. That’s a perfect real-world demonstration of your fourth point”.
However, even this feedback necessitates a pinch of salt*. Claude is inferring from my question that I’m after more critical feedback, so that’s what it delivers (for an extreme example of how AI chatbots can be inadvertently led by their human interlocutors, check out The Times’ One day in the life and death of an AI chatbot).
If I were to distil my four reminders down to just one (maybe you’ve got a smaller sticky note), it would be to treat AI assistants as fallible and to avoid suspending your own critical faculties whilst using them. It is essential to treat their output as input to your own thinking and never outsource your judgement.
For more on this, I recommend this article by Nick Bergson-Shilcock of the Recurse Center. It’s ostensibly about the impact of AI on software development but the insights have wider application. My favourite quote:
“You should use AI-powered tools to complement or increase your agency, not replace it”.
*For the record, I’ve left in the meandering opening, overly-long Lake Garda example and slightly preachy tone. You’re welcome.