Historically, when it came to deciding whether we were interacting with a human or a computer, we had the Turing Test. Created in 1949 by Alan Turing himself, there was once even a prize for whoever could create an artificial conversational entity (ACE) capable of successfully duping enough of the judges into believing that they were chatting with a real person.
As fate would have it, interest in the Turing Test ebbed, the prize became defunct, and feverish reports on Star Trek-style computers that could interact with us just like humans dwindled.
Seventy-five years later, however, the problem migrated off the pages of far-fetched sci-fi novels and it has now flooded across most, if not all social media platforms. Facebook, Instagram, X, Reddit, TikTok, and even – or perhaps especially – LinkedIn are now drowning under AI-generated content from accounts posing as humans, turning what used to be an after-dinner academic conversation piece into an irritating continual Bot or Not? sanity check.
But one thing has evolved: instead of the Turing Test, we now have… the Cupcake Test.
Where judges might once have asked Turing Test contenders about their family, hobbies, or travel plans, the Cupcake Test is form of prompt injection – an attack where a user issues a strange instruction designed to reveal that the “person” they are interacting with is, in fact, a conversational AI agent. In the inevitable arms race that has ensued, some chatbots are now smart enough to disregard the “disregard” command, or to even joke about attempted prompt injections, but not all, and that brings us to March 2026.
Hi, Henry!
Up to present, the majority of AI generated content has been in text format, but with the rapid advancements in synthetic speech, we are now seeing AI-powered scam calls, and a viral video posted on TikTok and Instagram by Lavizrap13 demonstrates that prompt injections for verbal scam attempts can be just as effective – at least for now.
Call-centre-style scams rely on the call-and-response nature of customer service interactions. In different words, as in Lavizrap13’s video, these interactions tend to be highly scripted. They start with their opening gambit ( “you’re owed money!”), proceed to closed or narrow questions (“can you hear me?”, “what’s your name?”) and carry on in a carefully choreographed sequence. Staying firmly in control of topics and turn-taking minimises the risk that the AI will be asked something unexpected that causes it to unwittingly reveal its true nature.
Perhaps most importantly, however, Lavizrap13’s example demonstrates additional elements of social engineering. Not only is the voice itself particularly realistic, it opens with a natural-sounding question that suggests the possibility of slightly difficult – and therefore delayed – communication: “Hi Henry, it might have been me who called you earlier. Can I just check, can you hear me okay?” It also weaves in the recipient’s name, and we can even hear background noises hinting at a busy call-centre or office-like environment. Convincing stuff – even more so if you’re in a rush and not primed to suspect a scam.
Processing speeds
So how do we tell that this is AI? Well, there is the rather obvious and very cheerful digression into cupcakes, of course, complete with hashtags, but there is another key tell, and it might not be the one you expect.
Latency.
Humans respond in milliseconds – quite literally. As we’re hearing the speaker’s words, we’re already modelling what they’re most likely saying next, drawing on past experiences, flexibly scaffolding a response based on favoured linguistic templates, and then just waiting for the precise moment to begin. By contrast, AI models are still leaving response-time gaps that we find notable, and suspicious. These can sometimes be accounted for by, say, noisy call-centres and distractions (remember the “can you hear me?” gambit) but only for so long and sure enough, throughout this call, the pauses before the AI answers are particularly prominent, especially when compared with the much more natural durations of Lavizrap13’s responses.
And this all takes us back to HackaCon. We know that AI is very good at monologues such as voicenotes, and that telltale cracks start to show in live conversations like scam-calls. We also know that using AI to generate behaviourally human-like, spontaneous-sounding conversation is nearly, if not fully impossible.
Right now.
We also know that agentive and conversational AI is being used by hostile state actors, organised crime groups, and petty criminals to create or manipulate audio, image, and video content in a practice now well-known as deepfaking. In fact, deepfake fraud attempts have surged by more than 1,300% in 2024. Given the many advantages offered by AI-powered crime, malicious users will inevitably keep pushing the boundaries of this technology for their own gains, in turn forcing ordinary people to run impromptu CAPTCHA tests every time they open a message or answer a call.
This is why HackaCon aims to understand and test the current conversational AI state of the art: seeing what can be done today tells us where things are likely to go tomorrow.