Making a bot sound like you

I called it Jonson (Jon + JSON). Getting it to know the facts was the easy half. Getting it to sound like me was harder. Models are parrots with a vivid imagination: they repeat whatever you show them, and when they run out, they make things up.

So I stopped hoping it would behave and designed a system that gets a reliable answer out of it. It has four parts: three shape what it says, and the fourth tells me whether any of it worked.

Directives

The directives are the guardrails. Whatever the bot says and however it says it, they keep it honest, on topic and out of trouble. They decide who’s speaking and the length and shape of an answer. They also list the phrases I’d never use and the red flags that should stop it dead.

They cover writing style too, but only as rules, never as samples. I keep the file free of any actual personality, because give a model an example and it’ll parrot it into every answer. I learnt that the hard way.

Personality

The personality comes in two parts.

The personality file

The file lives in my CMS, so I can keep it up to date without touching code. It covers who I am, what I do, who I work with, what I’m working on now, how I write and what I believe in, all in my own words.

Everything I’ve published

My note posts and case studies feed into Jonson word for word. One example gets copied; a few hundred thousand words average out into a style. It learns how I write from my own writing, so it sounds less like a generic chatty Englishman and more like me. And every time I publish, it gets a bit better at it.

Feeding in everything has a cost. The context grows with every post, and older pieces might not sound like me any more. For now, the tests tell me if either starts to bite. If it does, I’ll switch to a mix of featured pieces in full and short summaries of the rest. I won’t go back to summaries alone, though. That’s how it started, and the gaps got filled with "matey filler" I’d never say.

The fact checker

The bot speaks as me, often to people deciding whether to work with me. If it tells one of them I’ve worked with Jony Ive at Apple, that’s a claim made in my name that I’m not there to correct.

And it could easily happen. Apple and Spotify both show up in the bot’s background (in my notes, in my music), and models are good at joining the dots in creative but factually incorrect ways.

So every answer passes a check before it’s shown:

1. Any client it mentions has to be someone I’ve actually worked for.
2. Any employer it mentions has to be somewhere I’ve actually worked.

If it fails, the bot gets one rewrite with the mistake named. If it’s still wrong, the sentence gets cut.

Jonson doesn’t show text as it streams; it waits until the answer is complete. I’d always thought of that pause as dead time, but it turned out to be the perfect place for the fact check. Nobody waits a second longer.

I tested it by replaying every answer the bot had ever given. It flagged 8 of 7,705 (about 0.1%), including the two false claims I’d already spotted myself.

For now it only checks clients and employers, and it can’t tell me what it misses. The directives tell it not to invent numbers or dates, and the test harness looks for invented claims, but the odd generalisation still slips through.

Test harness

Every change I made to the directives or the personality fixed something and risked breaking something else. Chatting with the bot myself told me nothing, because models are a dice throw: the same question can get a great answer one time and a dodgy one the next. So how do you improve your odds?

I test voice and accuracy separately. The voice set checks whether it sounds like me, which only I can judge. The surfacing suite checks whether it says the right words and shows the right content, which is a job for code.

The voice set

22 questions, each asked in a fresh conversation and written out as a review sheet. One run per question, because I’m the one marking it and 132 answers would finish me off.

I tap each answer “me”, “partly” or “not me” and quote the phrase that’s wrong. The first baseline came back 16 “me” and 6 “partly”, all down to matey filler like “fairly obsessed” that I’d never say. After I fed in everything I’d published, all six were gone.

The surfacing suite

22 scenarios, each run six times in fresh conversations, checking what appears under each answer (case studies, client logos, photos, the contact buttons) and what mustn’t. 

For everything it can show, there’s a matching trick question. “Show me your Japan photos” should bring up the photos; “Do you like Japan?” should get an answer, not a slideshow. On top of that, a second model reads every answer against the facts in the CMS, looking for invented claims. A full run takes about fifteen minutes.

Over the course of those fixes, the surfacing suite went from 15 of 22 scenarios passing (68%) to 21 of 22 (95%). What’s left isn’t one stubborn test. A handful of one-in-six failures pop up on each run, in different places each time, and chasing that kind of noise stopped being worth it.

What I learnt

• Keep the rules and the voice separate from each other, and keep the rules abstract.
• Feed it everything you’ve written or at least a decent sample of your favourite content.
• One example gets copied; a few hundred thousand words average out into a style.
• Fact-check every answer before it reaches the reader.
• Mark the output yourself; another model can tell you a claim is wrong, but it can’t tell you a phrase isn’t you.