Making a bot sound like you

I called it Jonson (Jon + JSON). Getting it to know the facts was the easy half. Getting it to sound like me was harder. Models are parrots with a vivid imagination: they repeat whatever you show them, and when they run out, they make things up.

So I stopped hoping it would behave and designed a system that gets a reliable answer out of it. It has four parts: three shape what it says, and the fourth tells me whether any of it worked.

Directives

The directives are the guardrails. Whatever the bot says and however it says it, they keep it honest, on topic and out of trouble. They decide who’s speaking and the length and shape of an answer. They also list the phrases I’d never use and the red flags that should stop it dead.

They cover writing style too, but only as rules, never as samples. I keep the file free of any actual personality, because give a model an example and it’ll parrot it into every answer. I learnt that the hard way.

One gap took me longer to spot than it should have. Ask a language model to write something and it really wants to help, even when it's meant to be someone else; the persona drops and the assistant underneath gets on with the job. My scope rule only caught questions that had nothing to do with me, so a request for a piece of writing on a subject close to my life sailed straight through.

Staying on topic and being asked to do a task are different problems, and I'd only written a rule for one of them. The fix was a single line in the directives: nothing gets made for the visitor here. Jonson turns the request aside lightly and points back to the work, whilst the same subject asked as a question about me still gets a proper answer.

Personality

The personality comes in two parts.

The personality file

The file lives in my CMS, so I can keep it up to date without touching code. It covers who I am, what I do, who I work with, what I’m working on now, how I write and what I believe in, all in my own words.

Everything I’ve published

My note posts and case studies feed into Jonson word for word. One example gets copied; a few hundred thousand words average out into a style. It learns how I write from my own writing, so it sounds less like a generic chatty Englishman and more like me. And every time I publish, it gets a bit better at it.

Feeding in everything has a cost. The context grows with every post, and older pieces might not sound like me any more. For now, the tests tell me if either starts to bite. If it does, I’ll switch to a mix of featured pieces in full and short summaries of the rest. I won’t go back to summaries alone, though. That’s how it started, and the gaps got filled with "matey filler" I’d never say.

The fact checker

The bot speaks as me, often to people deciding whether to work with me. If it tells one of them I’ve worked with Jony Ive at Apple, that’s a claim made in my name that I’m not there to correct.

And it could easily happen. Apple and Spotify both show up in the bot’s background (in my notes, in my music), and models are good at joining the dots in creative but factually incorrect ways.

So every answer passes a check before it’s shown:

1. Any client it mentions has to be someone I’ve actually worked for.
2. Any employer it mentions has to be somewhere I’ve actually worked.

If it fails, the bot gets one rewrite with the mistake named. If it’s still wrong, the sentence gets cut.

Jonson doesn’t show text as it streams; it waits until the answer is complete. I’d always thought of that pause as dead time, but it turned out to be the perfect place for the fact check. Nobody waits a second longer.

I tested it by replaying every answer the bot had ever given. It flagged 8 of 7,705 (about 0.1%), including the two false claims I’d already spotted myself.

For now it only checks clients and employers, and it can’t tell me what it misses. The directives tell it not to invent numbers or dates, and the test harness looks for invented claims, but the odd generalisation still slips through.

Test harness

Every change I made to the directives or the personality fixed something and risked breaking something else. Chatting with the bot myself told me nothing, because models are a dice throw: the same question can get a great answer one time and a dodgy one the next. So how do you improve your odds?

I test voice and accuracy separately. The voice set checks whether it sounds like me, which only I can judge. The surfacing suite checks whether it says the right words and shows the right content, which is a job for code.

The voice set

22 questions, each asked in a fresh conversation and written out as a review sheet. One run per question, because I’m the one marking it and 132 answers would finish me off.

I tap each answer “me”, “partly” or “not me” and quote the phrase that’s wrong. The first baseline came back 16 “me” and 6 “partly”, all down to matey filler like “fairly obsessed” that I’d never say. After I fed in everything I’d published, all six were gone.

The surfacing suite

The second set checks what turns up under each answer rather than how it sounds. There are 46 scenarios now, each run six times in a fresh conversation, because the same question won't always get the same answer.

Each one checks that the right things appear (case studies, client logos, photos, the contact buttons) and that anything that shouldn't appear stays hidden. Everything Jonson can show has a trick question to go with it. "Show me your Japan photos" should get a slideshow; "Do you like Japan?" should get a written answer and nothing else.

A handful of scenarios ask Jonson to make something for the visitor instead of talking about me, alongside a couple of decoys to make sure that rule doesn't start refusing honest questions. On top of all that, a second model reads every answer and checks it against the facts in the CMS, looking for anything I didn't actually do.

When I started, 15 of the original 22 scenarios passed. The latest full run passed 268 of 276 runs, about 97%. The failures that are left are intermittent, failing roughly one run in six and somewhere different each time, so I've stopped chasing them.

What I learnt

• Keep the rules and the voice separate from each other, and keep the rules abstract.
• Feed it everything you’ve written or at least a decent sample of your favourite content.
• One example gets copied; a few hundred thousand words average out into a style.
• A persona is only as strong as its refusals; you need a rule for tasks as well as topics
• Fact-check every answer before it reaches the reader.
• Mark the output yourself; another model can tell you a claim is wrong, but it can’t tell you a phrase isn’t you.