Readers voted an AI-written text the most human. What I think about it now

At the end of 2024 the journalist Viktorija Žižiūnienė and I ran an experiment that still nags at me. The idea was simple: find out whether people could tell a text written by artificial intelligence (AI) from one written by a person. We even had a bet on it. Viktorija was rooting for the writers and was sure the machine would not beat them. I was more on the AI side. But I had my ceiling too. I thought one AI text might sneak into the top five, and on a very optimistic day into the top three. That it would come first never crossed my mind.

The whole thing started earlier, and not very nicely on my part. When the first widely known version of ChatGPT appeared, I used it to write a piece about diamonds and investing in them, sent it to Viktorija and asked her to edit it. I did not say who wrote it. She spent a good half day on that text, and when she was done she asked who on earth had written it. I told her. She was furious. But she is a good person, she forgave me quickly, and soon we were sitting down and working out how to put the same question not to one person but to a few hundred.

Tool

Twelve separate ChatGPT assistants, one per writer, plus one plain free ChatGPT with no preparation at all

Date run

Results published 02/12/2024

Task

25 texts on the theme "Why do I write?", up to 200 words each. Readers had to mark each one as written by a person or by AI

Time

Two weeks on a single instruction, seven to eight A4 pages. About 20 seconds per generated text

What was measured

How many of the 25 texts each voter marked correctly, and what share of voters assigned each text to a human. 233 people voted

How it was done

1. I spent two weeks writing a single instruction. It came to seven or eight A4 pages.
2. From it I built twelve AI assistants, one per writer taking part.
3. The assistants were fed only publicly available texts by those people: five samples each that they sent us, plus whatever we found ourselves on social media, in magazines and in newspapers.
4. Twelve professionals (writers, journalists, communications, advertising and PR specialists) each wrote up to 200 words on the theme "Why do I write?". The assistants never saw those texts, they were kept separately.
5. Each assistant generated its own version in about 20 seconds. No human touched them afterwards, even the mistakes were left in.
6. One more text came from plain free ChatGPT with no preparation at all. That made 25 texts in total: 13 from machines and 12 from people.

Then we loaded them into a separate platform and sent them out to be judged. 233 people voted, from experienced editors and publishing house staff to people with no connection to writing at all: IT specialists, engineers, hairdressers, builders, insurance brokers. One task. Read it and mark who wrote it, a person or AI.

The result

The text that the largest number of people marked as human-written came from a machine. It was the assistant copying the style of a writer who covers personal finance.

76.8 %The share of readers who decided that the most human of all 25 texts was written by a person. It was written by an AI assistant.

Second and third place went to actual humans: one at 74.2 %, the other at 72.8 %. In total, five AI texts were assigned to a human by more than half of the readers.

But first place ended up surprising me the least. The average surprised me more. Out of 25 texts, people got 13.66 right on average, an accuracy of 54.64 %. The worst score was 5 out of 25. The best was 22, and exactly one person managed that. Nobody marked all of them correctly. The number looks harmless until you remember that 13 of the texts came from a machine, so marking everything as AI without reading anything would already have scored 13.

And one more small thing, still my favourite. Some of the voters admitted they had ignored our request and quietly checked the texts with AI detection tools. Those tools called all 25 texts human-written. All of them.

What the experiment does not prove

The honest part, because back then I talked about it too little.

This was not a scientific study. 25 texts, one theme, up to 200 words each, one language, one December in 2024. The people who came to vote were the ones who wanted to, and someone like that is already interested in the subject, so the sample is not representative. The numbers I quote here are from the main round with 233 participants. A few days later, by the time I talked about the experiment on television, more than 700 people had filled in the open form, so you will find that figure quoted in places too. It is not a contradiction, just two different moments.

Most importantly, the experiment did not prove that machines write better. It only showed that in a short text people can no longer tell who wrote it. That is a completely different thing.

And one more detail that came across too quietly at the time. The result was not free. Two weeks of work and eight pages of instruction sit behind it. The text written by plain ChatGPT with no preparation was the one people recognised best of all: 77.7 % correctly marked it as machine-written.

What I think now

Almost two years have passed, and those eight pages of instruction are no longer needed. What took me two weeks in 2024 you can now do in an evening, and the result will be better. Today's version fits in a few lines.

You are going to write in my style. Here are [five] of my texts: [pasted].
First, write down what you notice: sentence length, which words I avoid,
how I open and how I close.
Then write a [200] word text on [topic].
Do not correct my habits and do not decorate anything.
If something in the style is unclear, ask instead of guessing.

Which means my conclusion at the time, that we still need a person who can train the machine well, aged faster than I expected.

"Can AI write like a human" was an interesting question in 2024. It has been answered since, and it stopped being interesting.

A different question is still interesting: what do I need that for. I do not want a machine that writes instead of me. Not out of principle, but for a very practical reason: when the machine writes the first draft, I no longer have anything to say, I only have things to fix. The difference is enormous. Work like that produces text that is tidy and empty.

This site is my answer. I write the first draft myself, because that is where all the content is. I bring AI in where the work is boring: reordering the structure, checking facts and dates, telling me which sentence does not work, translating into English. There is nothing romantic about it, it is just the split of labour that gives me the best result.

What I would not use this for

One thing I would not do, then or now. I would not build an assistant that copies a living person's style without telling them. In the experiment all twelve took part voluntarily and sent us their own texts, and that is the only reason any of it was decent. Without that consent, what is left is a convenient way to impersonate someone.

And I would not use it where the text is a signature: an opinion, an apology, a message of sympathy. Not on principle, but because a text like that has to be your own or it is worth nothing.

What to do on Monday: take one of your own texts from this week and underline where your thinking is and where it is only form. The form can go to the machine. The thinking is worth keeping.

One question from 2024 is still sitting there unanswered. Professionals will be fine, I am relaxed about that. But what about the people who are only learning to write and can see that a machine does the same thing better and in twenty seconds. Where do they find the motivation to go through the unpleasant stage when their own writing is still bad. I had no answer then and I have none now, and it does not only apply to writers.

The fair objection: 233 volunteers, one theme and up to 200 words is far too thin a base for any conclusion about writing. That is true, which is why I never call this a study. But one thing survives even a sample like that: not one of the 233 marked all 25 correctly, and the best score was reached by exactly one person.

p.s. Viktorija could not sleep for two nights after the results came in. I slept perfectly well that week, and I now think that says something bad about me rather than about her.