How to Run a Ten-Minute Test Between Two Assistants
5 min read
Most assistant comparisons online are one person's single try on one prompt. That is fine for a quick impression and poor evidence for a decision. You can do better in ten minutes, and the result will be about your work, not someone else's.
This article describes a small experiment: three tasks, two assistants, a blind score and a log. It is built to keep common mistakes out of the comparison, such as testing different things on each tool or scoring with a bias you did not notice. The examples use Claude as one of the two, and the method works for any pair. For the basics of Claude, see the Complete Beginner's Guide to Claude.
What goes wrong in a casual comparison
Common ways a test misleads you
- One run, so you compare luck, not tools
- Different prompts on each tool, so you compare prompts
- One tool has your memory or project loaded and the other does not
- You know which output is which and prefer the one you expected to like
How this test avoids them
- Three runs per task, so you see the spread
- One identical prompt, pasted into both
- Matching context: both loaded the same way, or both clean
- Outputs relabeled A and B before you score
Setup: three tasks that stand for your week
Choose tasks from real work, ideally one of each kind.
- A writing task with a defined audience and a voice sample.
- An analysis task where you already know the right answer: a document with a fact buried in it, or a small table with a known total.
- A messy task: a vague request you would really send, like "help me prepare for tomorrow's meeting."
The second one is the anchor. Because you know the answer, you can score accuracy without opinion.
The ten minutes
- 1
Minutes 1 to 2: write the three prompts once
Write each prompt fully, with context, task and format, as in The Prompt Framework That Fixes a Vague Claude Answer. Save them in a note. You will paste the identical text into both tools.
- 2
Minutes 3 to 4: match the conditions
Open a fresh chat in each assistant. Decide whether memory and web search are on in both or off in both, and keep them the same. If Claude has a project or memory that changes the answer, either give the other tool equivalent context or turn the extra context off, so the comparison is fair.
- 3
Minutes 5 to 7: run each task twice on each tool
That is twelve outputs. Paste them into a document and label them with a random letter so you cannot tell which tool wrote which. Ask a colleague to relabel them if you can.
- 4
Minutes 8 to 9: score each output from 1 to 5
Use four columns: accuracy, instructions followed, usefulness, and edit effort. Do the scoring before you look at the key.
- 5
Minute 10: reveal and record
Match letters to tools, average the scores per task and write the result in your log.
If ten minutes is not enough for your tasks, take twenty. What matters is that you finish all the steps, not that you hit the number.
A log you can reuse
Keep one line per test. After a few, you have a record that beats memory.
| Date | Task | Tool A | Tool B | Accuracy | Instructions | Edit effort | Notes |
|---|---|---|---|---|---|---|---|
| 24 Oct | Weekly client email | Claude | ChatGPT | 5 / 5 | 4 / 5 | 3 min / 6 min | Both kept facts; one ignored the word limit |
Add the model name and the settings you used in the notes. Models change, and "Claude in October" is not the same as "Claude in January."
Reading the results honestly
- A gap of one point on a five-point scale, on three tasks, is noise. Do not rebuild your workflow on it.
- A consistent gap across all three runs of a task is a signal. Even then it is a signal about that kind of task.
- Watch for the variance. If one tool gives you a five and a two on the same prompt, the average hides the thing you care about: how often it lets you down. Why Claude Gives You a Different Answer Every Time explains why the spread exists.
- Check the settings before you blame the tool. A thinking setting, a search setting or a model choice can change an answer. Claude's Thinking Mode and Choosing a Claude Model cover two of them.
Common mistake
Letting the speed of the response count as quality. A fast answer that you have to fix costs more than a slower one you can send. Score the edit effort, not the wait.
What the test cannot tell you
A ten-minute test measures output on three tasks. It does not measure how a tool fits your day: whether it opens in the right place, works with your files or is allowed by your employer. For that, ask the questions in Which AI Assistant for Which Job: A Decision Table. It also says nothing about a tool's behavior next month. Plan to repeat the test when a major model update arrives.
Make it a habit with a trigger
Re-run a version of this test when one of three things happens: a new model generation for a tool you rely on, a pricing or plan change, or a task you do often that has started to disappoint you. Each time, you will spend a short while and keep a decision you can defend. Using Two AI Assistants on One Task shows what to do when the answer is "both."
Official sources
Checked on October 24, 2026. Features, plans and names change often, so the vendor's own pages are the final word.