Choosing an AI classifier · Clarke Systems Group

Most teams choose an AI model the wrong way round. They pick a well-known model, write a prompt, try it on a handful of items and declare it good enough. The real test comes later, in production, when nobody is measuring. This guide sets out a better order: define the task, build a test set, measure accuracy and cost, and only then commit.

Define the task before you test anything

A model cannot be more precise than the task you give it. Start by writing down exactly what goes in, what comes out and who acts on the result.

The input. A product title, a support ticket, an invoice line. State which fields the model sees and which it does not.
The category list. Fix it before testing. Every category needs a one-line definition and a clear boundary with its nearest neighbours.
The output format. One label, a ranked list, or a label with a confidence flag. Decide whether "none of these" is allowed.
The cost of an error. A misrouted ticket may cost minutes. A miscategorised product may replicate across every channel it is sold through. This decides how much accuracy you need.

If two experienced members of your team would disagree on a category, the model will struggle too. Resolve that ambiguity in the definitions first.

Build a labelled test set from real data

Your test set is the ruler you measure every model against. Build it from real historical items, not invented examples.

Size and representativeness

Aim for enough items that every category appears a meaningful number of times, including the rare ones. A few hundred items is a sensible starting point for a modest category list. Larger lists need more. Sample across time periods, sources and formats, so the set reflects what the model will actually see.

Correct labels

Historical labels are often inconsistent, which is usually why you are replacing the manual process. Have a knowledgeable person review each label. Where reviewers disagree, record the agreed answer and update the category definitions.

Edge cases and hold-out

Deliberately include the difficult items: ambiguous wording, missing fields, items that sit between two categories. Then hold part of the set back. Use one portion while you refine prompts and keep a separate portion untouched for the final comparison. If you tune against the same items you score against, your results will flatter you.

Measure the right things

Overall accuracy is a starting point, not a verdict. A model can score well overall while failing badly on the categories that matter most.

Accuracy per category. Look for categories where performance drops. Rare categories often hide here.
Confusion between similar categories. Record which category the model chose when it was wrong. Repeated confusion between two neighbours usually points to a definition problem, not a model problem.
Consistency across runs. Run the same items more than once. A model that gives different answers to the same input is hard to trust and hard to audit.
Output validity. Count responses that are not on your category list or not in the required format. Each one needs handling in production.

Prompt design

Test every model with the same prompt first, so the comparison is fair. The prompt should include the full category list, each definition, and a short example or two for categories that are easily confused. Ask for a fixed output format. Once you have a baseline, refine the prompt using the development portion of the test set, never the hold-out.

Know the cost and speed per item

Every model has a cost per classification. Work it out before you commit, using your real items and your real prompt.

Tokens in and out. Input cost is driven by the prompt and the item. Long category lists and many examples make every call more expensive. Output cost is usually small if you ask for a label only.
Batching. Several providers offer lower rates for work that can wait. If results are not needed immediately, batch processing can reduce cost considerably.
Caching. The instructions and category list are identical on every call. Where a provider supports prompt caching, that repeated portion can be cheaper.
Latency. Measure the time per item and for a full daily volume. A routing step that sits in front of a person needs a fast answer. An overnight tagging run does not.

Multiply the cost per item by your expected monthly volume. That figure, alongside accuracy, is what you are choosing on.

Hosted APIs or self-hosted open-weight models

There are two broad routes, and each suits a different situation.

Hosted API models

Models from the major providers are available through an API. There is no infrastructure to run, capability is generally strong and you pay per use. The trade-off is that your data leaves your environment, so contract terms and provider controls matter.

Self-hosted open-weight models

Open-weight models run on infrastructure you control. Data stays within your environment and the cost profile shifts from per-item charges to compute you provision. The trade-off is operational: someone must deploy, scale, patch and monitor the model. At low volumes that burden often outweighs the savings.

Choose on accuracy and cost together

The model at the top of a public leaderboard is not necessarily the right one for your task. Leaderboards measure general ability on other people's data. What matters is accuracy on your categories at a cost you can sustain. A model slightly below the best on your test set may cost a fraction as much per item. Decide your minimum acceptable accuracy first, then choose the cheapest model that meets it reliably.

Our founder followed this approach in work for a previous employer, a global digital-commerce analytics business. It was not client work for Clarke Systems Group. Hosted models from OpenAI and Anthropic and self-hosted open-weight models were benchmarked on accuracy against real categories and on cost per classification before one was chosen. You can read the AI categorisation case study for the shape of that system.

Protect the data and monitor after launch

Classification inputs often contain customer, supplier or commercial data. Treat the service like any other system that handles it.

Use business API terms, under which your data is not used for model training by default.
Restrict access to the service and its results to the people and systems that need it.
Require multi-factor authentication for anyone with administrative access.
Keep full audit logs of who accessed what, and when.

Our FAQ sets out how we handle data on every project.

Monitoring accuracy

Accuracy at launch is not accuracy for ever. Your inputs change, new categories appear and providers update or retire models. Review a regular sample of live results against a human judgement and track accuracy over time. Watch for drift in particular categories. When a provider changes a model, or you change the prompt or the category list, re-run the hold-out test set before the change goes live.

A step-by-step evaluation plan

Write the task definition: inputs, output format, category list with definitions, and the cost of an error.
Set a minimum acceptable accuracy, overall and for the categories that matter most.
Sample real historical items, review the labels, and add deliberate edge cases.
Split the set into a development portion and an untouched hold-out portion.
Shortlist candidate models: a mix of hosted and, where data control matters, open-weight.
Run every candidate on the development portion with the same prompt, then refine.
Score each candidate on the hold-out: accuracy per category, confusion, consistency and output validity.
Calculate cost per item and monthly cost at expected volume, and measure latency.
Choose the cheapest model that meets your accuracy threshold reliably.
Launch with sampling, accuracy tracking and a process for re-benchmarking.

Common mistakes

Testing on a few hand-picked examples instead of a representative set.
Tuning the prompt against the same items used for the final score.
Reporting only overall accuracy and missing weak categories.
Choosing the most capable model without calculating cost at real volume.
Leaving category definitions vague and blaming the model for the confusion.
Treating launch as the end of measurement.

If you are planning to replace manual categorisation, send us a brief of your process. We will tell you whether it is a good fit.

Key takeaways

Define the task and every category precisely before you test a single model.
Build a labelled test set from real historical data, include edge cases and keep a hold-out portion untouched.
Measure accuracy per category, confusion between neighbours and consistency across runs, not just one overall figure.
Calculate cost per item at real volume and choose the cheapest model that meets your accuracy threshold.
Protect the data with business API terms, access control, MFA and audit logging, and keep measuring accuracy after launch.

Choosing an AI model for classification: accuracy and cost