Blog ·

For agencies: one model per client, with proof

Most agency LLM work looks the same from the client's side: a prompt, a frontier API key and a monthly invoice. The client can copy that. A model trained on the client's own data, tested against the models they would use otherwise, is harder to copy and easier to defend in a renewal meeting.

The alternative is one model per client, with a result table as the sales document.

Why "one model per client"

The data stays separate. Each model learns from one client's examples only. Under GDPR, reusing one client's data to improve another client's model turns your agency into a controller for that purpose. It also tends to break your contracts (EDPB Guidelines 07/2020). One model per client avoids that question.

The result belongs to the account. "We fine-tuned a model on your 2,000 past tickets and it routes 98% correctly" is a sentence about the client's business. "We use Claude" is a sentence about a vendor.

Each client can be on a different model. A high-volume e-commerce client gets the cheapest base that is accurate enough. A legal client with long documents gets a larger one. You choose per job.

The proof comes first

The hard question in a fine-tuning deal is "how do you know it's better?" Answer it before the contract.

  1. Ask for 500 to 2,000 real examples from one workflow the client cares about, such as past tickets with the team that handled them, or invoices with the fields their finance team entered.
  2. Train and test on held-out data. Honed holds back at least 100 examples, trains on the rest, and runs your model, its untrained base and up to five frontier models of your choice on the same test set.
  3. Send the table. It shows accuracy, cost per 1,000 calls and median latency for every model. If the fine-tune does not win on the client's data, you find out before you have promised anything.

With the lowest-cost goal, a run takes 13 to 26 minutes from upload to result table (see our support-routing run), so you can show a client a result in the same week as the first call.

HONED is a free beta: every new account gets $20 in credits for training, evaluation, retraining and API calls. No credit card or subscription is required.

What to put in the proposal

  • The task, in the client's words. "Route incoming tickets to one of 14 teams."
  • The result table from the free eval, with the frontier models the client already knows.
  • The cost line. Cost per 1,000 calls at the client's volume, for the fine-tune and for the model they would use otherwise. See how to price an AI feature when your model costs less.
  • The data paragraph. Where the data goes and when it is deleted. See client data in fine-tuning.
  • The retraining plan. How often you add new examples and train again, and the test set you check each new version on.

After the sale

The model is called through an OpenAI-compatible endpoint, so you can drop it into the client's existing code by changing the base URL, key and model name (how it works). Each call is logged. When the client's data changes, say a new product line or new ticket categories, you add examples and retrain. The new version runs on the same held-out set before it goes live, so you can show the client that it improved.

A fine-tune on an open base does not change unless you retrain it. That makes the monthly report easy: same model, same test-set accuracy, plus the calls and cost for the month.

Sources