Blog ·

Swap the base URL, keep your code: an OpenAI-compatible endpoint for your own model

Most LLM code in production calls one function: client.chat.completions.create. Retry logic, logging, timeouts, tests and dashboards all grew up around it. When you replace the model, you want to keep all of that and change as little as possible.

"OpenAI-compatible" means a server accepts POST /chat/completions with the OpenAI request body and returns the same response shape. Google, Anthropic, Together, Groq, OpenRouter and Ollama all publish such an endpoint (Gemini, Anthropic, Together, Groq, OpenRouter, Ollama). Honed does too, for the model you fine-tune on it.

The change

You change three values: the base URL, the API key and the model name.

TypeScript (openai package, option baseURL):

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://tryhoned.com/v1",
  apiKey: process.env.HONED_API_KEY,
});

const res = await client.chat.completions.create({
  model: "your-honed-model-id",
  messages: [{ role: "user", content: "Where is my refund? Order 48121." }],
  temperature: 0,
});

console.log(res.choices[0].message.content); // "refund_status"

Python (openai package, argument base_url):

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://tryhoned.com/v1",
    api_key=os.environ["HONED_API_KEY"],
)

res = client.chat.completions.create(
    model="your-honed-model-id",
    messages=[{"role": "user", "content": "Where is my refund? Order 48121."}],
    temperature=0,
)

Both SDKs also read the OPENAI_BASE_URL environment variable (openai-node, openai-python). If your code builds the client in one place, you can switch by environment and leave the code alone.

What is the same and what is different

Groq rejects some fields with an error, and Anthropic says its compatibility layer is meant for testing, not long-term production use (Anthropic docs). Here are Honed's edges, so you can check them against your code before you switch:

Feature Honed /v1/chat/completions
model Your Honed model ID (shown in the API tab)
messages User and assistant turns, ending with a user turn. System messages are ignored, because the model was trained with its own system prompt. Image and audio parts are dropped.
temperature 0 to 2
max_tokens or max_completion_tokens 1 to 4,096
stream Not supported. Requests with stream: true return a clear error.
tools, n, logprobs Ignored. You get one plain text answer.
Response Standard chat.completion object with choices[0].message and usage
Errors OpenAI error shape: { "error": { "message", "type", "code" } }
Rate limit 60 requests per minute per key, HTTP 429 with Retry-After

Streaming matters less for the jobs people fine-tune small models for. A routing label or an extracted JSON object is a few dozen tokens, so the whole answer arrives at once. On our support-routing run the median call took 530 ms.

Swapping models safely

The model is the part of the stack you should be able to swap. When the call site stays the same, you can:

  • run the old and new model side by side behind a flag,
  • send 5% of traffic to the new one and compare,
  • roll back by changing an environment variable.

The other half is knowing that the new model is better before you send it traffic. Honed tests every model it trains on at least 100 held-out examples against the frontier models you pick, and shows accuracy, cost per 1,000 calls and median latency. With the lowest-cost goal, a run takes 13 to 26 minutes from upload to result.

Sources