Evaluating open-weight models after fine-tuning
How I compare LoRA adapters for Qwen, Llama, and Mistral against a held-out task set, an explicit baseline, and the realities of serving them.
A fine-tuned model can look convincing in a handful of examples and still perform poorly on the work it was built for. The useful question is not whether the model learned something; it is whether it beats a simpler baseline on the task that matters.
Start from the task
I start with a held-out task set and an explicit baseline, then compare adapters across task quality and failure patterns. The cases include routine examples, awkward edge cases, and prompts designed to expose the errors that would be expensive in production.
I have used LoRA and adapter training with open-weight Qwen, Llama, and Mistral models across hosted and self-managed inference. For each candidate, I version the dataset, prompt contract, evaluation cases, model configuration, and deployment constraints together. A checkpoint on its own is not enough to reproduce a result.
Evaluate the serving path
Offline quality is only one part of the decision. I also compare latency, memory requirements, throughput, caching behaviour, and the cost of routing failures to a fallback model or person.
This exposes trade-offs that a single aggregate score hides. A model may be slightly stronger but too slow for the interaction, or cheap to serve but inconsistent on the cases that matter most.
Prefer the smallest model that clears the bar
A smaller or specialised model can be the better choice when its job is narrow, its failure modes are understood, and its latency or cost advantage is meaningful. The selection rule is therefore task-specific: clear the agreed quality bar, fit the serving constraints, and leave a fallback for uncertain cases.
Fine-tuning is valuable when it improves that whole path. If prompt design, retrieval, or a simpler model already clears the bar, adding an adapter only creates another component to own.