Field log
New models ship weekly. Each entry is one thing tested on real data, with the numbers, the caveats and what it means for a real deployment.
Liquid d1-3B (local, Apple M1 Pro)
A 3B model on a laptop, passing only its unsure calls to the cloud, matched cloud-only accuracy on 300 banking messages while keeping 68% of calls local (96.1% accurate on those). One dataset; re-test on messier data pending.
Setup. 300 real bank-support messages (Banking77): 100 fraud, 100 lookalikes that read like fraud but aren’t, 100 others. Two decisions each: a fraud flag and one of 7 teams. The local model returns a probability per answer, no generated text. The cloud baseline is Gemini 3.5 Flash-Lite, asked to state its confidence.
| d1-3B, local | Flash-Lite, cloud | |
|---|---|---|
| Fraud accuracy | 0.927 | 0.967 |
| Fraud AUROC | 0.971 | 0.985 |
| Fraud calibration error (ECE) | 0.058 | 0.032 |
| Routing accuracy | 0.873 | 0.883 |
| Latency | 633 ms p50 per message | 4.4 s per 25-message call |
The gate. At a 0.8 confidence threshold the local model keeps 68% of messages, 96.1% accurate on those, and escalates the rest. Combined accuracy 0.897 vs 0.877 cloud-only. The gap is 6 messages out of 300, so the honest claim is matches the cloud with a third of the cloud calls, not “beats it”.
Also measured. Gemini 3.8 Flash on the free tier took 203.8 s for one 5-message call; Flash-Lite took 2.5 s.
Decision models (calibrated yes/no in one pass) · the harness over the model · guardrails on agent tool calls · small local models (LFM3, Gemma 4) · cross-modal embeddings (EmbeddingGemma 2).