Benchmarks/Routing
A TF-IDF router instead of an LLM call
Can a classic classifier route requests instead of an LLM?
Short answer: TF-IDF plus logistic regression routed doc snippets into 5 categories in 0.628 ms per call at no API cost, with 0.600 accuracy against a 0.498 majority baseline. Its confidence tracked observed accuracy closely below 0.8.
- mean confidence
- observed accuracy
What the numbers say
- Good enough to gate or pre-route, not to replace a model on hard cases.
- Above 0.8 confidence it was overconfident (0.883 claimed vs 0.769 observed).
Results
| accuracy | 0.600 |
| macro F1 | 0.427 |
| majority-class baseline (always predict tutorial) | 0.498 |
| median latency, transform + predict, single snippet | 0.628ms |
| cost | $0 (local compute, no API call) |
Calibration
| 0.2-0.4 | 57 | 0.363 | 0.368 |
| 0.4-0.6 | 249 | 0.505 | 0.510 |
| 0.6-0.8 | 214 | 0.698 | 0.673 |
| 0.8-1.0 | 117 | 0.883 | 0.769 |
Setup
Analysis post. Own experiment: TF-IDF (scikit-learn 1.9.1 TfidfVectorizer) + LogisticRegression, Python 3.12.14, on the FastAPI English docs corpus (130 of 151 files map to 5 section categories; 2,187 heading-paragraph snippets, 1,550 train / 637 test, split by file). Not tested: Jev itself. No TypeSafe or OpenRouter API key was available in this environment, so no call was made to `typesafe/jev-1.13` or any LLM JSON-mode baseline. Every Jev number below is vendor-reported or reported by a named third party, with a link.