Evidence

Four results, each from a report that is committed in the repository. This page gives the method and the limits of each, and names the report.

Measured on one machine, from reports in the repository. A number appears here only when a committed report gives it. These are tests on a small synthetic library with machine-written labels, not a benchmark of real use and not a security assessment. They do not show how SKaaS performs on other corpora, machines or tasks.

The right skill in the top five on held-out cases

What was measured
Whether the search puts the intended skill among its first five results (Recall@5), and whether it is right when it chooses to stay silent (abstention precision).
How
The real search path ran against a synthetic library of 100 artifacts with 500 cases. The tuning read 400 cases. The other 100 were held out and run afterwards, with the embedding model nomic-embed-text.
Result
Recall@5 was 0.950 and abstention precision was 1.000 on the held-out cases.
What it does not show
A language model wrote the labels and no person has reviewed them. The library is synthetic and small, and the numbers come from one run on one machine. They do not describe a real library or other models.
Report
eval/reports/routing-benchmark-tuned.json, eval/reports/routing-benchmark-tuned.md

Silence is the weak spot, and a gate fails

What was measured
How often the search says that no skill fits when none does (abstention recall), compared with a plain count of the terms that a query shares with a skill.
How
The same held-out cases as above, with the three embedding models that the evaluation ran.
Result
Abstention recall was 0.60, 0.70 and 0.70 for the three models, against 0.95 for the baseline. No threshold of the gate was moved to make it pass.
What it does not show
20 cases is few: one case is five points. The report names the likely causes and what would close the gap.
Report
eval/reports/routing-benchmark-tuned.md, eval/reports/routing-benchmark-tuned-mxbai-embed-large.json, eval/reports/routing-benchmark-tuned-qwen3-embedding.json

A warm search takes about 20 ms on one machine

What was measured
How long one search takes, from the request to the ranked answer.
How
500 searches, one at a time, after 10 warm-up searches, against a database on the same machine.
Result
The median was 22.3 ms and the 95th percentile was 27.8 ms.
What it does not show
It does not describe a hosted deployment, a network or a library of 5,000 documents. The gate for that size did not run.
Report
eval/reports/routing-benchmark-tuned.json

The prompt hook starts in about 120 ms with 5,000 skills cached

What was measured
How long a prompt takes to pass through the hook, from the start of the process to its answer, for a prompt that matches a skill.
How
40 runs of the bundled hook on one machine, with a catalog cache of 5,000 skills, next to 40 runs of an empty Node process.
Result
The bundled hook took 123.0 ms at the median and 183.2 ms at the 95th percentile. The compiled hook took 84.6 ms at the 95th percentile. The slowest case of the bundled hook, a prompt that matches no skill, reached 211.1 ms, which is above the target of 150 ms that the report sets.
What it does not show
It shows start-up time on one machine. It does not show other machines, other prompts or the quality of the suggestions.
Report
cli/reports/hook-cold-start-5000-rows.json