Routing tests

Write the tasks a skill should answer and the ones it should ignore, run them on every publish, and block a publish that breaks one.

By The SKaaS team. Last updated . Markdown version

A skill that never gets picked is a broken skill. Routing tests check that the right skill is found for a task, and that no skill is suggested when none fits. They are in Cloud Pro and above, and in the Community edition.

How do I write one?

Put a skaas.tests.yaml file in the repository. Each case has a query and an expectation: either the skill that should appear within the top results, or that nothing should be suggested.

- query: "investigate a MongoDB latency regression"
  expect: { skill: mongodb-latency-triage, top_k: 3 }
- query: "good morning"
  expect: { none: true }

When do they run?

On every publish, against the candidate corpus. You can also run them offline with skaas test, which searches in your own process and has no semantic channel, or against the service with skaas test --remote. Results show in the dashboard, with the skill that won for each failing case and which search channel failed.

Can a failing test block a publish?

Yes. Switch on tests_block_publish in the repository policy, and a publish that breaks a routing test is refused, in the same way that it can be refused for audit findings.

What do they not prove?

A passing suite shows that your cases pass. It does not measure search quality in general.