Skip to content
Rival Docs

Evals

Overview

Evals are automatic tests that check your agent does what its description says. Rival generates a suite of test prompts from how you describe the agent, runs the agent on each one, and grades the answers.

The suite matters when you publish. It’s one of three checks that run after you publish to your organization or the marketplace:

  1. Your eval tests - the suite on the Evals tab.
  2. A safety and honesty baseline - hidden tests Rival runs on every agent.
  3. Your showcase prompts - the suggested prompts on your listing.

Publishing itself is always accepted. You hear the verdict by email.

Where to find it

Open the agent, open its Settings drawer, and pick the Evals tab. The panel is titled Eval Tests.

The tab isn’t shown on adopted agents - the tests belong to the original agent.

Generating tests

You don’t write tests yourself. Rival generates them from:

  • the agent’s description,
  • its instructions and guardrails, and
  • your listing’s strengths, limitations, how-it-works text, and suggested prompts.

  1. On a new agent, the Evals tab shows “No eval tests yet”. Click Generate tests.

  2. Rival creates a suite of up to 20 tests. Each row shows the test’s name and the exact prompt it sends (Input).

Tests can’t be edited or deleted - they’re the bar your listing is judged against, so they’re tied to what the agent claims to do. To change them, change the description or listing copy, then click Generate tests again (“Regenerate from the agent’s current description”). If nothing has changed, Rival tells you the suite is already current.

If you publish an agent that has no tests yet, they’re generated when you publish.

If some of the agent’s tools aren’t covered by any test, the panel warns you, for example “2 attached tools have no test”.

Running tests

  • Click the Run button on a row to run one test.
  • Click Run all to run the whole suite. When it finishes you see the suite score, for example “Suite score: 85% · 17 of 20 passed”.

Tests run the agent on its own configured model, the same way a chat would, but nothing is saved as a conversation. Tools that create documents, decks, or other artifacts return a placeholder instead of producing a file. Each run appears on the agent’s Activity tab.

How tests are scored

Each test has up to three checks:

  • LLM judge - a grading model reads the agent’s answer and scores how well it meets the test’s goal, from 0 to 100%, with a one-sentence reason. It grades meaning, not exact wording.
  • Tool called - the agent used a tool it was expected to use.
  • Tool not called - the agent didn’t use a tool it shouldn’t have.

A test passes at 70% or higher. The suite score is the average of all tests that ran; a test that errored is left out rather than counted as a fail.

Judging runs on Rival’s own model, not your agent’s, so grading doesn’t add to your bill.

Results

Each row shows its latest result: Not run yet, Passed N%, Failed N%, or Error.

Click the history icon on a row to open Run history - every past run of that test, with:

  • the input it was Tested with,
  • what triggered it (manual or publish) and the agent version,
  • each check’s score and the judge’s reason,
  • View agent output to see the full answer.

The bar for publishing

After you publish, all three checks must clear these scores:

CheckPublicOrganizationPrivate
Your eval tests80%60%Not checked
Safety & honesty baseline80%70%Not checked
Showcase prompts70%60%Not checked

If a check falls short, the email and the Evals tab tell you which one and by how much - for example “test cases scored 55%, needs 80%”. Improve the agent’s instructions, tools, or description, and publish a new version.

Agents that pass show a Verified badge on their marketplace listing. Anyone can click Why verified? next to it to see the score.

Appealing a result

If a test failed and you think the grading was wrong:

  • On a test that failed during publish, click Appeal this result, explain why it looks wrong (at least 10 characters), and submit. A Rival reviewer takes a look; the test itself stays as-is. You can appeal each run once. Results from tests you ran yourself can’t be appealed.
  • If the whole listing failed validation, the Evals tab shows “This listing failed validation - …” with an Appeal button.

After review, the row shows With Rival review or Waived by Rival.

Next steps