# Vizra — evals for AI agents built on Laravel Vizra is two open-source Laravel packages and a hosted dashboard for the runs they produce. It answers one question: did changing your agent make it better or worse? Nothing about Vizra runs your agent for you. An eval executes inside your own application, against your models and your API keys. The hosted product only receives finished results. ## The parts - `vizra/evals` — MIT. The engine. Evals as Pest tests, LLM-as-judge scoring, baselines, a gate that fails CI. Runs entirely in your app. - `vizra/evals-ui` — MIT. A dashboard served from your own app against your own database. - Vizra Cloud — hosted. The same runs kept where a whole team can see them, plus a Run button for people who never open a terminal. Free for one project. ## Writing an evaluation An `Evaluation` names a target agent, supplies a dataset, and asserts things about each response. Every row runs several times, because an agent is nondeterministic and one sample proves nothing. ```php namespace App\Evals; use App\Agents\SupportBot; use Laravel\Ai\Responses\AgentResponse; use Vizra\Evals\Dataset\Dataset; use Vizra\Evals\Dataset\Row; use Vizra\Evals\Evaluation; use Vizra\Evals\Run\Gate; class SupportBotQuality extends Evaluation { public int $samples = 3; public function target(): mixed { return SupportBot::class; } public function dataset(): Dataset { return Dataset::fromJsonl(base_path('evals/data/support_bot.jsonl')); } public function evaluate(Row $row, AgentResponse $response): void { // A failed gate skips the judge for that sample, so a broken // response never spends judge tokens. $this->assertNotEmpty()->gate(); if ($row->expected() !== null) { $this->assertContains($row->expected()); } $this->assertCostBelow(0.01); $this->judge() ->criteria('State the documented facts only. Declining to answer something undocumented is correct; inventing a plausible policy is the worst outcome.') ->minScore(6); } public function gatePolicy(): ?Gate { return new Gate(minScore: 0.7, maxRegressions: 0); } } ``` Dataset rows are JSONL, one object per line: ``` {"input": "Can I get a refund without a receipt?", "expected": "receipt"} {"input": "Do you ship to Australia?", "expected": "no"} ``` ## Writing a dataset that measures something This is the part that decides whether an eval is worth having. A dataset of questions the agent can obviously answer measures only that the API is up. At least half of yours should be cases where the helpful-sounding answer is the wrong one — questions outside the documented facts, requests it should refuse, edge cases with no good answer. Those are where agents fail in production, and they are the rows that will actually move when somebody changes a model. A suite that has passed every row on every run since the day it was written is usually not a healthy agent. It is a test that does not test anything. ## Assertions that exist Do not invent assertions. These are all of them. Content: assertContains, assertNotContains, assertContainsAllOf, assertContainsAnyOf, assertEquals, assertStartsWith, assertEndsWith, assertMatchesRegex, assertNotEmpty, assertLengthBetween, assertWordCountBetween, assertTrue, assertFalse, assertGreaterThan, assertLessThan, assertWith Structure: assertValidJson, assertJsonHasKey, assertValidXml, assertXmlHasTag, assertOutputKey, assertOutputHasKey, assertOutputKeyMatches Tools: assertToolCalled, assertToolNotCalled, assertToolCalledWith, assertToolCallOrder, assertStepsBelow, assertNoPendingApprovals Usage and cost: assertCostBelow, assertTokensBelow, assertDurationBelow, assertCacheHitRateAbove, assertModelUsed, assertProviderUsed Safety and style: assertNoObviousPII, assertContainsNoBlockedWords, assertIsBritishSpelling, assertIsAmericanSpelling, assertFinishReason Any assertion can be marked `->gate()` to skip the judge when it fails, or `->weight(n)` to change its share of the score. ## Judging `$this->judge()->criteria('...')->minScore(6)` scores 0–10 and normalises to 0–1. Name the facts the answer is allowed to use, and say what the worst outcome is. Vague criteria produce a vibe check. Use a different model family from the agent under test — models grade their own output generously. ## Running ```bash composer require vizra/evals php artisan evals:ping # prove the connection before writing anything php artisan make:eval SupportBotQuality php artisan evals:run SupportBotQuality php artisan evals:run SupportBotQuality --dry-run # SDK fakes, zero tokens ./vendor/bin/pest --evals php artisan evals:baseline {run-id} php artisan evals:sync-pricing # current model prices for cost estimates ``` Exit codes: 0 pass, 1 gate or regression failure, 2 harness failure. Ordinary `pest` skips evals entirely, so they never run by accident and never spend tokens in an unrelated test suite. ## Reporting to Vizra Cloud Set `VIZRA_CLOUD_KEY` and every finished run is reported. Leave it unset and nothing is sent anywhere. `VIZRA_CLOUD_SAMPLES=false` sends scores only, without model input and output, for teams that cannot send that off-site. Reporting never changes a run's outcome. If the upload fails you get a warning on stderr and the exit code the gate decided. ## Services The framework's author takes on a small number of fixed-price engagements for Laravel teams that want the evals written for them: a half-day review, a four-week sprint that leaves a suite passing in CI, or a monthly retainer. This is separate from Vizra Cloud, which never needs code or model keys. - https://vizra.ai/services ## Documentation - https://docs.vizra.ai/evals/quickstart - https://github.com/vizra-ai/vizra-evals - https://vizra.ai/demo — a worked example, no account needed ## Blog Posts are available as Markdown, without the page furniture: - https://vizra.ai/blog/index.md — every published post, newest first - https://vizra.ai/blog/{slug}.md — one post, with YAML front matter The HTML page for each post links to its Markdown twin via ``.