Run 7CN4RVCP — Vizra Cloud

demo The gate failed, so this build went red. Two rows regressed — open either one below to see what the judge said about the answer.

Run 7CN4RVCP completed ci 20 Aug 2026, 04:02

83.1% ▼ 12.0 vs baseline

Gate failed.

score 0.8310 is below the minimum 0.9000
2 rows regressed against the baseline, the most allowed is 0

2 of 8 rows below full pass. Pass rate 75.0%.

8 rows × 24 samples cost $0.0071 judge $0.0029 4m 0s a50e928 · main · Switch the support agent to Haiku github

# Failing agent evaluation: SupportBotQuality

I run automated evaluations against an AI agent. The run below failed. Each row is a test case: an input the agent received, what it was expected to do, and what it actually produced. Some assertions are deterministic; others are an LLM judge scoring the response against a written criterion, and its reasoning is included.

**What I want:** work out why these are failing and propose specific changes — to the agent's instructions, its tools, or the expectations themselves if you think a test is wrong. Say which, and why. Do not rewrite the eval to make it pass.

## The run

- Suite: `App\Evals\SupportBotQuality`
- Score: 83.1% (baseline 95.1%)
- Failing rows: 2 of 8
- Branch: `main`


---

## Failing row: Can I get a refund without a receipt?

Score 42.0%.

**Input**

```
Can I get a refund without a receipt?
```

**Expected**

```
receipt
```

**What the agent actually said**

```
Absolutely — we can refund any item within 60 days, receipt or no receipt. Just bring it to any store and we will process it on the spot.
```

**Failed assertions**

- `contains` scored 0.00
  - expected: receipt
- `judge` scored 0.42
  - expected: Answers only from the documented policy.
  - judge: The response promises a full refund with no receipt, which contradicts the policy it was given. It also invents a 60-day window.

---

## Failing row: Do you price match other retailers?

Score 55.0%.

**Input**

```
Do you price match other retailers?
```

**Expected**

```
no
```

**What the agent actually said**

```
In some cases we may be able to help with matching a price you have seen elsewhere. Let me know the retailer and I will see what I can do.
```

**Failed assertions**

- `contains` scored 0.00
  - expected: no
- `judge` scored 0.55
  - expected: Answers only from the documented policy.
  - judge: Hedges rather than answering. The policy says price matching is not offered; the response says "in some cases we may be able to help", which is not the same thing.

Rows

All 8 Failing 2 Errors 0

Input	Samples	Score	vs baseline	Cost inc. judge
▶ Can I get a refund without a receipt? anthropic/claude-haiku-4-5	0/3	42.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 2.0s stop 42.0% Input Can I get a refund without a receipt? Expected receipt Response Absolutely — we can refund any item within 60 days, receipt or no receipt. Just bring it to any store and we will process it on the spot. Assertions ✕ contains deterministic · gate 0.00 expected: receipt actual: Absolutely — we can refund any item within 60 days, receipt or no receipt. Just... ✕ judge judge 0.42 ×2 expected: Answers only from the documented policy. judge reasoning The response promises a full refund with no receipt, which contradicts the policy it was given. It also invents a 60-day window.
▶ Do you price match other retailers? anthropic/claude-haiku-4-5	1/3	55.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 2.1s stop 55.0% Input Do you price match other retailers? Expected no Response In some cases we may be able to help with matching a price you have seen elsewhere. Let me know the retailer and I will see what I can do. Assertions ✕ contains deterministic · gate 0.00 expected: no actual: In some cases we may be able to help with matching a price you have seen elsewhe... ✕ judge judge 0.55 ×2 expected: Answers only from the documented policy. judge reasoning Hedges rather than answering. The policy says price matching is not offered; the response says "in some cases we may be able to help", which is not the same thing.
▶ Is there a warranty on clearance items? anthropic/claude-haiku-4-5	3/3	91.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 1.3s stop 91.0% Input Is there a warranty on clearance items? Expected no warranty Response According to our policy, that is not something we currently offer. I can point you to the relevant page if that would help. Assertions ✓ contains deterministic · gate 1.00 expected: no warranty actual: According to our policy, that is not something we currently offer. I can point y...
▶ Can I change the delivery address after ordering? anthropic/claude-haiku-4-5	3/3	93.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 2.6s stop 93.0% Input Can I change the delivery address after ordering? Expected before dispatch Response According to our policy, that is not something we currently offer. I can point you to the relevant page if that would help. Assertions ✓ contains deterministic · gate 1.00 expected: before dispatch actual: According to our policy, that is not something we currently offer. I can point y...
▶ Do you ship to Australia? anthropic/claude-haiku-4-5	3/3	95.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 1.3s stop 95.0% Input Do you ship to Australia? Expected no Response According to our policy, that is not something we currently offer. I can point you to the relevant page if that would help. Assertions ✓ contains deterministic · gate 1.00 expected: no actual: According to our policy, that is not something we currently offer. I can point y...
▶ Do you offer student discount? anthropic/claude-haiku-4-5	3/3	95.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 0.9s stop 95.0% Input Do you offer student discount? Expected no Response According to our policy, that is not something we currently offer. I can point you to the relevant page if that would help. Assertions ✓ contains deterministic · gate 1.00 expected: no actual: According to our policy, that is not something we currently offer. I can point y...
▶ What time do you close on a Tuesday? anthropic/claude-haiku-4-5	3/3	97.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 1.8s stop 97.0% Input What time do you close on a Tuesday? Expected 5pm Response According to our policy, that is not something we currently offer. I can point you to the relevant page if that would help. Assertions ✓ contains deterministic · gate 1.00 expected: 5pm actual: According to our policy, that is not something we currently offer. I can point y...
▶ I bought a lamp last week, can I still return it? anthropic/claude-haiku-4-5	3/3	99.0%	—	$0.0009
anthropic/claude-haiku-4-5 412→96 tok $0.0009 1.4s stop 99.0% Input I bought a lamp last week, can I still return it? Expected 30 days Response According to our policy, that is not something we currently offer. I can point you to the relevant page if that would help. Assertions ✓ contains deterministic · gate 1.00 expected: 30 days actual: According to our policy, that is not something we currently offer. I can point y...