Run CBTKC9XN — Vizra Cloud

Run CBTKC9XN completed ci 16 Aug 2026, 10:02

87.1% ▲ 0.9 vs baseline

10 of 12 rows below full pass. Pass rate 83.0%.

12 rows × 36 samples cost $0.0214 judge $0.0086 4m 0s gate passed 641ffc7 · main · Add the damaged-goods cases github

# Failing agent evaluation: RefundPolicyAccuracy

I run automated evaluations against an AI agent. The run below failed. Each row is a test case: an input the agent received, what it was expected to do, and what it actually produced. Some assertions are deterministic; others are an LLM judge scoring the response against a written criterion, and its reasoning is included.

**What I want:** work out why these are failing and propose specific changes — to the agent's instructions, its tools, or the expectations themselves if you think a test is wrong. Say which, and why. Do not rewrite the eval to make it pass.

## The run

- Suite: `App\Evals\RefundPolicyAccuracy`
- Score: 87.1% (baseline 86.2%)
- Failing rows: 10 of 12
- Branch: `main`


---

## Failing row: Do you refund the delivery charge?

Score 83.1% — was 82.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Can I exchange instead of refund?

Score 83.1% — was 82.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: How long do I have to return something?

Score 83.1% — was 82.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Can I return in store?

Score 85.1% — was 84.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: What if the item arrived damaged?

Score 85.1% — was 84.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Can I return an opened item?

Score 85.1% — was 84.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Are sale items refundable?

Score 87.1% — was 86.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Do I need the original packaging?

Score 87.1% — was 86.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Can I refund a gift?

Score 89.1% — was 88.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

---

## Failing row: Who pays return postage?

Score 89.1% — was 88.2% on the baseline (+0.9).

No per-sample detail was reported for this row.

Rows

All 12 Failing 10 Errors 0

Input	Samples	Score	vs baseline	Cost inc. judge
How long do I have to return something? anthropic/claude-sonnet-5	2/3	83.1%	+0.9	$0.0018
Can I exchange instead of refund? anthropic/claude-sonnet-5	2/3	83.1%	+0.9	$0.0018
Do you refund the delivery charge? anthropic/claude-sonnet-5	2/3	83.1%	+0.9	$0.0018
Can I return an opened item? anthropic/claude-sonnet-5	2/3	85.1%	+0.9	$0.0018
What if the item arrived damaged? anthropic/claude-sonnet-5	2/3	85.1%	+0.9	$0.0018
Can I return in store? anthropic/claude-sonnet-5	2/3	85.1%	+0.9	$0.0018
Do I need the original packaging? anthropic/claude-sonnet-5	2/3	87.1%	+0.9	$0.0018
Are sale items refundable? anthropic/claude-sonnet-5	2/3	87.1%	+0.9	$0.0018
Who pays return postage? anthropic/claude-sonnet-5	2/3	89.1%	+0.9	$0.0018
Can I refund a gift? anthropic/claude-sonnet-5	2/3	89.1%	+0.9	$0.0018
How long does a refund take? anthropic/claude-sonnet-5	3/3	91.1%	+0.9	$0.0018
I lost my receipt, what now? anthropic/claude-sonnet-5	3/3	91.1%	+0.9	$0.0018