Evaluation and acceptance

The goal is to measure whether enrichment creates a trustworthy Arda Item and ItemSupply suggestion from objects people actually scan — not whether every provider returns something.

Setup

From arda-frontend-app, add provider credentials to .env.local (copy the relevant lines from .env.example; .env.local is gitignored):

cd arda-frontend-app
npm install
npm test
npm run eval:live

eval:live runs the in-process engine directly against the case corpus at src/server/lib/enrichment/live-cases.json — the same normalize → adapters → reconcile path the BFF route uses, without going through HTTP or the running app. Live adapters need their API keys in the environment; set ENRICHMENT_FIXTURES_ENABLED=true to run it offline against the fixture catalog. npm run eval:live:suite repeats every case (set ENRICHMENT_LIVE_SUITE_RUNS to change the count) to expose intermittent provider timeouts.

Use npm run dev:mock for deterministic fixtures and npm run dev:live for manual live testing in the UI with real provider keys (mock Operations, live enrichment).

Evaluation set

Start with 30–50 physical items from real Arda environments:

consumer products with clean UPC/EAN labels
industrial parts with manufacturer numbers but weak retail data
different package levels of the same product
products stocked under tenant-specific names
products available from multiple suppliers
discontinued or substituted parts
damaged, invalid, and unknown barcodes
deliberately ambiguous or incorrect search results

For each item record ground truth before testing:

scanned value and barcode type
expected product name, brand, and MPN
expected package and inventory unit
known supplier SKUs
whether an automatic exact match is safe
notes from the inventory manager

Do not use provider output to create the ground truth.

Review sheet

Capture one row per scan:

Field	Description
test ID	stable physical-item identifier
request ID	enrichment call
expected identity	ground-truth label
suggested identity	returned draft
identity verdict	exact, usable-after-edit, wrong, no-match
package verdict	exact, equivalent, wrong, missing
selected offer	supplier and SKU, if useful
response latency	POST duration
accepted edits	changed fields
source traces	adapters, status, and latency
provider cost	request cost when available

Initial acceptance targets

These are prototype gates, not contractual SLOs:

zero confidently wrong (matched) identities in the test set
at least 90% exact identity precision among matched results
at least 70% useful coverage across the mixed set
package correctness reported separately from identity correctness
p95 time to first authoritative candidate below 2 seconds
completed enrichment response within the enforced budgets: 5 seconds for a barcode, 10 for a URL or text lookup
one-provider failure still returns a usable result when another source hits
no result or trace exposes provider credentials

Prefer lowering coverage over accepting a confidently wrong match.

Automated live corpus

The corpus is src/server/lib/enrichment/live-cases.json in arda-frontend-app. Run it with npm run eval:live (or eval:live:suite to repeat every case). Add known UPC/ASIN or supplier-page pairs as they are collected; assert stable identity facts, not price, latency, or result order.

Scenarios

Exact barcode

Expect an exact identity, selected evidence showing the winning adapter, and Amazon only when its returned external ID matches the scan.

Package mismatch

Scan a single unit while a supplier sells a case. Expect the product identity to match but the offer to be equivalent or require review. Do not calculate unit cost without explicit package quantities.

Tenant-approved alias

Use a test TenantProjectionReader mapping. Expect the tenant name to win over external titles while external MPN, images, and offers remain usable.

Targeted discovery

Resolve a brand/MPN, then confirm the Exa request contains those terms and is limited by preferred supplier policy. Exa evidence must not make an otherwise unverified identity authoritative.

Provider failure

Disable or invalidate one nonessential provider at a time. Confirm the source trace says failed or timed_out and the call still completes when another adapter succeeds.

Tenant isolation

Run the same barcode for two tenants with different projection policies. Results, memory, aliases, and preferred suppliers must not cross tenant boundaries.

Decision after the pilot

Proceed toward production only if the false-positive rate is acceptable to the inventory managers. Then prioritize, in order:

operations projection and acceptance integration;
provider-specific cache expiry and invalidation;
package/UOM validation in the UI;
approved supplier feed/API imports;
dashboards for precision, coverage, latency, and cost;
ranking changes trained or tuned from acceptance decisions.