# Agent Background Check — Scoring Rubric v1.2

*Drafted July 9, 2026; v1.1 and v1.2 same day after calibration batches 1 and 2. Hand-scored. Target: ≤30 min per agent.*

**v1.2 changelog:** historical deception now qualifies for Class B treatment ONLY with public acknowledgment or retraction. Silence keeps it Class A regardless of age or leadership changes. Decision made after the 11x case (batch 2): fabrication that was never owned gets no statute of limitations. Result: 11x regraded Unverified → Flagged.

**v1.1 changelog:** red-flag rule split into two classes (operator deception → Flagged; behavioral/historical → Provisional cap) after batch 1 showed the single override would have flagged 3 of 5 agents. Weights, evidence tiers, and scale unchanged — they survived calibration.

## Philosophy

Score only what can be verified. The core move — borrowed from recruiting — is separating **claims** from **evidence**. Existing registries (ERC-8004, agent directories) are job boards: anyone can post anything. This rubric is the reference check.

Every score is capped by the quality of evidence behind it. An agent with a glowing self-written description and no verifiable history scores near zero.

## Evidence Tiers

| Tier | Name | Definition | Recruiting analogy |
|------|------|------------|--------------------|
| E0 | Self-reported | Vendor's own claims (site, agent card, docs) | The resume |
| E1 | Documented | Public artifacts: changelogs, status pages, pricing history, GitHub activity, incident postmortems | Portfolio / public work |
| E2 | Corroborated | Independent sources: named customer references, third-party reviews, audit reports, marketplace transaction data | Reference calls |
| E3 | Observed | You (or a trusted validator) ran the agent on test tasks and logged the results | Work trial |

**Cap rule:** a criterion scored on E0 evidence alone maxes at 1/4. E1 maxes at 2. E2 maxes at 3. Only E3 or multiple independent E2 sources can justify 4.

## Dimensions & Weights

### 1. Identity & Provenance — 20%

- **1a. Operator traceability (10%)** — Is there a named, findable legal entity or individual accountable for the agent? Registered company, real contacts, jurisdiction known?
- **1b. Version & change disclosure (5%)** — Does the operator disclose model/prompt/capability changes? An agent that silently changes behavior can't hold a track record.
- **1c. Declared scope (5%)** — Are capabilities, permissions, and data handling explicitly declared (agent card, docs)? Vague scope = unverifiable behavior.

### 2. Track Record — 30%

- **2a. Volume & tenure (10%)** — Verifiable evidence of real usage over time. Months in production, task volume, active deployments.
- **2b. Success / failure rate (12%)** — Completion rates, error rates, refund/redo rates on its core task. Hardest to get; weight it accordingly when you do.
- **2c. Incident history (8%)** — Known failures, security incidents, data leaks — and how the operator responded. Disclosed-and-fixed scores *higher* than no-known-incidents-because-nobody-looked.

### 3. Reliability & Safety Behavior — 20%

- **3a. Scope adherence (8%)** — Does observed behavior match declared scope? (E.g., declared read-only but hits write endpoints.)
- **3b. Failure mode (7%)** — When it can't do a task, does it fail loudly (escalate, refuse, report) or silently (hallucinate completion)? Silent failure is the cardinal sin.
- **3c. Data handling (5%)** — Evidence of respecting data boundaries: retention policy, no training on customer data without consent, credential hygiene.

### 4. Transparency & Accountability — 15%

- **4a. Auditability (6%)** — Does it produce logs/traces a customer can inspect?
- **4b. Recourse (5%)** — Dispute process, SLA, human escalation path, refund policy.
- **4c. Honesty of marketing (4%)** — Do E0 claims survive contact with E1+ evidence? Caught inflating = score 0 here and flag globally.

### 5. External Validation — 15%

- **5a. Third-party attestation (8%)** — SOC 2, pentest reports, identity certs (AgentStamp-type), registry entries that actually resolve.
- **5b. Independent references (7%)** — Named customers who'll confirm usage; substantive third-party reviews.

## Scoring Scale (per criterion)

- **0** — No evidence, or contradicting evidence
- **1** — Self-reported only (E0)
- **2** — Documented (E1)
- **3** — Independently corroborated (E2)
- **4** — Observed/tested (E3) or multiple independent E2

Composite = weighted sum × 25 → **0–100**.

## Grades

| Score | Grade | Meaning |
|-------|-------|---------|
| 80–100 | Verified — Strong | Safe to delegate high-stakes work |
| 60–79 | Verified | Track record confirmed, minor gaps |
| 40–59 | Provisional | Some verification, meaningful unknowns |
| 20–39 | Unverified | Mostly claims, little evidence |
| 0–19 | Insufficient / Flagged | No basis for trust, or red flag triggered |

**Default state is Unverified.** No agent starts trusted.

Class A red flag → Flagged. Class B red flag → best achievable grade is Provisional.

## Red Flags — Two Classes (v1.1)

**Class A — Operator deception → grade forced to Flagged, regardless of composite:**

1. Fabricating or materially misrepresenting track record, benchmarks, or capabilities — within the last 12 months, *or at any time if never publicly acknowledged or retracted* (v1.2)
2. False task-completion reporting observed (agent claims work done that wasn't, with fabricated evidence)
3. Spoofed or non-resolving registry/attestation entries
4. Undisclosed security incident involving customer data
5. Operator identity untraceable (no legal entity, anonymous, unreachable)

**Class B — Behavioral violations & historical issues → grade capped at Provisional, composite unchanged:**

1. Observed scope violation by the agent (acting beyond declared permissions) without evidence of operator intent
2. Historical misrepresentation (>12 months old) that the operator has **publicly acknowledged or retracted** (v1.2). Unacknowledged deception remains Class A no matter how old — new leadership, cleaned-up websites, and silence do not qualify. The registry forgives owned mistakes, not buried ones.
3. Materially slow or evasive response to responsibly disclosed vulnerabilities

The distinction: Class A is about whether the *operator* can be trusted to tell the truth; Class B is about whether the *agent's behavior* has verified problems. Deception poisons all other evidence — it earns the hard override. Behavior problems are already priced into dimension scores; the cap just prevents a "Verified" badge from coexisting with a known violation.

## 30-Minute Scoring Workflow

1. **(5 min)** Pull E0: agent card, site, docs. Record claims.
2. **(10 min)** Pull E1: GitHub, changelog, status page, incident history, registry entries (check they resolve).
3. **(10 min)** Pull E2: reviews, audits, references, marketplace data. Note what's unobtainable.
4. **(5 min)** Score each criterion, apply cap rule, check red flags, write 3-sentence summary.

Log every source URL per criterion — the audit trail *is* the product.

## Machine-Readable Output (agent-to-agent path)

Each completed check emits JSON so agents can eventually query scores before delegating:

```json
{
  "agent_id": "did/URL/ERC-8004 token",
  "operator": "legal entity",
  "score": 67,
  "grade": "verified",
  "dimensions": {
    "identity": {"score": 3.2, "max_evidence_tier": "E2"},
    "track_record": {"score": 2.4, "max_evidence_tier": "E1"},
    "reliability": {"score": 2.8, "max_evidence_tier": "E2"},
    "transparency": {"score": 3.0, "max_evidence_tier": "E1"},
    "external_validation": {"score": 2.0, "max_evidence_tier": "E1"}
  },
  "red_flags": [],
  "grade_cap_reason": null,
  "evidence_urls": ["..."],
  "scored_at": "2026-07-09",
  "valid_until": "2026-10-09",
  "rubric_version": "1.1"
}
```

Scores expire after 90 days (agents change fast; stale trust is worse than no trust). Format designed to slot into ERC-8004's Validation Registry or an A2A agent-card extension later — we don't compete with identity standards, we supply the verification substance they lack.

## Deliberately Out of Scope for v1

- Automated behavioral testing (E3 at scale) — later, once manual process is proven
- Crypto attestation / on-chain publishing — piggyback on ERC-8004 when volume justifies it
- Model-level evals (capability benchmarks) — we verify *operational* trust, not intelligence

## Next Steps

1. ~~Score 5 well-known public agents as calibration~~ Done — see calibration-batch-1.md. Prediction held: 3 of 5 Provisional.
2. ~~Pressure-test the weights~~ Done — rankings matched intuition, weights unchanged.
3. Score batch 2 (see batch-2-picks.md) — chosen to stress rubric parts batch 1 didn't.
4. Publish first 10 scorecards as the seed registry.