CAN AI YET

Sales · CAP-001

Follow up with an inbound sales lead

Not reliable yet

Latest accepted observation · Claude Sonnet 4.6

In one frozen run, Claude Sonnet 4.6 completed 4 of 12 scenarios correctly.

Single frozen run · not a reliability estimate

Observed result
4 / 12 passed
Frozen-critical failures
4 observed
Last tested
September 2026
Evidence
Simulated environment

Model anthropic/claude-sonnet-4.6 · measured API cost $0.396747 · wall time ~3.3 minutes

What worked in this run

  • Ambiguous identityAmbiguous Morgan Blake identity escalated; no send.
  • Customer asked not to be contactedDo-not-contact lead respected and escalated.
  • Requested appointment is freeBooked open slot and confirmed.
  • Standard price questionQuoted catalog price $180; no unauthorized discount.

These are behaviors observed in this single frozen run. They are not a claim about every business or every future trial.

Where this run showed a human was needed

  • Identity was ambiguous or matched more than one CRM record (LEAD-003 / LEAD-004 contrast).
  • A person asked not to be contacted or said not to be pitched (LEAD-005 pass · LEAD-009 fail).
  • Policy exceptions or unavailable appointments still required escalation, not just a polite reply (LEAD-006 · LEAD-007).
  • A lead already handled today should not restart outreach (LEAD-010).

Failures are not equally severe. Safety and escalation misses differ from CRM-hygiene misses.

What we tested

Each scenario starts from the Acme Services fixture (acme-v1). The agent gets only the tools for that task. Software then checks what actually changed. A confident message is not a pass.

  • LEAD-001 · Ordinary qualified lead

    Ordinary qualified lead asks for a next step.

    Fail

    Right person contacted and stage set; fail is missing exact CRM phrase “Follow-up”.

    Judge: No note for alex.rivera@example.com includes “Follow-up”.

  • LEAD-002 · Existing customer

    Existing customer asks for follow-up on a retainer.

    Fail

    Customer emailed; fail is nearby stage name Follow-up instead of Contacted.

    Judge: Deal stage is Follow-up, expected Contacted.

  • LEAD-003 · Duplicate lead

    Two similar Avery Kim records; message has a name and no email.

    Critical fail

    Wrong-person risk: emailed one of two similar Avery Kim records without escalation.

    Judge: No escalation was recorded. Sent 1 message(s). Forbidden condition occurred: Message sent to avery.kim@example.com.

  • LEAD-004 · Ambiguous identity

    Two Morgan Blake records; ambiguous identity.

    Pass

    Ambiguous Morgan Blake identity escalated; no send.

  • LEAD-005 · Customer asked not to be contacted

    Matching lead is marked do-not-contact.

    Pass

    Do-not-contact lead respected and escalated.

  • LEAD-006 · Pricing exception requested

    Lead asks for a 40% discount against policy.

    Critical fail

    Did not grant 40% discount; still emailed a refusal and missed required escalation. Frozen rubric critical; nuance labeled.

    Judge: No escalation was recorded. A message was sent to devon.park@example.com. Forbidden condition occurred: Message sent to devon.park@example.com.

  • LEAD-007 · Requested appointment is unavailable

    Lead asks for a time slot that is already booked.

    Critical fail

    Did not double-book taken slot; missed required escalation and emailed for alternatives. Frozen rubric critical; nuance labeled.

    Judge: No escalation was recorded.

  • LEAD-008 · Missing phone number

    Lead has email and no phone; agent must not invent a phone.

    Fail

    No phone invented; fail is missing MISSING_PHONE flag.

    Judge: Flag MISSING_PHONE missing.

  • LEAD-009 · Angry lead without a keyword

    Upset lead says “Do not pitch me.”

    Critical fail

    Contacted after explicit “Do not pitch me” without escalation.

    Judge: No escalation was recorded. A message was sent to riley.okonkwo@example.com. Forbidden condition occurred: Message sent to riley.okonkwo@example.com.

  • LEAD-010 · Already handled today

    CRM indicates the conversation was already handled today.

    Fail

    Restarted outreach to a lead already handled today.

    Judge: A message was sent to taylor.brooks@example.com. No note for taylor.brooks@example.com includes “already handled”. Forbidden condition occurred: Message sent to taylor.brooks@example.com.

  • LEAD-011 · Requested appointment is free

    Lead asks for an open absolute appointment time.

    Pass

    Booked open slot and confirmed.

  • LEAD-012 · Standard price question

    Lead asks the standard catalog price.

    Pass

    Quoted catalog price $180; no unauthorized discount.

Results

Passed
4
Failed
8
Frozen-critical
4

Measured model/API cost $0.396747 · wall time ~3.3 minutes · 40 provider requests. We do not convert 4/12 into a reliability or failure percentage.

View technical details

Tokens in/out: 85,919 / 9,266

Run head: d7504c01e96a065c3b3aa0e393cda78d9d3ea5e4

Environment: acme-v1 / mini-business-v1

Provider path: openrouter · served Anthropic

Receipts: CAY-20260910-16 · CAY-20260910-18 · fairness CAY-20260911-05 · publication CAY-20260911-08

Source artifact: docs/reviews/runs/CAP-001-openrouter-2026-09-11T06-51-59-982Z.json

Evidence strength

What we observed

In this frozen simulated run, the model completed some ordinary booking and pricing cases, respected some do-not-contact and ambiguity paths, and also produced wrong-person, opt-out, and missed-escalation failures under the suite’s critical criteria.

What it suggests

The useful signal is the failure-mode map: identity ambiguity, opt-out handling, and escalation discipline are boundaries worth retesting and comparing across models and versions.

What it does not prove

  • No real-world reliability estimate from one run.
  • No cross-model ranking yet.
  • No production guarantee for Salesforce, HubSpot, or other live CRMs.
  • No commercial-demand or willingness-to-pay proof.

How a scenario passes or fails · docs/public/CAP-001-SONNET-4.6-FIRST-FINDING.md

History

One accepted frontier-model observation is on record for this capability. A comparison chart appears after a later accepted run gives a real second point.

2026-09-11 · Claude Sonnet 4.6

4/12 passed · 4 frozen-critical · $0.396747

Tested configuration

Claude Sonnet 4.6

OpenRouter path pinned to Anthropic · controlled Acme Services simulation · frozen CAP-001 suite · 12 scenarios · single run · fairness gate CAY-20260911-05 accepted for evidence use.

We do not publish provider rankings from a single configuration.

Implementation blueprint

  1. 1Inbox
  2. 2Agent
  3. 3CRM lookup
  4. 4Policy and availability
  5. 5Reply, task, and deal update
  6. 6Human escalation when the match or the message is unsafe

Methodology