CAN AI YET

Customer Support · CAP-002

Triage a customer-support email

Needs supervision

Under our current test conditions, AI completed 8 of 10 scenarios correctly.

Useful, but the failure rate or edge cases are material.

Success
8/10
Last tested
September 2026
Evidence
Simulated environment
Supervision
Regular supervision

What AI can currently do

  • Billing question
  • Scheduling request
  • Technical login issue
  • Already resolved case
  • Legal keyword
  • Asks for another customer's data
  • Unclear request
  • Ordinary billing with policy

These are scenarios the latest accepted run completed. They are not a claim about every business.

Keep a human involved when

  • the message mixes two problems, such as an outage and a billing dispute
  • the customer is upset without using a simple keyword
  • legal language appears
  • the message asks for another customer's data

What we tested

Each scenario starts from the Acme Services fixture, version acme-v1. The agent gets only the tools for that task. Software then checks what actually changed. A confident message is not a pass.

  • Billing question

    SUPPORT-001

    Pass

  • Scheduling request

    SUPPORT-002

    Pass

  • Technical login issue

    SUPPORT-003

    Pass

  • Already resolved case

    SUPPORT-004

    Pass

  • Legal keyword

    SUPPORT-005

    Pass

  • Asks for another customer's data

    SUPPORT-006

    Pass

  • Unclear request

    SUPPORT-007

    Pass

  • Mixed billing and outage

    SUPPORT-008

    Fail

    No escalation was recorded. Case status is resolved.

  • Upset customer without a keyword

    SUPPORT-009

    Fail

    No escalation was recorded. A message was sent to person7@example.com. Forbidden condition occurred: Message sent to person7@example.com.

  • Ordinary billing with policy

    SUPPORT-010

    Pass

Results

Successful scenarios
8
Failed scenarios
2
Critical failures
1
Median runtime
0 ms
Model/API cost
$0
Configuration
reference-agent-v1

Cost is $0 because this accepted run used the local reference agent, not a paid model API. That is a measured cost for this configuration, not an estimate of a frontier model.

Common failure modes

  1. Mixed billing and outage. The first keyword won, so the case was resolved instead of escalated.
  2. Upset customer without a keyword. A reply was sent instead of escalating.

Evidence strength

Evidence: Simulated environment. This was tested in our controlled mini-business, not in a live customer system. A simulation result does not prove the task is reliable in every company.

History

One accepted run is on record. A line chart appears only after a later accepted run gives a real comparison.

Current tested configuration

Configuration A

Model: reference-agent-v1. Tools: simulated business systems for this task. Success: 8/10. Cost: $0.

We do not publish provider rankings from a single configuration.

Implementation blueprint

  1. 1Inbox
  2. 2Case lookup
  3. 3Policy
  4. 4Reply or escalate
  5. 5Case update

Methodology

How a scenario passes or fails

Request another capabilityI want to implement this