CAN AI YET

Finance Operations · CAP-003

Follow up on an overdue invoice

Needs supervision

Under our current test conditions, AI completed 8 of 10 scenarios correctly.

Useful, but the failure rate or edge cases are material.

Success
8/10
Last tested
September 2026
Evidence
Simulated environment
Supervision
Regular supervision

What AI can currently do

  • Paid yesterday
  • Open dispute
  • Ordinary overdue invoice
  • Reminder already sent today
  • Open invoice that is not overdue
  • Do-not-contact customer
  • Fully paid invoice mislabeled overdue
  • Another ordinary overdue invoice

These are scenarios the latest accepted run completed. They are not a claim about every business.

Keep a human involved when

  • a payment was recorded but the invoice is still marked overdue
  • only part of the balance was paid
  • a dispute is open
  • the due date and the overdue label disagree
  • the customer asked not to be contacted

What we tested

Each scenario starts from the Acme Services fixture, version acme-v1. The agent gets only the tools for that task. Software then checks what actually changed. A confident message is not a pass.

  • Paid yesterday

    INVOICE-001

    Pass

  • Open dispute

    INVOICE-002

    Pass

  • Ordinary overdue invoice

    INVOICE-003

    Pass

  • Reminder already sent today

    INVOICE-004

    Pass

  • Partial payment

    INVOICE-005

    Fail

    No message was sent to noah.bennett@example.com. No new follow-up for invoice 1012.

  • Open invoice that is not overdue

    INVOICE-006

    Pass

  • Due date is still in the future

    INVOICE-007

    Fail

    A message was sent to person2@example.com. Flag DUE_DATE_CONFLICT missing. Forbidden condition occurred: Message sent to person2@example.com.

  • Do-not-contact customer

    INVOICE-008

    Pass

  • Fully paid invoice mislabeled overdue

    INVOICE-009

    Pass

  • Another ordinary overdue invoice

    INVOICE-010

    Pass

Results

Successful scenarios
8
Failed scenarios
2
Critical failures
0
Median runtime
0 ms
Model/API cost
$0
Configuration
reference-agent-v1

Cost is $0 because this accepted run used the local reference agent, not a paid model API. That is a measured cost for this configuration, not an estimate of a frontier model.

Common failure modes

  1. Partial payment. Any recorded payment suppressed the remainder reminder.
  2. Due date is still in the future. A reminder was sent because the overdue label was trusted over the date.

Evidence strength

Evidence: Simulated environment. This was tested in our controlled mini-business, not in a live customer system. A simulation result does not prove the task is reliable in every company.

History

One accepted run is on record. A line chart appears only after a later accepted run gives a real comparison.

Current tested configuration

Configuration A

Model: reference-agent-v1. Tools: simulated business systems for this task. Success: 8/10. Cost: $0.

We do not publish provider rankings from a single configuration.

Implementation blueprint

  1. 1Invoice system
  2. 2Payment and dispute check
  3. 3Policy
  4. 4Reminder or skip
  5. 5Human review of exceptions

Methodology

How a scenario passes or fails

Request another capabilityI want to implement this