Marketing AI evaluation guide
How to evaluate a marketing AI agent
Eight question types, six subject areas and twenty-four ready-to-run requests. See where MCP supplies data and actions, and how to verify every result.
The four-layer test
Every tool shows a demo. A demo proves at most one of these layers.
- Data access Can it reach your systems, with definitions and freshness checked first?
- Output quality Does the answer hold? Totals reconcile, sources and limits stay visible.
- Action execution Does a change land on the intended object, confirmed by readback?
- Business effect Does it move a metric you measure? Only experiments prove lift.
The twenty-four questions below test all four.
What each layer proves, and what it misses
An agent can write a confident, well-structured answer about last month's cost per acquisition (CPA) that is wrong because it read the wrong column. A rubric or a judge model grades the writing. It catches the wrong column only when it can see the source data or the right answer, so the output test needs one of the two.
Execution tests start where output tests stop: a perfect budget recommendation can land on the wrong campaign, and only a read of the system state afterwards shows it. The agent's report of what it did is a claim to verify by reading the system.
The business effect needs a third test. A month of correct pauses and budget moves can leave CPA unchanged when every change sat on a campaign that carries 2% of spend, to take one example. A controlled read of the business number, such as a holdout, shows it. That read takes weeks.
| Layer | Question it answers | Evidence it needs | What it cannot see |
|---|---|---|---|
| Data access | What can it read and change, and is the data fresh? | A read-back of the connection: sources, permissions and the last load time | Whether the answer built on that data is right |
| Output quality | Is the answer right, and is it readable? | A known answer, or a rubric and a judge | Whether the agent can act, or whether acting helps |
| Action execution | Did it change the right thing, and nothing else? | A check of the system state after the run | Whether the change was worth making |
| Business effect | Did the number move, against a control? | A holdout and weeks of data | Which step of the agent's work caused it |
Eight question types
Every question in the map asks one of these. Each type carries an acceptance check: what a correct answer must show before you trust it.
-
Describe
What happened?
Done when Reconciled facts, comparisons and composition.
-
Explain
Why did it happen?
Done when Tested explanations and explicit limits.
-
Forecast
What might happen next?
Done when Conditional projections and assumptions.
-
Choose
What should we do?
Done when Options, constraints and trade-offs.
-
Prepare
What should we create?
Done when A reviewable brief, draft or plan.
-
Execute
What should change in a system?
Done when A confirmed change to the intended object.
-
Monitor
When should we intervene?
Done when Repeated checks and a delivered signal.
-
Verify
Can we trust this?
Done when Reconciliation or final-state confirmation.
Subjects are a separate axis: paid media, funnel, creative, budget, measurement, and CRM and operations. Execution mode is separate too - read, prepare a draft, change a system or repeat a check. A question type does not grant permission to act.
Twenty-four questions, ready to run
Filter by question type and subject, or search. Every card carries its inputs, its acceptance check, the MCP role and one limit, plus a request you can copy into your own agent.
Showing all 24 questions
-
How much did we spend, and what did we get?
You get Spend and results for an agreed period.
MCP Reads connected ad platforms or the warehouse.
How to use and verify
Inputs Account, dates, currency and the definition of a result.
Done when Totals reconcile with details. Sources and freshness are visible.
One limit Different attribution windows need an explicit reconciliation.
How much did we spend, and what did we get? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Spend and results for an agreed period. Acceptance: Totals reconcile with details. Sources and freshness are visible. Limitation: Different attribution windows need an explicit reconciliation. For now, read and prepare only. Do not change external systems.
-
Which channels generated qualified meetings?
You get A channel comparison using one definition of a meeting.
MCP Retrieves CRM data and observed lead journeys.
How to use and verify
Inputs CRM, lead source, statuses and event history.
Done when Meetings are deduplicated. Channel assignment rules are visible.
One limit An attributed meeting does not prove incremental channel impact.
Which channels generated qualified meetings? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A channel comparison using one definition of a meeting. Acceptance: Meetings are deduplicated. Channel assignment rules are visible. Limitation: An attributed meeting does not prove incremental channel impact. For now, read and prepare only. Do not change external systems.
-
Which ad formats perform best?
You get A format comparison with the volume of observations.
MCP Retrieves assets and performance from available sources.
How to use and verify
Inputs Assets, format labels, spend and conversions.
Done when Sample sizes, segments and comparability are visible.
One limit Different audiences can explain differences in average performance.
Which ad formats perform best? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A format comparison with the volume of observations. Acceptance: Sample sizes, segments and comparability are visible. Limitation: Different audiences can explain differences in average performance. For now, read and prepare only. Do not change external systems.
-
Why did cost per lead increase?
You get A breakdown of the change and tested explanations.
MCP Retrieves metrics, settings and available history.
How to use and verify
Inputs Clicks, spend, leads, settings and change history.
Done when Every explanation has evidence. Unproven causes are labelled.
One limit Changes happening together do not establish causation.
Why did cost per lead increase? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A breakdown of the change and tested explanations. Acceptance: Every explanation has evidence. Unproven causes are labelled. Limitation: Changes happening together do not establish causation. For now, read and prepare only. Do not change external systems.
-
Where are potential customers dropping out?
You get A loss map between agreed funnel stages.
MCP Retrieves CRM history and measurement events.
How to use and verify
Inputs Transition events and comparable lead cohorts.
Done when Conversion delays are considered. Cohorts have comparable maturity.
One limit A missing event can be a data problem.
Where are potential customers dropping out? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A loss map between agreed funnel stages. Acceptance: Conversion delays are considered. Cohorts have comparable maturity. Limitation: A missing event can be a data problem. For now, read and prepare only. Do not change external systems.
-
Did performance fall, or are the data delayed?
You get A freshness check and location of the gap.
MCP Reads load status and available source records.
How to use and verify
Inputs Latest source records, load status and report records.
Done when Source, load and calculation use the same period.
One limit An empty table does not establish zero business activity.
Did performance fall, or are the data delayed? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A freshness check and location of the gap. Acceptance: Source, load and calculation use the same period. Limitation: An empty table does not establish zero business activity. For now, read and prepare only. Do not change external systems.
-
Will we exceed the budget by month end?
You get A spend forecast under a stated spending pace.
MCP Reads spend, constraints and current settings.
How to use and verify
Inputs Spend, budget limit, calendar and planned changes.
Done when Calculation date, assumptions and alternative scenarios are visible.
One limit Extending the current pace is a conditional projection.
Will we exceed the budget by month end? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A spend forecast under a stated spending pace. Acceptance: Calculation date, assumptions and alternative scenarios are visible. Limitation: Extending the current pace is a conditional projection. For now, read and prepare only. Do not change external systems.
-
How many meetings could we generate by period end?
You get Scenarios that account for volume and conversion delay.
MCP Retrieves historical events and the current queue.
How to use and verify
Inputs Inbound history and time to a qualified meeting.
Done when Known volume is separated from future assumptions.
One limit A different lead mix can invalidate the historical conversion rate.
How many meetings could we generate by period end? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Scenarios that account for volume and conversion delay. Acceptance: Known volume is separated from future assumptions. Limitation: A different lead mix can invalidate the historical conversion rate. For now, read and prepare only. Do not change external systems.
-
What would change if we increased the budget?
You get Conditional scenarios with explicit uncertainty.
MCP Retrieves spend, outcome and settings history.
How to use and verify
Inputs Constraints, past budget changes and any available response curve.
Done when No linear increase in leads is promised without response evidence.
One limit Average cost per lead is not marginal cost per additional lead.
What would change if we increased the budget? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Conditional scenarios with explicit uncertainty. Acceptance: No linear increase in leads is promised without response evidence. Limitation: Average cost per lead is not marginal cost per additional lead. For now, read and prepare only. Do not change external systems.
-
Where should we reallocate part of the budget?
You get Allocation options and the cost of each trade-off.
MCP Supplies performance and constraints for the calculation.
How to use and verify
Inputs Objective, limits, response evidence and learning constraints.
Done when Keeping the current allocation is an option. Advice is separate from action.
One limit Tool access does not authorize spending changes.
Where should we reallocate part of the budget? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Allocation options and the cost of each trade-off. Acceptance: Keeping the current allocation is an option. Advice is separate from action. Limitation: Tool access does not authorize spending changes. For now, read and prepare only. Do not change external systems.
-
Which hypothesis should we test next?
You get A ranked set of hypotheses and ways to evaluate them.
MCP Retrieves past tests, assets and results.
How to use and verify
Inputs Past results, weak points and available resources.
Done when Hypotheses are testable and success is defined in advance.
One limit An idea list alone does not establish expected lift.
Which hypothesis should we test next? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A ranked set of hypotheses and ways to evaluate them. Acceptance: Hypotheses are testable and success is defined in advance. Limitation: An idea list alone does not establish expected lift. For now, read and prepare only. Do not change external systems.
-
Which campaigns should we investigate first?
You get An investigation queue based on possible loss and confidence.
MCP Collects current performance across campaigns.
How to use and verify
Inputs Metric changes, spend volume and data freshness.
Done when Priority is explained. High spend is not treated as a cause.
One limit Investigation priority does not authorize automatic pausing.
Which campaigns should we investigate first? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: An investigation queue based on possible loss and confidence. Acceptance: Priority is explained. High spend is not treated as a cause. Limitation: Investigation priority does not authorize automatic pausing. For now, read and prepare only. Do not change external systems.
-
Prepare a brief for the next campaign.
You get A reviewable brief with an objective, audience, message and constraints.
MCP Retrieves context and assets available to the current user.
How to use and verify
Inputs Product context, brand rules, past assets and findings.
Done when Requirements cite sources. Missing information is labelled.
One limit The quality of a brief still needs subject-matter review.
Prepare a brief for the next campaign. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A reviewable brief with an objective, audience, message and constraints. Acceptance: Requirements cite sources. Missing information is labelled. Limitation: The quality of a brief still needs subject-matter review. For now, read and prepare only. Do not change external systems.
-
Write ad variants from the approved brief.
You get Drafts in the required format with supported claims.
MCP Retrieves the brief and supporting material.
How to use and verify
Inputs Approved brief, format limits and allowed evidence.
Done when No invented facts. The tone and format match the requirements.
One limit Copy is proven by a click-through or reply test against the control, not by the draft.
Write ad variants from the approved brief. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Drafts in the required format with supported claims. Acceptance: No invented facts. The tone and format match the requirements. Limitation: Copy is proven by a click-through or reply test against the control, not by the draft. For now, read and prepare only. Do not change external systems.
-
Prepare a campaign measurement plan.
You get Events, metric definitions and validation checks.
MCP Exposes available fields and sources.
How to use and verify
Inputs Objective, user journey, available systems and data.
Done when Every metric has a source, calculation and owner.
One limit The plan still needs implementation and verification on actual events.
Prepare a campaign measurement plan. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Events, metric definitions and validation checks. Acceptance: Every metric has a source, calculation and owner. Limitation: The plan still needs implementation and verification on actual events. For now, read and prepare only. Do not change external systems.
-
Create the approved campaign in a paused state.
You get Created objects with confirmed settings.
MCP Submits supported requests and reads the resulting objects.
How to use and verify
Inputs Approved brief, account, permissions and budget constraints.
Done when Objects are read back. Delivery has not been activated.
One limit A prepared campaign and a live campaign are different outcomes.
Prepare a plan for this task: Create the approved campaign in a paused state. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Created objects with confirmed settings. Verification: Objects are read back. Delivery has not been activated. For now, read and prepare only. Do not change external systems.
-
Update the exclusion audience from CRM.
You get A change to the authorized audience membership only.
MCP Reads CRM and writes through a supported advertising API.
How to use and verify
Inputs Unambiguous selection rules and an agreed source.
Done when Included and excluded records are checked. No extra changes.
One limit Support and permissions are checked for the specific connection.
Prepare a plan for this task: Update the exclusion audience from CRM. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: A change to the authorized audience membership only. Verification: Included and excluded records are checked. No extra changes. For now, read and prepare only. Do not change external systems.
-
Update verified contact fields.
You get Confirmed changes to the intended records.
MCP Reads records, makes authorized updates and checks them.
How to use and verify
Inputs Exact contact key, value provenance and allowed fields.
Done when The right records changed and unrelated records did not.
One limit Identify a contact by its exact key, an email or a CRM id, and treat a name match as a candidate.
Prepare a plan for this task: Update verified contact fields. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Confirmed changes to the intended records. Verification: The right records changed and unrelated records did not. For now, read and prepare only. Do not change external systems.
-
Alert me when spend deviates from plan.
You get A repeated check with a clear alert.
MCP Supplies data. A workflow handles scheduling and delivery.
How to use and verify
Inputs Plan, schedule, threshold, recipient and expiry condition.
Done when Alerts are delivered, duplicates controlled and check failures visible.
One limit Monitoring counts as working after a deliberately failed check produces a visible alert.
Prepare monitoring for this request: Alert me when spend deviates from plan. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Alerts are delivered, duplicates controlled and check failures visible. For now, read and prepare only. Do not create schedules or send messages.
-
Warn me when data arrive late.
You get An alert when agreed freshness is violated.
MCP Reads status and timestamps of the latest data.
How to use and verify
Inputs Expected update frequency and acceptable delay.
Done when Source delay, load delay and report delay are distinguished.
One limit Silence proves nothing. Confirm monitoring with a test alert.
Prepare monitoring for this request: Warn me when data arrive late. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Source delay, load delay and report delay are distinguished. For now, read and prepare only. Do not create schedules or send messages.
-
Flag signs of ad fatigue.
You get Candidates for review or asset refresh.
MCP Retrieves ad history and results.
How to use and verify
Inputs History of frequency, clicks, conversion and delivery conditions.
Done when Periods are comparable. A signal is separated from a proven cause.
One limit Read a click-through drop together with frequency and conversion rate before calling it fatigue.
Prepare monitoring for this request: Flag signs of ad fatigue. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Periods are comparable. A signal is separated from a proven cause. For now, read and prepare only. Do not create schedules or send messages.
-
Why does the report disagree with the ad platform?
You get A staged reconciliation and any unexplained remainder.
MCP Reads the source system and the final table.
How to use and verify
Inputs One account, period, currency and attribution definition.
Done when The discrepancy is quantified and localized.
One limit Different metric definitions are not automatically a load error.
Why does the report disagree with the ad platform? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A staged reconciliation and any unexplained remainder. Acceptance: The discrepancy is quantified and localized. Limitation: Different metric definitions are not automatically a load error. For now, read and prepare only. Do not change external systems.
-
Do we have enough evidence to name a test winner?
You get A review of design, sample and the permitted conclusion.
MCP Retrieves available events and test history.
How to use and verify
Inputs Assignment, metric, dates, stopping rules and data loss.
Done when Cohort comparability and uncertainty are assessed.
One limit A spend threshold is not a test of statistical significance.
Do we have enough evidence to name a test winner? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A review of design, sample and the permitted conclusion. Acceptance: Cohort comparability and uncertainty are assessed. Limitation: A spend threshold is not a test of statistical significance. For now, read and prepare only. Do not change external systems.
-
Did the change actually persist?
You get Confirmation of the intended object state.
MCP Reads the final state through the relevant connection.
How to use and verify
Inputs Exact object, expected fields and operation response.
Done when There is a readback. An unknown result does not trigger a blind retry.
One limit A dispatched command is not a confirmed change.
Did the change actually persist? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Confirmation of the intended object state. Acceptance: There is a readback. An unknown result does not trigger a blind retry. Limitation: A dispatched command is not a confirmed change. For now, read and prepare only. Do not change external systems.
No questions match. Clear the search or choose All.
Where MCP participates: it retrieves accessible facts and sends supported, authorized actions. The model selects the analytical approach. Data models, calculations and checks decide whether the conclusion is acceptable.
Why cost per lead rose: three causes the numbers point to
We chose "Why did cost per lead increase?" because that number rose 3.9x between March and April 2026 in our own accounts. It is card Q04 above. We ran it once, on our own ad accounts, read-only, on 19 October 2026, so treat the ratios below as one reading. The figures compare March with April 2026.
-
Data access
In our run the agent named its data limits before answering. Spend was loaded through 19 October, and the newest lead was from 18 October. One account had lost its connection in August, and the agent flagged it. It did not estimate that account's numbers. The test needs the ad platforms, the site forms and the CRM through one connection with set permissions, read-only for this question.
-
Output quality
The answer passed on four counts. It agreed on definitions first: what counts as a lead, which period, which currency, which timezone. "How many leads did we get in March" had four honest answers in our accounts, depending on which form and which attribution you count. The smallest was at most a third of the largest, so the agent had to name the one it used. It reconciles the numbers: one platform's spend matched the warehouse, and the gap on another was traced to a second account whose connection had broken. Causes it could not test stay labelled as untested. The agent proposed three causes and checked each against the data. None was tested against a control.
The three causes and the evidence behind each, March to April 2026 Cause Evidence A paused source The biggest lead source had all its campaigns paused in March. Dropped campaigns Two cheap mid-funnel campaigns were dropped that month. Spend with no leads One platform took half of all spend in both months and brought no leads at all. In that run, comparing March with April, the mix shift alone works out to a 3.2x factor. The remaining source's own cost per lead rose 1.2x on top. Both are estimates from that one run, built on each source's March and April spend and lead counts, and no control tested them. The two rounded factors multiply to 3.84, and the unrounded figures give 3.9x. The next step was written down with its verification and left for a person to approve.
-
Action execution
The agent proposed one action, pausing the empty campaign, with a way to verify it, and stopped. This question writes nothing by design: its card says read and prepare only. The write is tested on an Execute card, such as Q16 above: create the approved campaign in a paused state. After the run you read the platform back through the API. The campaign exists, its status is paused, its budget is the agreed number, and the count of campaigns rose by exactly one. Then check that nothing else changed. If the agent reports success and the read-back disagrees, the read-back wins. Pass it on five fresh briefs, one after the account's data has changed. If all five pass, the Execute type reaches "checked now". A clean sheet still leaves the failure rate at most about 45%. That is the highest rate that fits zero failures in five independent runs, at 95% confidence and a steady failure rate. The formula is 1 minus the fifth root of 0.05. "Works repeatedly" is earned over the weeks of runs that follow.
-
Business effect
The business effect here is a plan. The read starts once a person acts on the answer. Pause the empty campaigns, hold back a matched set, and compare cost per lead once the number has settled. The scorecard below gives the sample sizes for this read. That read measures the decision. The agent is measured by running the same question through your current process as the control and comparing the two. In our accounts the four answers above moved with the attribution setting, so platform-reported conversions on their own cannot separate the agent's effect from the market's. Our run covered data access and output quality. The agent proposed the pause, and the business effect read starts once a person runs it.
Sixteen public benchmarks, and none measures business effect
| Benchmark | What it grades, and how | Test it covers | Limit for your stack | Verdict |
|---|---|---|---|---|
| Mark-Bench (Optimizely, September 2026) | Marketing deliverables such as briefs, plans and copy. Each scenario has a brief, input files and a rubric of 12 to 30 criteria. Large language model (LLM) judges, Claude and Gemini, score the output. The launch article counts 297+ tasks and scenarios across 17 functions, with 3,564 criteria. | Output quality | It never acts in a system. The judge is a model, and the cost figures compare the vendor's own test setup, the software around the model, with a raw model. | Screen only |
| AD-Bench (Tencent, February 2026, revised June 2026) | 225 real marketing-analysis requests from a production advertising platform, in three difficulty levels. An LLM judge, Gemini-3.1-Pro, grades the answer and whether the agent's tool calls match an expert's. Best model Claude-Opus-4.7 at Pass@1 76.9%, falling to 61.4% on the hardest level. | Output quality on Describe and Explain questions | Any change to the platform. The judge is also one of the models under test, which the authors flag. | Screen only: a public test of Describe and Explain questions, the same types as the cards above |
| AutomationBench (Zapier, April 2026) | Cross-app workflows in sales, marketing, operations, support, finance and HR. 600 public tasks, 100 per domain, plus a harder private set. Code compares the final system state with the expected state using assertion-based rubrics. A task passes only if every assertion passes. Top of the leaderboard (v1.0.6): Gemini 4 Argon (High) at about 51% on Zapier's private, harder task set, leaderboard read on 20 October 2026 (Zapier says run-to-run variance is typically within 1%). | Action execution | It grades the end state of a task on its own set of apps. The quality of a deliverable and your own accounts are outside it. | Run your own: copy its grader design |
| GDPval (OpenAI, October 2025) | 1,320 one-shot work tasks across 44 occupations. The 220-task open subset is graded by blinded expert pairwise comparison. The automated grader agrees with human experts 65.7% of the time. Humans agree with each other 70.8%. | Output quality | Anything interactive. None of the 44 occupations is a marketing one. | Not evidence for marketing |
| tau-bench (Sierra, June 2024) | Customer service agents talking to a simulated user in retail and airline settings. Graded by comparing the database state after each task with the expected outcome. Introduced pass^k, the share of tasks an agent passes all k times. | Action execution, with a reliability measure | Its tasks are customer-service chats in retail and airline settings, so no ad account or CRM (customer records) is touched. Its authors now mark the original task sets outdated. | Screen only: borrow pass^k |
| BFCL (Berkeley Function Calling Leaderboard) V4, updated 12 April 2026 | Whether a model calls functions accurately. Two checks: the call's syntax tree against possible answers, and executable runs. Overall accuracy is the unweighted average of the sub-categories. | The first step of execution: the call goes out, the result stays unchecked | Whether the right thing changed in the system afterwards. | Screen only |
| MCP-Universe (August 2025) | 231 tasks run through 11 live MCP (Model Context Protocol) servers, the open-standard connections that give an agent tools and data. GPT-5 at 43.72% success, as published. | Action execution through tools | The servers belong to the benchmark, and no business number is read: a task says whether a call reached the right tool. | Screen only: it shows what tool use reached on those tasks |
| MCP-Bench (August 2025) | 104 tasks across 28 live MCP servers and 250 tools. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only |
| MCPMark (September 2025) | 127 tasks on live MCP servers. Best model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. | Action execution through tools, with a reliability measure | Your servers and your permissions. Any business number. | Screen only: borrow pass^4 |
| LiveMCPBench (August 2025) | 95 tasks on 70 live MCP servers with 527 tools. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only |
| CRMArena-Pro (Salesforce, May 2025) | Nineteen expert-validated tasks across sales, service and configure-price-quote processes, in single-turn and multi-turn settings. Its abstract reports leading agents at around 58% single-turn success, dropping to approximately 35% in multi-turn settings, and near-zero inherent confidentiality awareness. | Output quality on CRM reads and multi-turn conversations | A Salesforce sandbox: no ad platform, and none of your own records. | Run your own: copy the confidentiality test |
| TheAgentCompany (December 2024) | 175 workplace tasks, counted only when every checkpoint passes, with cost printed per task instance next to every score. | Execution on general office work | Its tasks are software-company office work, such as code, admin and HR tickets, so no ad account or CRM is touched. | Not evidence |
| OSWorld 2.0 (leaderboard read on 20 October 2026) | 108 long computer-use workflows. Claude Opus 4.8 at 20.6%, as published. | Execution by computer use | Its tasks are desktop operations on an operating system, so no ad platform or CRM is touched. | Not evidence |
| Vending-Bench (February 2025) | A simulated vending business scored on net worth, each experiment run five times. | A simulated business number | A simulated shop with simulated money: no ad spend and no real profit and loss. | Not evidence |
| xbench, marketing track (tested May 2025) | 50 influencer-search tasks graded by a model judge. | Output quality | Influencer search only, graded by a model: no system is changed and no business number is read. | Screen only |
| AIMultiple agentic marketing benchmark | Three workflows built on its own pages. Its authors publish one score per model and say no repeated runs sit behind it. | Output quality | Three workflows on the authors' own pages, run once each: no system is changed and no spread between runs is reported. | Not evidence |
None of the sixteen measures business effect. Vending-Bench comes closest, and its dollars are simulation money. The two execution benchmarks built on business systems, AutomationBench and tau-bench, both check the system state after the run against a fixed expected outcome. Copy that grader design. The scores themselves come from tasks outside marketing.
How three public skill libraries group marketing work
Three public libraries show how skill authors group marketing work. The counts are folder counts from the GitHub API (application programming interface) on 20 October 2026.
| Library and skill count | How it groups them | Our run on 22 October 2026 |
|---|---|---|
| marketingskills, 52 skills | Nine categories: Conversion Optimization, Content & Copy, SEO & Discovery, Paid & Distribution, Measurement & Testing, Retention, Growth Engineering, Strategy & Monetization, Sales & RevOps | 9 of 10 copywriting tests passed on Sonnet 5.5 in one run on 22 October 2026, 68 of 69 assertions, a self-check since Sonnet 5.5 graded its own output. The miss wanted facts the brief never gave, so it answered with fill-in placeholders. All 52 skills ship an evals file, and the 69 assertions we ran check structure and wording only. |
| Anthropic marketing plugin, 8 skills | One folder per slash command: draft-content, campaign-plan, brand-review, competitive-brief, performance-report, seo-audit, email-sequence, plus content-creation | brand-review: 10 findings, 6 pinned to a sentence. seo-audit: 55 items, 22 pinned to the page. brand-review ran with no voice guide and no link access. seo-audit's rankings, backlinks and speed items came from the page text alone, since no rank tracker was connected. |
| anthropics/skills, 19 skills | Four sets: Creative & Design, Development & Technical, Enterprise & Communication, Document Skills. | A Word file in 46 seconds, 5 tables, 116 bullets and 40 links kept. It keeps the content and drops the behaviour: the 8 collapsible question cards flattened into headings. |
| Skill | Question type | Why |
|---|---|---|
| Performance report | Describe | It states what happened. |
| Competitive brief | Describe, with an Explain inside it | It states what rivals do and why it matters. |
| Campaign plan | Prepare | It drafts something a person will approve. |
| Email sequence | Prepare | It drafts a sequence with send timing, for an approver to edit. |
| Content draft | Prepare | It drafts a post or a page, for an approver to edit. |
| Brand review | Choose | It scores a draft against a rubric. |
That covers four of the eight types: Describe, Explain, Prepare and Choose. Forecast, Execute, Monitor and Verify fall outside these libraries, since the last three need a live connection to a system. A skill can only tell the agent to use whatever connection its host provides.
A skill library tells you which deliverables an author has practised, and what the agent can do in your accounts shows only once you connect one. The one library with test prompts gives you prompts and expected outputs, which a person or a test setup you build has to run. The front matter of a skill holds a name, a description and an argument hint, and no score. Grade a skill by running it.
Six checks for any leaderboard score
- AutomationBench ranks models on a private, harder set. Zapier ships 600 public tasks. Its AutomationBench README says scores you measure locally may not match the official leaderboard 1:1.
- The grader can be code, people or a model. AutomationBench compares the final environment state with the expected state using assertion-based rubrics. GDPval used blinded human experts and its paper (October 2025) reports that its automated grader agreed with them 65.7% of the time, against 70.8% between two humans. Mark-Bench uses Claude and Gemini as judges. When a model did the grading, ask for its agreement rate with people. GDPval paper.
- One run hides reliability. Zapier reports run-to-run variance typically within 1%. That is the spread of the average. It says nothing about one task's reliability. The reliability number is pass^k, the share of tasks an agent passes every time in k tries. Sierra's tau-bench paper found that function-calling agents such as gpt-4o succeeded on fewer than half the tasks. Its blog (20 June 2024) puts the retail pass^8 score at about 25%, which the post calls a 60% drop from its pass^1 score. MCPMark prints both numbers for one model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. A weekly task runs 52 times a year. Its pass^52 is the share of tasks it passes on all 52 runs, and that share cannot be higher than pass^4 for the same tasks. Our weekly task has no pass^52 yet. MCPMark's published pass^4 comes from other tasks, so it does not give our number.
- Versions drift. The tau-bench repository, read on 20 October 2026, says its airline and retail tasks are outdated. BFCL is on V4, last updated 12 April 2026. Mark-Bench's own pages disagree with each other: the launch article counts 297+ tasks across 17 functions, the press release 285 tasks across 15. Quote the count from the page you link, with its date. A score without a version and a date cannot be compared with another score.
- A model row can hide a fallback model. Zapier's leaderboard, read on 20 October 2026, notes that some rows use default fallbacks: a step the model refuses is routed to the provider's own fallback model. Its footnote for the "Claude Fable 5.1 (with Opus 5 Fallback)" row says Opus 5 handled 260 of 657 tasks, about 40%. Those completions count toward the score. TheAgentCompany prints cost per task instance next to every score. Ask for both the fallback policy and the cost per completed task. Zapier leaderboard.
- Mark-Bench's test setup belongs to its vendor. Its cost claim compares Optimizely's own test setup with a raw model. A vendor's setup can also go away: OpenAI's deprecations page, read on 21 October 2026, says Agent Builder shuts down on 30 November 2026. Keep your own 12 questions and your own log. OpenAI deprecations.
How to use the map
-
As a user
Choose a question and fill in the context. Start with reading and preparation.
-
As a product owner
Add recurring customer questions. Keep question type and subject as separate fields.
-
As an engineer
Connect each question to sources and acceptance checks. A connected tool proves reach. Readiness needs each question proven end to end.
-
Evaluating an agent
Test reasoning on fixed data, then evaluate data collection through MCP separately.
A 12-question scorecard you can run this quarter
Pick five questions from the cards above and run each one twice as a pilot. The person who scores an answer must not be the person who ran the agent. A question that passes both runs goes into the full scorecard below.
Four readiness levels label how far an agent is proven on a question.
| Level | What it means | How you get there |
|---|---|---|
| Described | The vendor says the agent can do it. | Read the docs. Nothing else. |
| Historically checked | It answered correctly on past data with known answers. | Run it on last quarter's questions. Compare to what actually happened. |
| Checked now | It answered correctly on live data this week. | Run the same questions on current data. Verify against the source system. |
| Works repeatedly | It has passed the same check several times, including after a data change. | Schedule the check. Keep a log. |
| Block | Question types |
|---|---|
| Four reads | One Describe, one Explain, one Forecast, one Choose |
| Four that draft or change | Two Prepare, two Execute |
| Four that repeat a check | Two Monitor, two Verify |
We ran one of the twelve questions, Q04, through data access and output quality.
- Pick 12 questions from the three blocks in the table above. Spread them over the ad accounts, the CRM, the site and the warehouse.
- Write the known answer for each Describe question before you run the agent, pulled from the source system by hand. For Forecast, ask about a past month, give the agent only the data up to that month's start, and score against what happened. For Explain, pick a change whose cause is on record, such as a budget cut or a tracking outage logged at the time.
- Run each question five times. The spread across those runs is the reliability read, as the pass^k check above explains.
- Score reads on correctness first, readability second. Score Execute on system state after the run, and give the agent's summary no weight.
- For the one business number you care about, set a holdout or another control. Read it once the outcome has had time to land and the sample is large enough. A two-sided read for a 30% change at 95% confidence with 80% power needs about 200 leads per arm. That assumes leads arrive at random, so the count sets its own noise and no baseline rate is needed. The arithmetic: 1.96 plus 0.84, squared, times 2.3 (the two arms' rates, 1 and 1.3, added), divided by 0.3 squared, gives 200. At 50 leads a week per arm that is four weeks. A 20% change needs about 430 per arm on the same assumptions, nine weeks at that volume. At 15 leads a week the same reads take 13 and 29 weeks. The control is what makes it an effect test.
- Label each question with its readiness level. Ship only what reaches "checked now" for reads and "works repeatedly" for writes. Until a write reaches "works repeatedly", it stays behind a human approval.
Frequently asked questions
Is a high benchmark score enough to let an agent change campaigns?
A benchmark score covers one layer at most, output quality or execution, and your business number sits outside it. Letting an agent change campaigns needs the execution test on your own accounts. Run the change in a paused or sandbox state, read the system back, and confirm nothing else moved. Repeat it before you remove the human approval step.
How many questions do we need to test an agent properly?
Twelve covers the eight types in three blocks: four reads, four that draft or change, four that repeat a check. Reliability comes from the runs, so run each five times. Report the lowest and highest score across the five runs next to the average.
What is the difference between a skill and an MCP server?
A skill is a set of instructions and files that tells the agent how to do one kind of work, such as a brand review. An MCP server is a connection that gives the agent data or actions from a system. A skill tested on fixed inputs proves the instructions. The agent needs its own run on your data, through a connection or an export. Our list of MCP servers for marketing data compares the connections.
Should we trust the agent's own report that it finished a task?
Trust it after a read-back confirms the system state, through the same connection the agent used or a second one. tau-bench compares the database state after each task with the expected outcome.
How long does the business effect test take?
Weeks, and the count depends on weekly volume and the lift you need. Pick one number and set a holdout or another control. A 30% change at 95% confidence and 80% power needs about 200 leads per arm. At 50 leads a week, that is four weeks. A 20% change at the same confidence and power needs about 430 per arm, nine weeks at that volume.
Run this test on your own stack
Book a demo and see these questions answered from your own data, with sources, freshness and readback visible.
Book a demo