Marketing AI evaluation guide
How to evaluate a marketing AI agent
Eight question types, six subject areas and twenty-four ready-to-run requests. See where MCP supplies data and actions, and how to verify every result.
The four-layer test
Every tool shows a demo. A demo proves at most one of these layers.
- Data access Can it reach your systems, with definitions and freshness checked first?
- Output quality Does the answer hold? Totals reconcile, sources and limits stay visible.
- Action execution Does a change land on the intended object, confirmed by readback?
- Business effect Does it move a metric you measure? Only experiments prove lift.
The twenty-four questions below test all four.
What each layer proves, and what it misses
An agent can write a confident, well-structured answer about last month's cost per acquisition (CPA) that is wrong, because it read the wrong column. A rubric or a judge model grades the writing. It catches the wrong column only when it can see the source data or the right answer, so the output test needs one of the two.
A perfect budget recommendation can land on the wrong campaign. Execution tests catch that only when someone checks the system state after the run. The agent's own report of what it did proves nothing.
| Layer | Question it answers | Evidence it needs | What it cannot see |
|---|---|---|---|
| Data access | What can it read and change, and is the data fresh? | A read-back of the connection: sources, permissions and the last load time | Whether the answer built on that data is right |
| Output quality | Is the answer right, and is it readable? | A known answer, or a rubric and a judge | Whether the agent can act, or whether acting helps |
| Action execution | Did it change the right thing, and nothing else? | A check of the system state after the run | Whether the change was worth making |
| Business effect | Did the number move, against a control? | A holdout and weeks of data | Which step of the agent's work caused it |
Eight question types
Every question in the map asks one of these. Each type carries an acceptance check: what a correct answer must show before you trust it.
-
Describe
What happened?
Done when Reconciled facts, comparisons and composition.
-
Explain
Why did it happen?
Done when Tested explanations and explicit limits.
-
Forecast
What might happen next?
Done when Conditional projections and assumptions.
-
Choose
What should we do?
Done when Options, constraints and trade-offs.
-
Prepare
What should we create?
Done when A reviewable brief, draft or plan.
-
Execute
What should change in a system?
Done when A confirmed change to the intended object.
-
Monitor
When should we intervene?
Done when Repeated checks and a delivered signal.
-
Verify
Can we trust this?
Done when Reconciliation or final-state confirmation.
Subjects are a separate axis: paid media, funnel, creative, budget, measurement, and CRM and operations. Execution mode is separate too - read, prepare a draft, change a system or repeat a check. A question type does not grant permission to act.
Twenty-four questions, ready to run
Filter by question type and subject, or search. Every card carries its inputs, its acceptance check, the MCP role and one limit, plus a request you can copy into your own agent.
Showing all 24 questions
-
How much did we spend, and what did we get?
You get Spend and results for an agreed period.
MCP Reads connected ad platforms or the warehouse.
How to use and verify
Inputs Account, dates, currency and the definition of a result.
Done when Totals reconcile with details. Sources and freshness are visible.
One limit Different attribution windows need an explicit reconciliation.
How much did we spend, and what did we get? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Spend and results for an agreed period. Acceptance: Totals reconcile with details. Sources and freshness are visible. Limitation: Different attribution windows need an explicit reconciliation. For now, read and prepare only. Do not change external systems.
-
Which channels generated qualified meetings?
You get A channel comparison using one definition of a meeting.
MCP Retrieves CRM data and observed lead journeys.
How to use and verify
Inputs CRM, lead source, statuses and event history.
Done when Meetings are deduplicated. Channel assignment rules are visible.
One limit An attributed meeting does not prove incremental channel impact.
Which channels generated qualified meetings? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A channel comparison using one definition of a meeting. Acceptance: Meetings are deduplicated. Channel assignment rules are visible. Limitation: An attributed meeting does not prove incremental channel impact. For now, read and prepare only. Do not change external systems.
-
Which ad formats perform best?
You get A format comparison with the volume of observations.
MCP Retrieves assets and performance from available sources.
How to use and verify
Inputs Assets, format labels, spend and conversions.
Done when Sample sizes, segments and comparability are visible.
One limit Different audiences can explain differences in average performance.
Which ad formats perform best? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A format comparison with the volume of observations. Acceptance: Sample sizes, segments and comparability are visible. Limitation: Different audiences can explain differences in average performance. For now, read and prepare only. Do not change external systems.
-
Why did cost per lead increase?
You get A breakdown of the change and tested explanations.
MCP Retrieves metrics, settings and available history.
How to use and verify
Inputs Clicks, spend, leads, settings and change history.
Done when Every explanation has evidence. Unproven causes are labelled.
One limit Changes happening together do not establish causation.
Why did cost per lead increase? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A breakdown of the change and tested explanations. Acceptance: Every explanation has evidence. Unproven causes are labelled. Limitation: Changes happening together do not establish causation. For now, read and prepare only. Do not change external systems.
-
Where are potential customers dropping out?
You get A loss map between agreed funnel stages.
MCP Retrieves CRM history and measurement events.
How to use and verify
Inputs Transition events and comparable lead cohorts.
Done when Conversion delays are considered. Cohorts have comparable maturity.
One limit A missing event can be a data problem.
Where are potential customers dropping out? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A loss map between agreed funnel stages. Acceptance: Conversion delays are considered. Cohorts have comparable maturity. Limitation: A missing event can be a data problem. For now, read and prepare only. Do not change external systems.
-
Did performance fall, or are the data delayed?
You get A freshness check and location of the gap.
MCP Reads load status and available source records.
How to use and verify
Inputs Latest source records, load status and report records.
Done when Source, load and calculation use the same period.
One limit An empty table does not establish zero business activity.
Did performance fall, or are the data delayed? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A freshness check and location of the gap. Acceptance: Source, load and calculation use the same period. Limitation: An empty table does not establish zero business activity. For now, read and prepare only. Do not change external systems.
-
Will we exceed the budget by month end?
You get A spend forecast under a stated spending pace.
MCP Reads spend, constraints and current settings.
How to use and verify
Inputs Spend, budget limit, calendar and planned changes.
Done when Calculation date, assumptions and alternative scenarios are visible.
One limit Extending the current pace is a conditional projection.
Will we exceed the budget by month end? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A spend forecast under a stated spending pace. Acceptance: Calculation date, assumptions and alternative scenarios are visible. Limitation: Extending the current pace is a conditional projection. For now, read and prepare only. Do not change external systems.
-
How many meetings could we generate by period end?
You get Scenarios that account for volume and conversion delay.
MCP Retrieves historical events and the current queue.
How to use and verify
Inputs Inbound history and time to a qualified meeting.
Done when Known volume is separated from future assumptions.
One limit A different lead mix can invalidate the historical conversion rate.
How many meetings could we generate by period end? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Scenarios that account for volume and conversion delay. Acceptance: Known volume is separated from future assumptions. Limitation: A different lead mix can invalidate the historical conversion rate. For now, read and prepare only. Do not change external systems.
-
What would change if we increased the budget?
You get Conditional scenarios with explicit uncertainty.
MCP Retrieves spend, outcome and settings history.
How to use and verify
Inputs Constraints, past budget changes and any available response curve.
Done when No linear increase in leads is promised without response evidence.
One limit Average cost per lead is not marginal cost per additional lead.
What would change if we increased the budget? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Conditional scenarios with explicit uncertainty. Acceptance: No linear increase in leads is promised without response evidence. Limitation: Average cost per lead is not marginal cost per additional lead. For now, read and prepare only. Do not change external systems.
-
Where should we reallocate part of the budget?
You get Allocation options and the cost of each trade-off.
MCP Supplies performance and constraints for the calculation.
How to use and verify
Inputs Objective, limits, response evidence and learning constraints.
Done when Keeping the current allocation is an option. Advice is separate from action.
One limit Tool access does not authorize spending changes.
Where should we reallocate part of the budget? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Allocation options and the cost of each trade-off. Acceptance: Keeping the current allocation is an option. Advice is separate from action. Limitation: Tool access does not authorize spending changes. For now, read and prepare only. Do not change external systems.
-
Which hypothesis should we test next?
You get A ranked set of hypotheses and ways to evaluate them.
MCP Retrieves past tests, assets and results.
How to use and verify
Inputs Past results, weak points and available resources.
Done when Hypotheses are testable and success is defined in advance.
One limit An idea list alone does not establish expected lift.
Which hypothesis should we test next? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A ranked set of hypotheses and ways to evaluate them. Acceptance: Hypotheses are testable and success is defined in advance. Limitation: An idea list alone does not establish expected lift. For now, read and prepare only. Do not change external systems.
-
Which campaigns should we investigate first?
You get An investigation queue based on possible loss and confidence.
MCP Collects current performance across campaigns.
How to use and verify
Inputs Metric changes, spend volume and data freshness.
Done when Priority is explained. High spend is not treated as a cause.
One limit Investigation priority does not authorize automatic pausing.
Which campaigns should we investigate first? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: An investigation queue based on possible loss and confidence. Acceptance: Priority is explained. High spend is not treated as a cause. Limitation: Investigation priority does not authorize automatic pausing. For now, read and prepare only. Do not change external systems.
-
Prepare a brief for the next campaign.
You get A reviewable brief with an objective, audience, message and constraints.
MCP Retrieves context and assets available to the current user.
How to use and verify
Inputs Product context, brand rules, past assets and findings.
Done when Requirements cite sources. Missing information is labelled.
One limit The quality of a brief still needs subject-matter review.
Prepare a brief for the next campaign. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A reviewable brief with an objective, audience, message and constraints. Acceptance: Requirements cite sources. Missing information is labelled. Limitation: The quality of a brief still needs subject-matter review. For now, read and prepare only. Do not change external systems.
-
Write ad variants from the approved brief.
You get Drafts in the required format with supported claims.
MCP Retrieves the brief and supporting material.
How to use and verify
Inputs Approved brief, format limits and allowed evidence.
Done when No invented facts. The tone and format match the requirements.
One limit Good copy does not guarantee more sales.
Write ad variants from the approved brief. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Drafts in the required format with supported claims. Acceptance: No invented facts. The tone and format match the requirements. Limitation: Good copy does not guarantee more sales. For now, read and prepare only. Do not change external systems.
-
Prepare a campaign measurement plan.
You get Events, metric definitions and validation checks.
MCP Exposes available fields and sources.
How to use and verify
Inputs Objective, user journey, available systems and data.
Done when Every metric has a source, calculation and owner.
One limit The plan still needs implementation and verification on actual events.
Prepare a campaign measurement plan. Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Events, metric definitions and validation checks. Acceptance: Every metric has a source, calculation and owner. Limitation: The plan still needs implementation and verification on actual events. For now, read and prepare only. Do not change external systems.
-
Create the approved campaign in a paused state.
You get Created objects with confirmed settings.
MCP Submits supported requests and reads the resulting objects.
How to use and verify
Inputs Approved brief, account, permissions and budget constraints.
Done when Objects are read back. Delivery has not been activated.
One limit A prepared campaign and a live campaign are different outcomes.
Prepare a plan for this task: Create the approved campaign in a paused state. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Created objects with confirmed settings. Verification: Objects are read back. Delivery has not been activated. For now, read and prepare only. Do not change external systems.
-
Update the exclusion audience from CRM.
You get A change to the authorized audience membership only.
MCP Reads CRM and writes through a supported advertising API.
How to use and verify
Inputs Unambiguous selection rules and an agreed source.
Done when Included and excluded records are checked. No extra changes.
One limit Support and permissions are checked for the specific connection.
Prepare a plan for this task: Update the exclusion audience from CRM. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: A change to the authorized audience membership only. Verification: Included and excluded records are checked. No extra changes. For now, read and prepare only. Do not change external systems.
-
Update verified contact fields.
You get Confirmed changes to the intended records.
MCP Reads records, makes authorized updates and checks them.
How to use and verify
Inputs Exact contact key, value provenance and allowed fields.
Done when The right records changed and unrelated records did not.
One limit Identify a contact by its exact key, an email or a CRM id, and treat a name match as a candidate.
Prepare a plan for this task: Update verified contact fields. Context: [account or object], [objective], [constraints]. Show the exact objects, fields and proposed changes. After explicit authorization, the intended result is: Confirmed changes to the intended records. Verification: The right records changed and unrelated records did not. For now, read and prepare only. Do not change external systems.
-
Alert me when spend deviates from plan.
You get A repeated check with a clear alert.
MCP Supplies data. A workflow handles scheduling and delivery.
How to use and verify
Inputs Plan, schedule, threshold, recipient and expiry condition.
Done when Alerts are delivered, duplicates controlled and check failures visible.
One limit Monitoring counts as working after a deliberately failed check produces a visible alert.
Prepare monitoring for this request: Alert me when spend deviates from plan. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Alerts are delivered, duplicates controlled and check failures visible. For now, read and prepare only. Do not create schedules or send messages.
-
Warn me when data arrive late.
You get An alert when agreed freshness is violated.
MCP Reads status and timestamps of the latest data.
How to use and verify
Inputs Expected update frequency and acceptable delay.
Done when Source delay, load delay and report delay are distinguished.
One limit Silence proves nothing. Confirm monitoring with a test alert.
Prepare monitoring for this request: Warn me when data arrive late. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Source delay, load delay and report delay are distinguished. For now, read and prepare only. Do not create schedules or send messages.
-
Flag signs of ad fatigue.
You get Candidates for review or asset refresh.
MCP Retrieves ad history and results.
How to use and verify
Inputs History of frequency, clicks, conversion and delivery conditions.
Done when Periods are comparable. A signal is separated from a proven cause.
One limit Read a click-through drop together with frequency and conversion rate before calling it fatigue.
Prepare monitoring for this request: Flag signs of ad fatigue. Context: [account], [threshold], [recipient], [expiry]. Show sources, rules, schedule and failure handling. Verification after activation: Periods are comparable. A signal is separated from a proven cause. For now, read and prepare only. Do not create schedules or send messages.
-
Why does the report disagree with the ad platform?
You get A staged reconciliation and any unexplained remainder.
MCP Reads the source system and the final table.
How to use and verify
Inputs One account, period, currency and attribution definition.
Done when The discrepancy is quantified and localized.
One limit Different metric definitions are not automatically a load error.
Why does the report disagree with the ad platform? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A staged reconciliation and any unexplained remainder. Acceptance: The discrepancy is quantified and localized. Limitation: Different metric definitions are not automatically a load error. For now, read and prepare only. Do not change external systems.
-
Do we have enough evidence to name a test winner?
You get A review of design, sample and the permitted conclusion.
MCP Retrieves available events and test history.
How to use and verify
Inputs Assignment, metric, dates, stopping rules and data loss.
Done when Cohort comparability and uncertainty are assessed.
One limit A spend threshold is not a test of statistical significance.
Do we have enough evidence to name a test winner? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: A review of design, sample and the permitted conclusion. Acceptance: Cohort comparability and uncertainty are assessed. Limitation: A spend threshold is not a test of statistical significance. For now, read and prepare only. Do not change external systems.
-
Did the change actually persist?
You get Confirmation of the intended object state.
MCP Reads the final state through the relevant connection.
How to use and verify
Inputs Exact object, expected fields and operation response.
Done when There is a readback. An unknown result does not trigger a blind retry.
One limit A dispatched command is not a confirmed change.
Did the change actually persist? Context: [account or object], [period], [objective]. Check access, definitions and freshness first. Deliver: Confirmation of the intended object state. Acceptance: There is a readback. An unknown result does not trigger a blind retry. Limitation: A dispatched command is not a confirmed change. For now, read and prepare only. Do not change external systems.
No questions match. Clear the search or choose All.
Where MCP participates: it retrieves accessible facts and sends supported, authorized actions. The model selects the analytical approach. Data models, calculations and checks decide whether the conclusion is acceptable.
One question, end to end
Take one Explain question and walk it through all four layers. The question is one paid media teams ask an agent sooner or later: "Why did cost per lead increase?" It is the first Explain card above. We ran it on our own ad accounts on 19 October 2026, read-only, as a reference. The figures below come from that run and compare March with April 2026.
-
Data access
The agent needs the ad platforms, the site forms and the CRM through one governed connection, read-only for this question. Before it answers, it says what it will read and when each source was last loaded. In our run, spend was loaded through 19 October and the newest lead was from 18 October. One account had lost its connection in August. The agent said so instead of guessing at its numbers.
-
Output quality
A passing answer does four things. It agrees on definitions first: what counts as a lead, which period, which currency, which timezone. In our run, "how many leads did we get in March" had four honest answers, depending on which form and which attribution you count. The smallest was under a third of the largest. The agent had to name the one it used. It reconciles the numbers. In our run, one platform and the warehouse agreed to within two cents. The gap on another was traced to a second account whose connection had broken. Causes it could not test stay labelled as untested. The agent found three tested causes in our run. The biggest lead source had all four of its campaigns paused by 9 March, and two cheap mid-funnel campaigns were dropped on 11 and 13 March. One platform took half of all spend in both months and brought no leads at all. That mix explained a 3.2x rise in cost per lead. The decay in the one source left explained 1.2x more. Together that is 3.9x (the two factors are rounded, the total comes from the cost-per-lead figures themselves), and the report showed 3.9x. The two factors are rounded. The total comes from the cost-per-lead figures themselves. It changes nothing: the next step was written down with its verification and left for a person to approve.
-
Action execution
This question writes nothing, by design. Its card says read and prepare only. The agent proposed one action, pausing the empty campaign, with a way to verify it, and stopped. The write is tested on an Execute card, such as the first one above: create the approved campaign in a paused state. After the run you read the platform back through the API. The campaign exists, its status is paused, its budget is the agreed number, and the count of campaigns rose by exactly one. Then check that nothing else changed. If the agent reports success and the read-back disagrees, the read-back wins. Pass it five times on five fresh briefs, one of them after the account's data has changed, and the Execute type earns "works repeatedly".
-
Business effect
Nothing to measure yet. The effect test starts when a person acts on the answer, with a holdout or an incrementality test and enough weeks of data for the number to settle. That test measures the decision. The agent is measured by running the same question through your current process as the control and comparing the two. Platform-reported conversions and last-click attribution cannot show the effect on their own, because they move with attribution settings as much as with the market. An agent that answers this question correctly every time has passed three layers out of four. Its business effect is still unmeasured.
What the public benchmarks measure
| Benchmark | What it grades, and how | Test it covers | What it cannot show | Verdict |
|---|---|---|---|---|
| Mark-Bench (Optimizely, September 2026) | Marketing deliverables such as briefs, plans and copy. Each scenario has a brief, input files and a rubric of 12 to 30 criteria. Large language model (LLM) judges, Claude and Gemini, score the output. The launch article counts 297+ tasks and scenarios across 17 functions, with 3,564 criteria. | Output quality | Whether an agent can act. The judge is a model, and the cost figures compare the vendor's own harness, the software around the model, with a raw model. | Screen only |
| AD-Bench (Tencent, February 2026, revised June 2026) | 225 real marketing-analysis requests from a production advertising platform, in three difficulty levels. An LLM judge, Gemini-3.1-Pro, grades the answer and whether the agent's tool calls match an expert's. Best model Claude-Opus-4.7 at Pass@1 76.9%, falling to 61.4% on the hardest level. | Output quality on Describe and Explain questions | Any change to the platform. The judge is also one of the models under test, which the authors flag. | Screen only: the closest public test to analytics questions |
| AutomationBench (Zapier, April 2026) | Cross-app workflows in sales, marketing, operations, support, finance and HR. 600 public tasks, 100 per domain, plus a harder private set. Code compares the final system state with the expected state using assertion-based rubrics. A task passes only if every assertion passes. Top of the leaderboard (v1.0.6): Gemini 4 Argon (High) at 51.29%, as published (Zapier says run-to-run variance is typically within 1%). | Action execution | Deliverable quality. Your own tools and accounts. | Run your own: copy its grader design |
| GDPval (OpenAI, October 2025) | 1,320 one-shot work tasks across 44 occupations. The 220-task open subset is graded by blinded expert pairwise comparison. The automated grader agrees with human experts 65.7% of the time. Humans agree with each other 70.8%. | Output quality | Anything interactive. None of the 44 occupations is a marketing one. | Not evidence for marketing |
| tau-bench (Sierra, June 2024) | Customer service agents talking to a simulated user in retail and airline settings. Graded by comparing the database state after each task with the expected outcome. Introduced pass^k, the share of tasks an agent passes all k times. | Action execution, with a reliability measure | Marketing work. The original task sets are now marked outdated by their authors. | Screen only: borrow pass^k |
| BFCL (Berkeley Function Calling Leaderboard) V4, updated 12 April 2026 | Whether a model calls functions accurately. Two checks: the call's syntax tree against possible answers, and executable runs. Overall accuracy is the unweighted average of the sub-categories. | The first step of execution: the call goes out, the result stays unchecked | Whether the right thing changed in the system afterwards. | Screen only |
| MCP-Universe (August 2025) | 231 tasks run through 11 live MCP (Model Context Protocol) servers, the open-standard connections that give an agent tools and data. GPT-5 at 43.72% success, as published. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only: it shows what tool use reached on those tasks |
| MCP-Bench (August 2025) | 104 tasks across 28 live MCP servers and 250 tools. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only |
| MCPMark (September 2025) | 127 tasks on live MCP servers. Best model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. | Action execution through tools, with a reliability measure | Your servers and your permissions. Any business number. | Screen only: borrow pass^4 |
| LiveMCPBench (August 2025) | 95 tasks on 70 live MCP servers with 527 tools. | Action execution through tools | Your servers and your permissions. Any business number. | Screen only |
| CRMArena-Pro (Salesforce, May 2025) | Nineteen expert-validated tasks across sales, service and configure-price-quote processes, in single-turn and multi-turn settings. Leading agents drop to approximately 35% in multi-turn settings. Agents showed near-zero inherent confidentiality awareness. | Output quality on CRM reads and multi-turn conversations | Ad platforms. Your own records. | Run your own: copy the confidentiality test |
| TheAgentCompany (December 2024) | 175 workplace tasks, counted only when every checkpoint passes, with cost printed per task instance next to every score. | Execution on general office work | Marketing work. | Not evidence |
| OSWorld 2.0 | 108 long computer-use workflows. Claude Opus 4.8 at 20.6%. | Execution by computer use | Marketing work. | Not evidence |
| Vending-Bench (February 2025) | A simulated vending business scored on net worth, each experiment run five times. | A simulated business number | Marketing work. A real profit and loss. | Not evidence |
| xbench, marketing track (tested May 2025) | 50 influencer-search tasks graded by a model judge. | Output quality | Any system. Any business number. | Screen only |
| AIMultiple agentic marketing benchmark | Three workflows built on its own pages. Its authors say they do not report repeated runs, seeds, confidence intervals or error bars. | Output quality | Any system. Reliability across runs. | Not evidence |
None of the sixteen measures business effect. Vending-Bench comes closest, and its dollars are simulation money. The two execution benchmarks built on business systems, AutomationBench and tau-bench, both check the system state after the run against a fixed expected outcome. Copy the design, and leave the scores, which were earned on tasks outside marketing.
How three public skill libraries group marketing work
Three public libraries show how skill authors group marketing work. The counts are folder counts from the GitHub API (application programming interface) on 20 October 2026.
| Library and skill count | How it groups them | Our run on 22 October 2026 |
|---|---|---|
| marketingskills, 52 skills | Nine categories: Conversion Optimization, Content & Copy, SEO & Discovery, Paid & Distribution, Measurement & Testing, Retention, Growth Engineering, Strategy & Monetization, Sales & RevOps | 9 of 10 copywriting tests passed on Sonnet 5.5, 68 of 69 assertions, graded by the same model. The miss wanted facts the brief never gave, so it answered with fill-in placeholders. All 52 skills ship an evals file. None checks a fact, a link or a number. |
| Anthropic marketing plugin, 8 skills | One folder per slash command: draft-content, campaign-plan, brand-review, competitive-brief, performance-report, seo-audit, email-sequence, plus content-creation | brand-review: 10 findings, 6 pinned to a sentence, in 2 minutes. seo-audit: 55 items, 22 pinned to the page, in 35 seconds. brand-review wanted a voice guide first, seo-audit a rank tracker. We had neither. |
| anthropics/skills, 19 skills | Four sets: Creative & Design, Development & Technical, Enterprise & Communication, Document Skills. No marketing set. | A Word file in 46 seconds, 5 tables, 116 bullets and 40 links kept. The 8 collapsible question cards flattened into headings. |
Four runs, four different limits. The marketingskills tests pass with placeholders standing where the facts should be. brand-review needs a voice guide written first and cannot open a link. seo-audit leaves rankings, backlinks and speed to tools it was not connected to. docx keeps the content and drops the behaviour. A library tells you how it groups the work. What a skill does on your content, you learn by running it and counting.
| Skill | Question type | Why |
|---|---|---|
| Performance report | Describe | It states what happened. |
| Competitive brief | Describe, with an Explain inside it | It states what rivals do and why it matters. |
| Campaign plan | Prepare | It drafts something a person will approve. |
| Email sequence | Prepare | Same: a draft for approval. |
| Content draft | Prepare | Same: a draft for approval. |
| Brand review | Choose | It scores a draft against a rubric. |
That covers four of the eight types: Describe, Explain, Prepare and Choose. No skill here forecasts. Execute, Monitor and Verify need a live connection to a system, and no skill in these libraries carries one. A skill can only tell the agent to use whatever connection its host provides. The libraries sort skills by the marketer's job and ignore what each skill may change.
So a skill library tells you which deliverables an author has practiced. It says nothing about what the agent can do in your accounts. And a skill in these libraries ships with no proof that it works. Even the one library with test prompts gives you prompts and expected outputs, which a person or a harness you build has to run. The front matter of a skill holds a name, a description and an argument hint, and no score. Grade a skill by running it. Its file shows nothing about results.
How to read a leaderboard without fooling yourself
Six questions sort a useful score from a rumour. Each has a public example.
- Which task set? Zapier ships 600 public tasks but ranks models on a private, harder set. The AutomationBench README says scores you measure locally may not match the official leaderboard 1:1. A vendor quoting a public-set score is quoting a different test. AutomationBench README.
- Who graded? Code, people or a model. AutomationBench compares the final environment state with the expected state using assertion-based rubrics. GDPval used blinded human experts and its paper (October 2025) reports that its automated grader agreed with them 65.7% of the time, against 70.8% between two humans. Mark-Bench uses Claude and Gemini as judges. When a model did the grading, ask for its agreement rate with people. GDPval paper.
- How many runs? Zapier reports run-to-run variance typically within 1%. That is the spread of the average. It says nothing about one task's reliability. The reliability number is pass^k, the share of tasks an agent passes every time in k tries. Sierra's tau-bench paper found that function-calling agents such as gpt-4o succeeded on fewer than half the tasks, and fewer than a quarter on pass^8 in retail. Its blog (20 June 2024) puts the retail pass^8 score at about 25%, a 60% fall from the single-attempt score. MCPMark prints both numbers for one model, gpt-5-medium, as published: 52.56% pass@1, the share passed in one try, and 33.86% pass^4, the share passed all four times. For a weekly task, the second number is the one you will live with.
- Which version? The tau-bench repository now says its airline and retail tasks are outdated. BFCL is on V4, last updated 12 April 2026. Mark-Bench's own pages disagree with each other: the launch article counts 297+ tasks across 17 functions, the press release 285 tasks across 15. Quote the count from the page you link, with its date. A score without a version and a date cannot be compared with another score.
- What is in the model row? Zapier's leaderboard notes that some rows use default fallbacks: a step the model refuses is routed to the provider's own fallback model. Its footnote for the "Claude Fable 5.1 (with Opus 5 Fallback)" row says Opus 5 handled 260 of 657 tasks, about 40%. Those completions count toward the score. TheAgentCompany prints cost per task instance next to every score. Ask for both the fallback policy and the cost per completed task. Zapier leaderboard.
- Who owns the harness? Mark-Bench's cost claim compares Optimizely's own harness with a raw model. OpenAI's deprecations page, read on 21 October 2026, says Agent Builder shuts down on 30 November 2026. Its Evals platform goes read-only on 31 October 2026 and shuts down on 30 November 2026. A vendor's tooling can be withdrawn while your tests still depend on it. Keep your own 12 questions and your own log. OpenAI deprecations.
How to use the map
-
As a user
Choose a question and fill in the context. Start with reading and preparation.
-
As a product owner
Add recurring customer questions. Keep question type and subject as separate fields.
-
As an engineer
Connect each question to sources and acceptance checks. A connected tool proves reach. Readiness needs each question proven end to end.
-
Evaluating an agent
Test reasoning on fixed data, then evaluate data collection through MCP separately.
A 12-question scorecard you can run this quarter
Start small. Pick five questions from the cards above and run each one twice. Keep the judge apart from the doer: the person who scores an answer is never the person who ran the agent. A question that passes both runs goes into the full scorecard below.
Four readiness levels describe how far an agent has been proven on a given question. They are labels, and they carry no score.
| Level | What it means | How you get there |
|---|---|---|
| Described | The vendor says the agent can do it. | Read the docs. Nothing else. |
| Historically checked | It answered correctly on past data with known answers. | Run it on last quarter's questions. Compare to what actually happened. |
| Checked now | It answered correctly on live data this week. | Run the same questions on current data. Verify against the source system. |
| Works repeatedly | It has passed the same check several times, including after a data change. | Schedule the check. Keep a log. |
Run the scorecard like this.
- Pick 12 questions in three blocks of four. Four reads: one Describe, one Explain, one Forecast, one Choose. Four that draft or change: two Prepare, two Execute. Four that repeat: two Monitor, two Verify. Spread them over the six subjects.
- Write the known answer for each Describe question before you run the agent, pulled from the source system by hand. For Forecast, ask about a past month, give the agent only the data up to that month's start, and score against what happened. For Explain, pick a change whose cause is on record, such as a budget cut or a tracking outage logged at the time.
- Run each question five times. Agents vary between runs, and one pass proves nothing.
- Score reads on correctness first, readability second. Score Execute on system state after the run, and give the agent's summary no weight.
- For the one business number you care about, set a holdout or another control. Read it once the outcome has had time to land and the sample is large enough. Four weeks is a rule of thumb for paid media, with no source behind it. The control is what makes it an effect test.
- Label each question with its readiness level. Ship only what reaches "checked now" for reads and "works repeatedly" for writes. Until a write reaches "works repeatedly", it stays behind a human approval.
Frequently asked questions
Is a high benchmark score enough to let an agent change campaigns?
No. A benchmark score covers one test at most, output quality or execution. Your business number sits outside it. Letting an agent change campaigns needs the execution test on your own accounts. Run the change in a paused or sandbox state, read the system back, and confirm nothing else moved. Repeat it before you remove the human approval step.
How many questions do we need to test an agent properly?
Twelve is enough to start: four reads, four that draft or change, four that repeat a check, spread over the six subjects. Run each five times. The spread between runs tells you as much as the average.
What is the difference between a skill and an MCP server?
A skill is a set of instructions and files that tells the agent how to do one kind of work, such as a brand review. An MCP server is a connection that gives the agent data or actions from a system. A skill tested on fixed test inputs proves the instructions. It proves nothing about the agent on your data. For that you also need the data, through a connection or an export. Our list of MCP servers for marketing data compares the connections.
Should we trust the agent's own report that it finished a task?
Not on its own. The execution test reads the system state after the run, through the same connection the agent used or a second one. AutomationBench compares the final system state with the expected state using assertion-based rubrics. tau-bench compares the database state after each task with the expected outcome. An agent's own account of its work is not evidence.
How long does the business effect test take?
Weeks. Pick one number and set a holdout or another control. Read it once the outcome has had time to land and the sample is large enough to show the change you care about. Four weeks is a rule of thumb for paid media numbers, with no source behind it. The smaller the lift you need to see, the longer the read. A test with no control cannot tell the agent's effect from the market's.
Run this test on your own stack
Book a demo and see these questions answered from your own data, with sources, freshness and readback visible.
Book a demo