Make Agents Evaluatable, Controllable, and Iterative with Business-Outcome Evaluation
Many Agent evaluations still ask whether the answer resembles a correct answer. That can work for QA, summaries, or code with a clear test result, but not for Agents that generate marketing, advertising, subsidy, or operations strategies.
Strategy Agents produce configurations that may enter real business execution. Quality must cover policy compliance, evidence, reasoning, decision quality, tool execution, and incremental business effect. An offline answer that looks reasonable does not prove higher ROI, GMV, or conversion online.
This article is adapted from a public Alibaba business-technology article. Its teams, experimental numbers, and conclusions remain source claims, not built-in GPT88 capabilities.
Five Layers, Three Methods, Four Steps, Three Stages
| Question | Method | Purpose |
|---|---|---|
| What to evaluate | Five evaluation layers | Split Prompt constraints through business effect |
| How to evaluate | Rules, models, experiments | Match method to determinism |
| What to do afterward | Four-step attribution | Move from anomaly to fix and regression |
| How to release | Gate, canary, monitoring | Control rollout and feed data back |
business goal
→ dataset and evaluator
→ layered diagnosis
→ root cause
→ Prompt / knowledge / tool / algorithm change
→ offline regression
→ canary experiment
→ monitoring and feedback
The principles are evaluatability, control, and iteration: know which layer failed, add release and online safeguards, and route every finding to a change that can be checked again.
Why Strategy Agents Are Hard to Evaluate
Online A/B feedback is scarce and delayed. Good strategies trade off objectives such as ROI, GMV, scale, and budget safety. Absolute online metrics are also confounded by traffic, inventory, seasonality, and cold-start items.
Use incremental comparison rather than attributing every change directly to the Agent:
strategy lift = treatment-bucket result - control-bucket result
Offline evaluation helps screen and rank candidates, but it cannot replace a controlled business experiment.
Five Evaluation Objects
L1: Prompt and constraints
Check role, input/output protocol, priorities, forbidden actions, budget limits, and exception handling. Conflicting instructions and unconstrained schemas are common failures.
L2: Knowledge and context
Check product data, historical strategies, experiment buckets, operating rules, and time windows. Correct reasoning over the wrong item, category, or stale rule is still wrong.
L3: Tools and execution
Check parsing, parameters, permissions, retries, and write-back. Deterministic gates can enforce:
subsidy_rate ∈ [min_rate, max_rate]
spend ≤ budget_limit
required fields are present
strategy version matches experiment bucket
L4: Reasoning and strategy
Check evidence use, constraint understanding, explanation of trade-offs, and priority alignment. LLM Judges need Rubrics, gold labels, randomized order, consistency checks, and Bad Case monitoring.
L5: Business effect and regression
Check lift against a control, experiment stability, risk, and long-term regression. Only real experiments and online data can validate this layer.
The layers cascade: unstable L1 distorts every later layer; wrong L2 makes correct L3 execution wrong; passing L4 with failed L5 may indicate Judge misalignment or experimental noise.
Three Evaluation Methods
- Rules: schema, ranges, budget, required fields, permissions, and tool success. Rules are cheap, deterministic, and auditable.
- Models: semantic citation, strategy direction, trade-offs, and Rubric compliance. Judge scores are evaluator-relative, not business profit.
- Experiments: treatment, control, and stability buckets. Include sample size, confidence intervals, duration, bucket consistency, and external events.
Build Datasets Around Failures
Maintain normal-flow, regression, and Bad Case sets. Add online feedback, adversarial boundaries, red-team attempts to bypass budgets or permissions, safety cases, and an independent Judge-validation set.
Let risk determine sampling. Refresh data when business goals, Prompts, context, or strategy distribution changes. Evaluate the evaluator itself; a Judge that performs well on its construction set may degrade sharply on a held-out set.
Four-Step Root-Cause Attribution
1. Layered diagnosis
Track format compliance, factual accuracy, reasoning, decision quality, tool execution, multi-turn stability, and business effect separately.
2. Root cause
A low reasoning score may come from Prompt conflict, retrieval, missing tool data, or Judge overfitting. Keep asking why instead of blaming the model by default.
3. Route the fix
Prompt ambiguity belongs to product and operations; context errors to knowledge and retrieval; tool failures to schema, permissions, and retries; reasoning bias to examples, model, Prompt, or algorithm; Judge drift to Rubric and gold data; online anomalies to experiment conditions and external events.
4. Regression
Run the right regression set after every Prompt, data, algorithm, knowledge, or tool change. Pair offline results with experiments and check retrieval and business outcomes.
Three Release Stages
- Admission: hard rules, offline datasets, Judge validation, risk limits, and rollback readiness.
- Canary: small traffic, treatment/control comparison, manual review, and automatic circuit breaking.
- Monitoring: online lift, risk, drift, evaluator disagreement, bad cases, and feedback into versioned regression sets.
Summary
Business-outcome evaluation prevents a strategy Agent from being declared good because its prose sounds plausible. Separate constraints, context, tools, reasoning, and business effect; use rules, models, and experiments for the questions each can answer; then connect every failure to an owner and a regression.