Independent AI evaluation · Capital markets

We evaluate AI for capital markets.

AI Alpha Labs tests frontier models on real trade-operations workflows — scored by a deterministic engine, and published with its methodology and known limitations.

See the benchmark results Why independent evaluation

2,000

Benchmark cases across AAL-D-001 through D-008

Frontier models on AAL-D-007 (v1.2 runs), from seven labs

0.0%

Run-to-run detection flip on AAL-D-007 (v1.2 runs) — no model contradicted itself across 250 cases, three runs each

49.8–74.5%

Category attribution on AAL-D-007 (v1.2 runs) — detection is solved; naming the cause is not

AAL-D-001 Trade Confirmation Exception Identification · 250 cases · 7 asset classesAAL-D-002 Margin Call Dispute Detection · 250 cases · 17 dispute categoriesAAL-D-003 Equity-Options Confirmation Exceptions · 250 cases · 19 categories · 6,000 observationsAAL-D-004 Settlement Fail Root Cause · 250 cases · false-positive rate 15.2% to 49.6% separates the fieldAAL-D-005 P&L Reconciliation Break Attribution · 250 cases · results pending a results fileAAL-D-006 Collateral Eligibility & Substitution · 250 cases · 9 models · value accuracy 15% to 58%AAL-D-007 SIMM/UMR Initial Margin Dispute Detection · 250 cases · 9 models · attribution 49.8% to 74.5% (v1.2 runs)AAL-D-008 Repo & Securities Lending Rate Disputes · 250 cases · 99% or 5% depending on the hostAAL-RS-007 GPT-4o Margin Call Disputes · 99.9% detection · 76.1% amount accuracyAPI v3.1 Live · validate · validate-raw · batch · disputes/analyzeMARKITWIRE Deterministic FIXML parsing · all 6 asset classes · no LLM requiredBATCH Up to 500 confirmation pairs · concurrentTHE AI ALPHA BRIEF Weekly newsletter · aialphalabs.beehiiv.comEVIDENCE Beta, Provisional, Confirmed, PublishedAAL-D-001 Trade Confirmation Exception Identification · 250 cases · 7 asset classesAAL-D-002 Margin Call Dispute Detection · 250 cases · 17 dispute categoriesAAL-D-003 Equity-Options Confirmation Exceptions · 250 cases · 19 categories · 6,000 observationsAAL-D-004 Settlement Fail Root Cause · 250 cases · false-positive rate 15.2% to 49.6% separates the fieldAAL-D-005 P&L Reconciliation Break Attribution · 250 cases · results pending a results fileAAL-D-006 Collateral Eligibility & Substitution · 250 cases · 9 models · value accuracy 15% to 58%AAL-D-007 SIMM/UMR Initial Margin Dispute Detection · 250 cases · 9 models · attribution 49.8% to 74.5% (v1.2 runs)AAL-D-008 Repo & Securities Lending Rate Disputes · 250 cases · 99% or 5% depending on the hostAAL-RS-007 GPT-4o Margin Call Disputes · 99.9% detection · 76.1% amount accuracyAPI v3.1 Live · validate · validate-raw · batch · disputes/analyzeMARKITWIRE Deterministic FIXML parsing · all 6 asset classes · no LLM requiredBATCH Up to 500 confirmation pairs · concurrentTHE AI ALPHA BRIEF Weekly newsletter · aialphalabs.beehiiv.comEVIDENCE Beta, Provisional, Confirmed, Published

Why AI Alpha Labs

Not a model vendor. An independent evaluator.

Independent

We don't build the models we score. Certification is a fixed fee, paid whatever the result, and every result ships with the data and code to check it.

Reproducible

Every benchmark ships with its dataset, prompts, and a deterministic scorer. The same answer always gets the same score.

Transparent

We publish confidence intervals, variance, and the cases models fail — not just a headline accuracy figure.

Financial-services focused

Benchmarks built on real trade-operations workflows — confirmations, exceptions, settlement — not academic tasks.

Products

Two ways to work with AAL.

APILive

Discrepancy Detection API

Integrate real-time confirmation validation into your ops stack. Six asset classes. Pay-as-you-go or monthly.

v3.14 endpointsGet API key →

AAL CERTIFIEDAudit

Independent AI Audit

Third-party validation for AI systems in capital markets. Deterministic scoring, published methodology, risk-committee-ready reports. For vendors and buy-side firms.

3 tiers2,000 casesGet certified →

Want to see the API before you integrate it? Try the live demo →

Latest Research

A growing body of evidence.

All research →Series Roadmap →

AAL-D-008Published

Repo and Securities Lending Rate Disputes

Frontier models scored 100% whether the computed figures were printed or withheld. The same model scored 99% or 5% on identical cases, depending on which host served it.

v0.1 · Sep 2026Explore →

AAL-D-007Published

SIMM / UMR Initial Margin Dispute Detection

Nine frontier models, three runs each. Detection is 100% and stable on every model; naming the cause of a margin dispute runs from 49.8% to 74.5% (v1.2 runs).

9 models · Sep 2026Explore →

AAL-D-006Published

Collateral Eligibility & Substitution

250 cases across 12 eligibility categories on nine frontier models. Detection is solved; no model values the ineligible exposure above 58%, and escalation splits the field from 5.6% to 98.8%.

9 models · Aug 2026Explore →

AAL-D-005Published

P&L Reconciliation Break Attribution

162 breaks across 15 root-cause categories. Model results pending a results file (2026-10-12).

Aug 2026Explore →

AAL-D-004Published

Settlement Fail Root Cause Identification

250 settlement cases, 88 of them clean traps. Every model catches every real exception; the false-positive rate, from 15.2% to 49.6%, is what separates the field.

4 models · 3,000 obs · Jul 2026Explore →

AAL-RS-010Published

The Reproducibility Crisis in Capital Markets AI Benchmarking

Most vendor AI accuracy claims cannot be reproduced. This note sets out why, and ends with a ten-question scorecard you can send any AI vendor.

Publication · Jul 2026Read →

The AAL Operating System

How evidence is built — end to end.

Benchmark Engineering

Construct validated cases across asset classes with ground truth and scoring criteria.

Open →02

Evaluation

Run models blind to ground truth; grade with a deterministic engine, not model judgment.

Open →

QA

Cases are built by the founder with AI assistance; there is no independent review panel. Answers are fixed in code before any model runs, and automated checks, including a prompt-leakage scan, run on every dataset.

Evidence

Findings advance Beta → Provisional → Confirmed → Published as evidence accumulates.

Publications

Results released with methodology and known limitations. Each dataset page states what can be reproduced.

Open →

The AI Alpha Brief

Benchmark findings, documented model failures and practical implications for capital-markets teams — delivered weekly.

Subscribe to the Brief →

AAL Benchmark Series · charts cover AAL-D-001 through D-007 (D-005 results pending a results file) · corpus 2,000 cases across eight datasets

Detection is solved. Reliability is not.

Each chart shows two bars per model. Navy = detection accuracy — did the model correctly identify whether a problem exists? Gold = the discriminating metric — calculation accuracy (D-001–003), trap-clean accuracy (D-004), value accuracy (D-006), or category attribution accuracy (D-007). The gap between them is the finding.

AAL-D-001 · Trade confirmation exception identification

Can the model catch a discrepancy between a counterparty confirmation and an internal record — and quantify the exposure?

Navy = detectionGold = exposure (re-scored)

Gold bars: exposure accuracy re-scored 2026-10-12 on the corrected answer key. Gemini 2.5 Pro is its v1.0-prompt run (same run as its detection score); for the other three, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases.

AAL-D-002 · Margin call dispute detection

Can the model identify a disputed margin call — and compute the correct call amount after applying threshold, haircut, and MTA rules?

Navy = detectionGold = Amount acc.

Claude Haiku removed (2026-10-12): no results file exists for its D-002 run, so its figures have been taken off the chart and the table. Gemini 2.5 Pro removed (2026-10-12) for the same reason: no results file exists for its D-002 run.

AAL-D-003 · Equity-options confirmation exceptions · v1.1 prompt

Can the model detect an exception in an equity options confirmation — and calculate the correct exposure, including greeks and multi-leg structures?

Navy = detectionGold = Exposure acc.

Gold bars (exposure accuracy): Under review: answer-key currency error found (D-003). Values unchanged.

AAL-D-004 · Settlement fail root cause identification

Can the model catch a real settlement failure — without crying wolf on trap cleans (settlements that look wrong but aren't)? Gold bar = trap-clean accuracy (100 − false-positive rate).

Navy = detectionGold = 100 − false-positive rate (trap-clean accuracy)

AAL-D-006 · Collateral eligibility & substitution

Can the model decide whether collateral is eligible, value the ineligible exposure, and propose a valid substitute? Gold bar = value accuracy, on the seven newest frontier models. No model clears 58% on the number — and no single model wins across the board: Gemini leads value, Opus 5 escalation (98.8%), Grok 4.5 substitution (98.2%), yet each hits the same wall.

Navy = detectionGold = Value acc.

AAL-D-007 (v1.2 runs) · SIMM / UMR initial margin dispute detection

Can the model tell a real margin dispute from a within-tolerance trap — and name the root-cause category? Gold bar = category attribution accuracy, nine models, 486 disputed observations each. Detection is 100.0% on every model, with a 0.0% false-flag rate.

Navy = detectionGold = Category acc.

The finding: Frontier models now detect exceptions at or near 100%. What happens after detection is less reliable. On D-007 (v1.2 runs), detection was 100% on every model, yet naming the cause of a margin dispute ran 49.8% to 74.5%. On D-008, frontier models with reasoning on computed every figure correctly, but the same model scored 5% on identical cases when its host switched reasoning off. The result depends on the model, its configuration and how fully the task is specified. This is why the AAL API routes classification to a language model and all arithmetic to a deterministic Python engine.

3 runs per case, Wilson 95% CI. Sampling varies by model and is recorded per run in each published scorecard — earlier datasets ran at temperature 0, while reasoning models use vendor-recommended settings and several newer endpoints reject a custom temperature outright. Full methodology at aialphalabs.ai/research

Full resultsAll research →

Show model-by-model results, AAL-D-001 to D-007

AAL-D-001 · Trade Confirmation Exception IdentificationDataset v1.0 · published Jun 2026250 cases · 7 asset classes →

#	Model	Detection	Exposure acc.	False pos.	Status
01	Gemini 2.5 Pro	99.6	80.7%	0.0%	Corrected 2026-10-12
02	GPT-4ov1.2 prompt	99.2	66.1%	0.0%	Corrected 2026-10-12
03	Gemini 2.5 Flash	99.5	79.5%	0.0%	Corrected 2026-10-12
04	Claude Sonnet 4.6	98.8	64.3%	0.0%	Corrected 2026-10-12

Exposure re-scored 2026-10-12 on the corrected answer key (11 FX-forward cases had recorded foreign-currency amounts as USD). Gemini 2.5 Pro: v1.0-prompt run. GPT-4o, Flash and Sonnet: the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases.

AAL-D-002 · Margin Call Dispute DetectionDataset v1.0.1 · published Jul 2026250 cases · 17 dispute categories →

#	Model	Detection	Amount acc.	False pos.	Status
01	GPT-4ogpt-4o-2024-08-06	99.9%	76.1%	0.0%	Published

AAL-D-003 · Equity-Options Confirmation ExceptionsDataset v1.0.2 · published Jul 2026250 cases · 19 categories · 6,000 obs →

#	Model	Detection	Exposure acc. v1.1	Category acc. v1.1	Status
01	Claude Sonnet 5claude-sonnet-5	100%	62.8% Under review: answer-key currency error found (D-003)	94.4%	Published
02	Claude Sonnet 4.6claude-sonnet-4-6	100%	63.0% Under review: answer-key currency error found (D-003)	—	Published
03	Gemini 2.5 Progemini-2.5-pro	100%	59.9% Under review: answer-key currency error found (D-003)	—	Published
04	GPT-4ogpt-4o-2024-08-06	100%	39.1% Under review: answer-key currency error found (D-003)	—	Published

AAL-D-004 · Settlement Fail Root Cause IdentificationDataset v1.0 · published Jul 2026250 cases · 88 trap cleans · 3,000 obs →

#	Model	Detection	False-pos. rate	Category acc.	Status
01	Claude Sonnet 5thinking · claude-sonnet-5	100%	15.2%	—	Published
02	Claude Sonnet 4.6claude-sonnet-4-6	100%	38.6%	86.1%	Published
03	GPT-4ogpt-4o-2024-08-06	100%	38.6%	—	Published
04	Gemini 2.5 Progemini-2.5-pro	100%	49.6%	—	Published

AAL-D-005 · P&L Reconciliation Break AttributionDataset v1.0 · published Aug 2026250 cases · 162 breaks / 88 traps →

Results pending a results file (2026-10-12): no D-005 results file has been found, so the D-005 model figures have been removed until it is located.

AAL-D-006 · Collateral Eligibility & SubstitutionDataset v1.1 · published Aug 2026250 cases · 162 ineligible / 88 traps · 9 models →

#	Model	Detection	Value acc.	Escalation	Status
01	Gemini 3.1 Progemini-3.1-pro-preview	100%	57.8%	86.0%	Published
02	Qwen 3.8-Maxdashscope · qwen3.8-max	100%	39.2%	51.9%	Published
03	GPT-5.6 Solgpt-5.6-sol	100%	38.3%	79.8%	Published
04	Claude Sonnet 5thinking · claude-sonnet-5	99.7%	35.5%	98.4%	Published
05	Kimi K3thinking · kimi-k3	99.9%	31.6%	69.8%	Published
06	Muse Spark 1.2meta/muse-spark-1.2	100%	31.1%	83.5%	Published
07	DeepSeek V4 Prothinking · deepseek-v4-pro	93.1%	24.0%	5.6%	Published
08	Claude Opus 5thinking · claude-opus-5	95.7%	22.5%	98.8%	Published
09	Grok 4.5x-ai/grok-4.5	100%	15.4%	54.1%	Published

AAL-D-007 · SIMM / UMR Initial Margin Dispute Detection · v1.2 runsDataset v1.2 · published Sep 2026250 cases · 162 disputes / 88 traps · 9 models →

#	Model	Detection	Category	Value acc.	Value flip	Status
01	Claude Opus 5thinking · claude-opus-5	100%	74.5%	87.0%	3.7%	Published
02	Gemini 3.1 Progemini-3.1-pro-preview	100%	65.0%	82.5%	4.3%	Published
03	Muse Spark 1.2meta/muse-spark-1.2	100%	64.6%	83.3%	16.7%	Published
04	Claude Sonnet 5thinking · claude-sonnet-5	100%	63.2%	76.5%	21.6%	Published
05	Grok 4.5x-ai/grok-4.5	100%	62.3%	79.4%	16.0%	Published
06	GPT-5.6 Solgpt-5.6-sol	100%	61.5%	85.8%	7.4%	Published
07	Kimi K3thinking · kimi-k3	100%	59.1%	85.2%	9.3%	Published
08	GPT-6 Astragpt-6-astra	100%	58.4%	68.9%	3.1%	Published
09	DeepSeek V4 Prothinking · deepseek-v4-pro	100%	49.8%	67.7%	37.0%	Published

v1.2 runs. Detection is 100.0% on every model with a 0.0% false-flag rate across all 88 traps — and a 0.0% run-to-run flip rate, meaning no model ever contradicted itself about whether a dispute exists. Value flipis the share of disputed cases where a model's amount changes between runs. DeepSeek V4 Pro's 67.7% value accuracy sits on a 37.0% flip rate, part of it caused by host routing rather than the model, as disclosed on the D-007 page: an average over a partly random process, not the profile of a model wrong a third of the time. A benchmark run once cannot tell the two apart. Every model has all 750 runs scored. Runs that failed at first publication were repeated on 8 Oct 2026; the repeated runs for four OpenRouter models requested reasoning explicitly, as disclosed on the D-007 page.

All evaluations use deterministic scoring engines with per-case tolerances. Where shown, 95% intervals are Wilson intervals computed by case (the 3 runs of each case grouped). Methodology and scoring rules are published with each dataset; prompts, drivers and raw results are in the public repository for AAL-D-006 to AAL-D-008.

Evaluation principle

We don't benchmark models. We benchmark models on the work your desk actually does.

General leaderboards measure academic capability. AAL benchmarks measure the operations work your desk actually does, from trade confirmations to repo disputes.

Custom benchmarks and private briefings for institutional operations and risk teams.

Contact for access