VerifiQuant

The problem

A number is not an answer

An LLM in a financial workflow fails most dangerously with a confident, well-formatted, wrong number, and nothing marks it as different from a correct one. Program-aided reasoning makes the arithmetic reliable. It does not make the answer legitimate: a program that runs returns a clean number even when the input cannot justify one.

Trap E-test-1080, from the paper's evaluation

“What is the Price Rate of Change (Price ROC) of the stock over the week? Provide your answer as a percentage to two decimal places.”

At the beginning of the week, the stock was priced at ~~$150~~$2.5 per share. By the end of the week, the stock's price changed to $165 per share.

One value was changed so that no valid number exists. The contract for this formula declares the check abs(current − previous) / previous > 2.0.

R-PAL6,500.00% Sees the same four retrieved contracts, writes a program, runs it. Silent wrong.

R-PAL⁺6,500.00% Same, with every check described in its prompt. Answers in 3 of 3 runs.

VerifiQuantStopped at E Executes the contract's declared check price_coherence_extreme_change. Stops in 3 of 3 runs.

Replay this trap and 79 more →

Abstract

Governability, not accuracy

Large language models embedded in financial workflows fail most dangerously by emitting confident, well-formatted but wrong numbers with no signal distinguishing them from correct ones. We argue that the central deployment problem is governability: a system must determine when a question supports a numeric answer and when it should instead clarify, intercept, or refuse. We present VerifiQuant, a governance-by-execution framework that packages financial logic into inference contracts and evaluates each query through an executable diagnostic funnel before deterministic computation. This separation makes computation, clarification, and refusal structured and auditable while keeping financial arithmetic outside the language model. Against matched retrieval-and-execution baselines, and within a declared contract library, the principal benefit is not higher accuracy on well-posed questions but safer behavior when an input does not justify a number. When it finds no applicable contract, it says so instead of guessing. Coverage sweeps and failure analysis put a price on this boundary: safety is bought with coverage rather than with accuracy, and residual failures remain attributable to specific runtime frontiers.

Named exits

Five ways a question fails to justify a number

Every response is a correct number, a silent wrong, or a named exit that says why no number was given. The deployment question is not “how accurate?” but “when is emitting a number legitimate?”

N

Out of scope

No contract covers the ask. VerifiQuant refuses and says so.

retrieval score selector

M

Unclear intent

Several quantities could be meant. It asks which one.

selector

F

Missing input

A required value is absent. It asks for the field by name.

input gate

E

Impossible value

An input breaks a declared boundary. The alert names the check and the value.

boundary gate

I

Open convention

Percent or decimal? Start or end of period? HARD: stop and ask. SOFT: answer with a warning.

ambiguity gate

C

Compute

Every gate passes. The contract's own Python computes the number.

locked code

The mechanism

An executable funnel decides before anything is computed

An analyst does not re-derive a loan formula for each client. They open a vetted sheet, type in the inputs, and its validation refuses impossible values. VerifiQuant turns each formula family into such a sheet, a Financial Inference Contract: typed inputs, boundary checks, declared conventions with their options, and locked code.

code

Retrieve

BM25 over contract metadata, top 4

N

LLM 1

Select

One contract, or none

NM

LLM 2

Bind

Text to typed inputs

code

Inputs

Every required input present?

F

code

Boundary

Declared checks hold?

E

LLM 3

Ambiguity

Declared convention left open?

I-hardI-soft warn

code

Compute

The contract's own Python

output

Number + report

DiagnosticReport on every path

LLM stepdeterministic codenamed exita number leaves

The LLM gets three jobs, and only three

Pick the contract from the four retrieved, or none. Bind values in the text to its typed inputs. Flag which declared conventions the text leaves open. It never computes, and it never edits the formula.

Every path writes a DiagnosticReport: contract and version, candidates and scores, bound values, checks passed, and why it stopped. Deterministic steps replay exactly from the stored binding. The 250 contracts were built offline (three LLM calls per formula family, then lint and smoke tests), none was hand-tuned, and all are fixed at run time.

Who does what	Picks formula	Arithmetic	Decides if a number may leave
CoT	LLM	LLM	LLM
PAL, R-PAL	LLM writes a program	interpreter	nothing checks
R-PAL⁺	LLM writes a program	interpreter	LLM, told the checks
VerifiQuant	LLM selects a contract	contract's code	executable gates

Results

Execution is not governance

Every system uses Gemini 2.5 Flash, the same top-4 retrieved contracts, deterministic execution and one scorer. R-PAL⁺ and VerifiQuant trap figures are means of three runs.

Traps: false numeric acceptance (lower is safer)

80 traps built so that no valid number exists (F, E, I and N, 20 each). FNAR is the share still answered with a number.

R-PAL48.8%

R-PAL⁺27.5%

VerifiQuant10.4%

−17.1 points versus R-PAL⁺, base-question-clustered 95% CI [6.8, 27.0]. Describing the checks in a prompt is not the same as enforcing them.

FNAR by trap type	R-PAL	R-PAL⁺	VerifiQuant
E · boundary	90%	30%	8.3%
I · ambiguity	95%	80%	21.7%
F · missing input	10%	0%	10%
N · scope	0%	0%	1.7%

The gain sits in E and I. A missing field or a blatant out-of-scope ask is visible enough for a prompt to refuse.

The price: safety costs coverage, not accuracy

250 well-posed questions. Matched execution already does well here, and governance adds no accuracy.

Clean set (n = 250)	Acc	SWR	Coverage
CoT, single shot	77.6	18.0	95.6
PAL, no retrieval	91.2	7.2	98.4
R-PAL, top-4 contracts	96.8	2.4	99.2
R-PAL⁺, checks in prompt	92.0	0.9	92.9
VerifiQuant, K = 1	81.2	2.8	84.0
VerifiQuant, K = 3 †	93.6	3.2	96.8

SWR = silent-wrong rate. K = 1 is fully autonomous. † K = 3 is an intent-oracle upper bound: the oracle sees the intent specification, never the answer.

A staged control that reads the same contract text matches VerifiQuant on traps (8.8%) but answers fewer clean questions (74.8% vs 84.0%), mostly because its self-written programs fail to run. The full funnel takes 12.2 s per question against 10.5 s for R-PAL, about 1.2×.

Outside the library: a scope signal, not more answers

88.9% of 199 out-of-library questions declined

against 19.6% of in-library ones, on a fixed split whose 199 questions come from formula families absent from a 50-family library (same benchmark). R-PAL declines 0.5% and answers 95.0% correctly from parametric memory, with no signal when it is wrong. VerifiQuant is not more accurate out of library: it answers 7.0% correctly, and its silent-wrong rate there is no lower (4.0% vs 4.5%). The paper names this as the result that most constrains its claim.

On an 80-query set labeled blind by five finance-trained annotators (Fleiss κ = 0.84), VerifiQuant emits a number on 8.7% of the 46 queries that call for none, against 13.0% for R-PAL⁺, 21.7% for CoT and 56.5% for R-PAL. Only the gap to R-PAL is significant, most of VerifiQuant's declines are scope refusals, and it answers 20.6% of the answerable items.

Verified edits, not regeneration

97.7% of 879 wrong patches rejected

A non-default convention changes the computation through an edit declared on the contract at build time and checked before it runs: bounded AST size, whitelisted names, and agreement with a declared algebraic relation on sampled bindings. 179 transform specs × 6 patch-error classes gave 906 mutants, 27 of them equivalent and excluded. 20 false accepts remain (2.3%), and 100% of structural attacks are rejected.

This is bounded numerical validation, not formal equivalence: the 20 false accepts sit in conditional transforms whose guard branch none of the five bindings exercised. Try the loan transform →

Boundary

Where the claim holds, and where it fails

Holds for

Single-number, formula-grounded questions whose formula family is in the library and whose contract is correct.

Fails at

Residual silent wrongs trace to three LLM steps: contract selection, input binding, and recall of a declared convention. Gates check schema and bounds, not where a value came from.

BibTeX

@inproceedings{ko2026verifiquant,
  title     = {VerifiQuant: Governance-by-Execution for Financial LLM Workflows},
  author    = {Ko, Yu-Chi and Yeh, Mi-Yen and Lin, Shou-De},
  booktitle = {Proceedings of the 7th ACM International Conference on AI in Finance (ICAIF '26)},
  year      = {2026},
  address   = {Milan, Italy},
  publisher = {ACM},
  doi       = {10.1145/3858063.3858688}
}

Published under CC BY 4.0. The DOI resolves once the ICAIF '26 proceedings are online.

A number is not an answer

Governability, not accuracy

Five ways a question fails to justify a number

Out of scope

Unclear intent

Missing input

Impossible value

Open convention

Compute

An executable funnel decides before anything is computed

The LLM gets three jobs, and only three

Execution is not governance

Traps: false numeric acceptance (lower is safer)

The price: safety costs coverage, not accuracy

Outside the library: a scope signal, not more answers

Verified edits, not regeneration

Where the claim holds, and where it fails

Holds for

Fails at

Next

BibTeX