An AI research workflow can return a different conclusion without changing any of its source reports. The final model decides which arguments deserve influence, which disagreements matter and what reaches the reader. That decision needs its own acceptance test.
For a team comparing research across APAC markets and languages, a fluent English summary can make that decision hard to see. Translation, retrieval and synthesis may sit inside the same workflow. If the final view changes, the team needs to identify which stage changed before judging whether the change is useful.
What identical reports revealed
In my study, When Models Judge Models, published in October, I examined an archived panel of six equities and one crypto asset. Each asset had fourteen saved source reports. Two deployed synthesis systems received those reports and assigned influence weights, which deterministic code used to combine the reports’ quarterly forecasts.
The baseline comparison reversed the projected five-year return direction for two of the six equities. The reports were unchanged; the resulting forecast depended on how the synthesizer weighted them. This was a descriptive workflow experiment, not evidence that either system predicted prices better or produced superior investment results.
The design preserved a blinded baseline, an unchanged-presentation repeat, a reversed report order and a condition disclosing the source models. Across seven assets, that produced 98 source reports and 56 synthesis outputs. The public replication materials retain the reports, requests, weights and calculations. An independent offline reproduction passed 392 numerical checks without making new model calls.
Those checks establish that the archived calculations can be reproduced. They do not establish that the underlying reports are true. Each pool contained twelve reports from one model family and one from each of two others. Fourteen reports therefore did not mean fourteen independent sources of intelligence.
The comparison also covered deployed systems rather than isolated model architectures. Web access was available, provider controls differed and execution times were separated. Fixed reports did not guarantee identical information retrieved during synthesis. The finding supports inspecting the aggregation stage; it does not support a general vendor ranking.
Preserve the research object before comparing the prose
Consider a hypothetical regional team reading a Chinese-language filing, an English broker note and an internal analyst memo about the same issuer. An apparently shared company name does not establish that every document concerns the same listing, currency or time horizon. A synthesized narrative can be coherent while combining incompatible objects.
Before running a comparison, I would preserve the original document alongside its translation and extracted claims. The record should contain the issuer identifier, exchange and listing, source currency, reporting period, publication timestamp, retrieval timestamp and source version. A document hash helps establish whether two runs actually received the same file. A timestamp alone cannot prove that.
Keep accounting currency, security price currency and presentation currency as separate fields. If a workflow converts values, save the conversion rate, its timestamp and the original amounts. Preserve whether a forecast describes price return or total return, its anchor date and its target horizon. A twelve-month view cannot silently become a five-year view during summarization.
Translation should be a visible transformation. Retain the source language, translator or model version, and the exact translated artifact supplied to the synthesizer. An analyst should be able to trace a disputed claim back to the original passage. Testing translation changes separately from synthesis changes makes the result easier to diagnose.
These are proposed controls for regional workflows, not findings from a multilingual APAC benchmark. The experiment did not measure translation accuracy or deployments across the region.
Replay one stage at a time
Start with a frozen bundle and a saved baseline. Record the prompt, report order, model version, settings and tool policy. Decide whether retrieval will be disabled or restricted to an archived evidence bundle. If live retrieval remains necessary, save its results and describe the comparison as a test of the deployed workflow rather than the model alone.
Repeat the unchanged setup before replacing the synthesizer. That repeat gives a reference for ordinary variation, although a small number of repeats cannot establish a reliable distribution. Then change one intended factor at a time: the synthesis model, report order, source labels or translated artifact. Preserve the full request for every run so unintended prompt changes remain detectable.
Compare three things separately: the final numerical output, the allocation of influence and the explanation. A small numerical difference can coexist with a substantial change in which evidence counts. Conversely, strongly worded changes in reasoning may have little effect on the calculation. Examine whether the explanation actually supports the weights and whether the cited evidence supports the explanation.
For numerical workflows, include a transparent deterministic comparator, such as equal weighting, and state the aggregation operator. Averaging quarterly returns and then compounding them is different from averaging complete forecast paths. Teams need to know which calculation their final narrative represents.
Define the release decision before seeing the result
A practical acceptance record should identify the input bundle, the intended change, numerical differences, changed assumptions, unresolved disagreements and the responsible reviewer. Set tolerances before running the comparison. Missing provenance, an unexplained currency conversion or a changed horizon should trigger investigation even when the prose reads well.
The study’s limits matter here. It used a small, deliberately selected panel, with limited repeats. An extra checklist confounded the crypto asset’s order and disclosure conditions, so those treatment comparisons were separated from the matched equity analysis. No forecast target had matured at the manuscript cutoff. Source disagreement was not a calibrated prediction interval, and the observed changes did not establish causal provider bias.
APAC teams can use the same experimental discipline without assuming the same results. Preserve the research object, replay the synthesis stage and make changes reviewable. A replacement model should arrive with a record of what changed in the conclusion, how that change follows from the evidence and who accepted it.
Russlan Ramdowar is the founder of Future Edge Group and iPulse AI, an Open Agentic Investment Research Platform. Based in Abu Dhabi, he works on inspectable AI research workflows, evidence preservation and synthesis evaluation. He authored When Models Judge Models; the experiment discussed here was conducted within iPulse AI’s research pipeline. These affiliations are relevant to this contribution.
Editor’s note: This contributed article has been lightly edited for clarity and TNGlobal house style. The substance of the author’s contribution has been preserved.
Share your perspective: TNGlobal welcomes contributed insights and expert commentary from across Asia’s technology and innovation ecosystem. Submit a contribution for editorial consideration, or explore more perspectives in TNGlobal INSIDER and conversations in the TNGlobal Q&A and Interviews archive.
Featured image: Steve A Johnson on Unsplash