Autonomous AI systems are beginning to move beyond coding assistance into longer-horizon research and engineering workflows. That raises practical questions about how systems choose experiments, validate evidence, control costs and determine when human review is still required.
TNGlobal spoke by email with William Chen, Co-Founder and Head of Strategy & Management at Sapient Intelligence, about PRAXIST, the company’s autonomous R&D platform, its research-peer architecture, benchmark methodology, failure modes, data handling and how enterprises should judge whether autonomous R&D is useful in production.
The benchmark and engineering performance figures discussed below are company-reported.
How does PRAXIST differ technically from a coding agent, automated search system or hyperparameter optimization tool when it is given a long-horizon R&D problem? What capabilities allow it to decide how the problem itself should be approached rather than simply execute a predefined workflow?
The main distinction is that our technology is not designed around executing a known solution path. The user defines the objective, evaluator, environment, operating constraints and research budget, but the technical route to the objective can remain unknown. The system then determines which approaches should be explored through successive research generations.
Conventional coding agents are generally strongest when the user already broadly understands what needs to be built or tested. They can execute tasks, write code and run experiments, but they typically operate along a more linear trajectory. Our approach is designed for longer-horizon research, where the system may need to test competing hypotheses, learn from both successful and unsuccessful experiments and continually determine what should be investigated next.
Rather than preserving only the highest-scoring candidate, the system retains useful evidence across generations. This can include mechanisms from lower-performing candidates, diagnostics from failed experiments, uncertain findings requiring validation and insights that may become useful when combined with later discoveries.
The research process also tries to prevent every research peer from simply pursuing the current best-performing approach. Different peers can explore alternatives, reproduce findings, challenge assumptions, investigate failures or refine promising solutions. Their findings are then reviewed together to determine whether the next round of research should continue an approach, validate it, diagnose a failure, explore an alternative or stop a particular direction.
Experiments are expected to identify the mechanism being tested, how it relates to prior evidence, what evidence would support or weaken it and how the result can be validated. The system also deliberately distributes research across different approaches rather than allowing every peer to converge on the same candidate. Findings are tracked according to their maturity, and the research history records what supported, challenged or superseded earlier decisions.
An important boundary is that the system does not autonomously redefine an enterprise’s top-level objective, safety boundaries or deployment criteria. Humans remain responsible for those. The autonomy lies in how the technology investigates the technical solution space within those boundaries.
For the 75 MLE-Bench competitions cited in the launch, could you walk us through the evaluation setup in more detail? In particular, what model and tool versions, compute limits, retry policies and human intervention were permitted; what counted as a highest-level result; how was the Claude Code comparison run; and how were the approximately US$3,000 and US$38,000 model costs calculated?
The MLE-Bench evaluation tested our system across 75 Kaggle competitions covering areas such as image recognition, natural language processing and tabular data. For each task, the system was required to produce a valid submission, which was then scored using the official MLE-Bench grader against the competition’s private leaderboard.
A gold medal is the highest medal tier in MLE-Bench and indicates that a submission has met the benchmark’s gold-level performance threshold, which is calibrated against top human performance. In our evaluation, the system achieved 49 gold medals across the 75 competitions.
The evaluation was conducted without human intervention after a task was launched. Both our system and Claude Code were evaluated on the same 75 tasks using the same grading harness and the same shared pool of 16 NVIDIA H100 80GB GPUs.
| System | Underlying model | Benchmark setup | Gold medals | Any medals | Recorded model cost |
|---|---|---|---|---|---|
| PRAXIST | DeepSeek V4 Pro (preview), 1M-token context window | Full 75-task MLE-Bench sweep | 49 / 75 (65.3%) | 61 / 75 (81.3%) | ~$3,054 |
| Claude Code | Claude Opus 4.8, maximum thinking budget | Same 75 tasks, same grading harness and hardware pool | 34 / 75 (45.3%) | 55 / 75 (73.3%) | ~$38,370 |
Hardware: All evaluations used the same shared pool of 16 NVIDIA H100 80GB GPUs.
PRAXIST uses multiple research peers to explore hypotheses in parallel. How does the system decide which experiments are worth pursuing, and what prevents multiple peers from converging on the same incorrect assumption or optimizing for the benchmark rather than the underlying problem?
The system does not simply let every research peer chase the current best result. Instead, different peers are given different research directions to explore. The system considers factors such as how promising an idea appears, whether it introduces a genuinely different approach, the risks involved and whether an earlier result needs to be tested or challenged.
To reduce the risk of premature convergence, we use diversity mechanisms that limit how many peers can work on the same type of approach, the same part of the problem or the same earlier solution. Exact duplicate experiments are normally avoided unless the objective is specifically to reproduce a result.
Peers can also be assigned different research roles, such as exploring an alternative idea, validating a result, challenging an assumption, diagnosing a failure or improving a leading solution. A strong-performing approach therefore does not automatically become accepted as correct.
We also retain failed experiments and negative results so later research does not simply repeat the same mistakes. A high score is treated separately from a valid result. If a result performs well but relies on weak evidence or violates the defined constraints, it is not automatically treated as confirmed. Promising results can instead be sent for further validation, reproduction or ablation testing.
What validation and reproducibility checks take place before an output is treated as a useful research result? Which stages still require a domain expert to review the methodology, interpret the result or reproduce the finding independently?
We evaluate findings based on the strength and maturity of the supporting evidence. Before a result is treated as confirmed, it can go through increasingly rigorous stages of validation, including full evaluation and scoring, reproduction, ablation studies and additional follow-up testing.
Reproducibility is also built into the research process. For key experiments, the system records the evaluators used, experimental code, results, known limitations and resulting findings. That gives researchers a trail they can inspect to understand how a conclusion was reached and to reproduce the underlying experiment.
Human experts remain an important part of the process. They define the research objective, evaluators, constraints and safety boundaries, and they review findings before deciding whether a result is appropriate for integration or deployment. The technology is intended to automate much of the repeated experimentation, evaluation and validation work around that process, rather than replace domain expertise or expert judgment.
In the rocket-landing and robotic SLAM examples, what parts of the problem were fixed by the partner and what was PRAXIST allowed to optimize? How were the baselines established, and what information would an independent engineering team need in order to reproduce those results?
In the rocket-landing example, the system was given a fixed simulator and a fixed definition of a successful landing. It could then experiment with parts of the controller and certain vehicle or contact parameters to improve performance.
One change involved adjusting a landing-leg parameter from 0.95 to 0.90. When the improved version was retained and tested, it achieved 30,720 successful landings out of 30,720 trials. A frozen version was also tested on another 40,960 unseen trajectories, where it again achieved a 100 percent success rate under the nominal conditions used in the evaluation.
It is important to note that this was not a comparison with a separate state-of-the-art rocket controller. The initial comparison was against the previous version of the same controller.
Baselines depended on the task. For the rocket-landing work, we also evaluated the same environment using Weco as a baseline. Under that setup, our system achieved a 100 percent landing success rate compared with 17 percent for Weco. For the SLAM task, the baseline was an algorithm designed by human experts.
To support independent verification, we have made the research trajectories available online, including the problem setup and information needed to reproduce the experiments. An independent team can use those materials to rerun the work in the same environment and compare its results with those we reported.
PRAXIST can operate with proprietary data in private or customer-controlled environments. How are data isolation, access permissions, logging, retention and intellectual property handled? Can customer data or research outputs be used to train or improve models outside that customer’s environment?
Customers’ proprietary research data and research outputs are not used for model training. The information we collect is limited to high-level operational metrics, such as the number of runs, runtime duration and error categories. It excludes the private details of users’ research tasks and is collected only with user consent.
For organizations with stricter privacy requirements, the technology can also be deployed locally. A compatible foundation model can run on the customer’s own hardware and connect directly to the system, allowing research data to remain within the customer-controlled environment.
What failure modes have you seen during the beta stage? For example, when might PRAXIST pursue an unproductive research path, produce a technically valid but impractical result, or consume more compute without meaningful improvement? What causes the system to stop, escalate or ask for human intervention?
During the beta stage, the main failure modes we have seen are similar to those found in human-led research. An unproductive direction can consume experiments without producing enough improvement, or a solution can be technically valid but impractical because an important real-world constraint was not captured in the original setup.
Our research process is designed to manage these cases by preserving unsuccessful results as findings and allowing the system to shift toward other approaches rather than continuing indefinitely in the same direction.
The evaluator defines what counts as progress, the constraints define what solutions are acceptable or feasible, and the research budget limits how much time and compute can be spent exploring. Users can also define task-specific stopping criteria.
Human intervention becomes particularly important when the issue is not the research process itself, but whether the original problem, environment or constraints accurately represent the real-world objective.
Beyond benchmark medals, what should an enterprise measure to determine whether autonomous R&D is actually useful in production, for example time to a reproducible solution, expert hours saved, compute cost, failure rate, or the proportion of outputs that can move into real engineering work? Which industries or use cases in Asia Pacific appear to be the strongest near-term fit today?
Beyond benchmark performance, enterprises should assess autonomous R&D by whether it can produce reproducible technical improvements while reducing the time, expert effort and cost required to reach them.
Four measures are particularly useful: the improvement achieved against the target objective, time to a reproducible and usable solution, expert hours saved and the total cost of reaching that result. Ultimately, the stronger test is whether an output can move beyond experimentation and into real engineering work.
Our rocket-landing project is one example. Based on our estimate, a traditional team of around eight engineers could typically need about a month to build and validate a proof of concept in simulation and bring the landing success rate close to 100 percent. In our test, the system reached a 100 percent safe-landing rate in 12 hours, bringing the work to Technology Readiness Level 3, or proof-of-concept stage.
The relevant metric is not simply the final performance, but how much specialist research time can potentially be compressed into a shorter cycle.
We see a similar effect in research throughput. A single overnight run can generate roughly 300 experimental reports. We estimate that this is comparable in volume to the annual experimental output of around ten PhD researchers. That does not mean every experiment becomes a useful or production-ready solution. Rather, it illustrates the volume of research that an expert team can potentially explore and evaluate with additional automation.
For enterprises, the value therefore should not be judged only by whether the technology finds a better solution. It should also be measured by how much specialist time it saves, how quickly the result can be reproduced and validated and how efficiently useful findings can move into engineering.
In terms of near-term applications, we see the strongest fit in R&D-intensive problems with clearly defined objectives, measurable outcomes and environments where proposed solutions can be experimentally validated. In Asia Pacific, that includes areas such as manufacturing, industrial automation, robotics and engineering optimization, where relatively small technical improvements can translate into gains in throughput, cost, reliability or resource utilization.
William Chen is Co-Founder and Head of Strategy & Management at Sapient Intelligence. His background spans robotics, embedded systems and commercialization, including R&D roles at DJI and Hesai Technologies and venture development at the Tsinghua Innovation Center.
Editor’s note: This Q&A has been edited for clarity, concision and TNGlobal house style. The substance of the interviewee’s responses, including views and attributed claims, has been preserved.
Share your perspective: TNGlobal welcomes contributed insights and expert commentary from across Asia’s technology and innovation ecosystem. Submit a contribution for editorial consideration, or explore more conversations in our TNGlobal INSIDER and TNGlobal Q&A and Interviews archive.