Einsia AI Launches Visual Cipher Bench to Evaluate Long-Horizon Visual Reasoning in AI Agents
Top-performing model achieves just 26.7% accuracy, revealing a significant gap in autonomous rule discovery
SINGAPORE, Nov. 18, 2026 /PRNewswire/ — As AI agents evolve from answering questions to completing complex tasks, a fundamental challenge remains: Can they solve problems when instructions are absent and rules must be discovered entirely from visual clues?
Einsia AI‘s Navers Lab today announced Visual Cipher Bench, a new benchmark evaluating long-horizon visual reasoning in AI agents, focusing on rule discovery, logical planning, and spatial operations.
Across eight frontier models and 29 reasoning configurations, the benchmark recorded 3,480 attempts. No configuration achieved accuracy above 30%. The top performer, GPT-6 Astra (max), scored 26.7%. DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro failed to solve any puzzles.
The findings expose a critical gap between executing known instructions and independently discovering unknown rules. While spatial reasoning proved relatively manageable, accuracy on rule induction was just 0.6%.

Figure 1: Overview of Visual Cipher Bench. Left: 40 original puzzles (67 images) in four categories. Middle: each model’s best configuration, averaged over three runs. Right: the evaluation pipeline, from puzzle images to a single scored answer.
Testing AI Beyond Conventional Visual Question Answering
Unlike conventional visual benchmarks, which typically provide explicit questions and expected answers, Visual Cipher Bench tests scenarios where instructions are hidden in images, with no explicit rules or immediate correctness feedback.
The benchmark comprises 40 original visual puzzles represented by 67 black-and-white illustrations. Agents receive one instruction: examine the images and write the final answer to a file. They must independently identify examples, understand the target problem, infer the required output, and solve each task.
The puzzles cover four categories:
- Spatial Operations: Folding, rotation, projection, and geometric transformations.
- Logical Planning: Extracting visual constraints and finding valid solutions.
- Rule Induction: Discovering hidden patterns from limited examples.
- Knowledge Decoding: Interpreting Morse code, Braille, and word puzzles.
Frontier Models Struggle with Autonomous Rule Discovery
Performance varied significantly across the eight evaluated models:

Figure 2: Accuracy (%) of the eight evaluated models. Each bar is one reasoning-effort setting, from lowest to highest; the number shown is the model’s best configuration.
- GPT-6 Astra (max): 26.7%, averaging 10.7 correct answers per 40-puzzle run.
- GPT-6.1 Sol (max) and Claude Opus 5.5 (xhigh): Both scored 21.7%.
- GPT-6 Luna and Claude Sonnet 5.5: Both scored 0.8%, solving one puzzle each across 120 attempts.
- DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro: Zero correct answers.
Across all 3,480 attempts, accuracy by category was:
- Spatial Operations: 22.8%
- Knowledge Decoding: 3.3%
- Logical Planning: 2.4%
- Rule Induction: 0.6%
Models solved only five of 870 rule-induction attempts. Even GPT-6 Astra (max) achieved 80% on spatial operations, while no configuration exceeded 13.3% in the other three categories. Researchers also found that some correct answers resulted from thematic guesses rather than systematic rule discovery.

Figure 3: Full results for all 29 model–effort configurations, with accuracy by category and overall. Shaded rows mark each model’s best configuration.
Providing Rules Raises Accuracy from 12.2% to 51.1%
To distinguish rule discovery from rule execution, researchers conducted an additional experiment on 10 puzzles using GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna.

Figure 4: Hint experiment on 10 puzzles with three models. A one-sentence mechanism hint raises overall accuracy from 12.2% to 51.1%, and rule induction from 0% to 85.2%.
When provided with the core rules—but not intermediate reasoning steps or answers—overall accuracy increased from 12.2% to 51.1%. Rule-induction accuracy rose from 0% to 85.2%.
GPT-6 Astra improved from 23.3% to 70.0%, while GPT-6.1 Sol increased from 13.3% to 66.7%. Timeout occurrences also fell from 51 to 29 across 90 attempts.
A control experiment providing additional information about task-state representation, but not the core rules, achieved only 16.7% accuracy. Even with rule hints, logical planning accuracy remained at 22.2%.
Why Do AI Agents Fail?
Analysis of failure trajectories revealed two recurring patterns.
Premature conclusions: Agents often adopted rules that explained only some examples without validating them against all available evidence. Some submitted answers despite acknowledging incomplete understanding.
Inefficient search: Agents frequently relied on extensive tool calls and exhaustive combinations instead of using visual constraints to narrow the search space, eventually timing out.
More effective reasoning involved generating multiple hypotheses, testing them against examples, eliminating inconsistent rules, and applying the remaining valid hypothesis.
From Following Instructions to Discovering Rules
In software development, agents typically receive explicit requirements and can rely on test results to evaluate their solutions. Real-world problems often provide neither.
Agents must observe their environment, identify relevant signals, formulate hypotheses, and validate their understanding without explicit instructions or immediate feedback.
Visual Cipher Bench provides a systematic evaluation of this capability gap. Its findings suggest that independently discovering, validating, and transferring hidden rules remains a significant challenge for frontier AI agents —a capability increasingly important as AI moves toward more autonomous problem-solving.
Research and Resources
- Paper: Visual Cipher Bench: Benchmarking LLM Agents on Long-Horizon Visual Reasoning
- Project Website: https://lab.einsia.ai/visual-cipher-bench/
- Research Paper: https://arxiv.org/abs/2611.16861
- GitHub: https://github.com/Einsia/Visual-Cipher-Bench

