Why LLMs Struggle to Replicate AlphaGo's Tree Search Miracle

Why Large Language Models Cannot Replicate AlphaGo's Tree Search Miracle: A Deep Technical Analysis

📖 Glossary

AI Box (also known as Agent Computer / Agent PC), is a dedicated local hardware device that runs AI Agents. Pre-installed with an AI agent management system, plug-and-play, running 24/7. Users can remotely command AI to work via Discord, Slack, Telegram, WhatsApp, and more.

Abstract: AlphaGo defeated the world Go champion using Monte Carlo Tree Search (MCTS) combined with deep learning, achieving what many thought impossible in a game with more possible positions than atoms in the universe. Yet despite massive scaling in parameters, data, and compute, Large Language Models (LLMs) cannot simply apply the same tree search methodology to conquer open-world problems. This article provides a rigorous technical analysis of why scaling law improvements don't translate to tree search capabilities, why LLM "reasoning" is fundamentally pattern matching rather than genuine search, and what this means for the future of AI agent development.

I. The AlphaGo Miracle: When Tree Search Met Deep Learning

In March 2016, AlphaGo defeated Lee Sedol 4-1 in a match that sent shockwaves through both the AI community and the broader public. The victory was remarkable not just because Go had been considered impenetrable to AI for decades, but because of the elegance of the approach.

AlphaGo's core architecture combined two techniques:

Monte Carlo Tree Search (MCTS)

MCTS is a heuristic search algorithm for decision processes with large branching factors. Its four-phase loop operates as follows:

Selection: Starting from the root node, select child nodes according to a policy (typically UCB1) that balances exploration and exploitation, traversing the existing tree until reaching a node that hasn't been fully expanded.
Expansion: Add one or more child nodes to the selected node, representing unexplored moves from that game state.
Simulation (Rollout): From the newly expanded node, perform a rapid simulation (typically using a fast rollout policy) to the end of the game to obtain an outcome estimate.
Backpropagation: Propagate the simulation result back up the tree, updating visit counts and value estimates for all nodes on the path.

Through thousands of iterations, MCTS gradually constructs an asymmetric search tree that focuses computation on the most promising branches. The key insight is that you don't need to search the entire tree—you just need to search the right parts of it.

Deep Neural Networks

The genius of AlphaGo was in using deep neural networks to dramatically improve MCTS efficiency:

Policy Network: Predicts the probability distribution over legal moves from any given board position. This narrows the search space by focusing MCTS on moves that strong players would consider, rather than exploring all 200+ legal moves at each position.
Value Network: Estimates the probability of winning from any given board position. This provides a much more accurate evaluation than random rollouts, allowing MCTS to make better decisions about which branches to explore.

The policy network reduces breadth (fewer moves to consider), and the value network reduces depth (no need to simulate to the end of the game). Together, they transform an intractable search problem into a manageable one.

AlphaGo's success was fundamentally the synergy of search and learning: search provides precise direction, learning provides efficient evaluation. Neither alone could have defeated a world champion.

Why Go Was the Perfect Testbed

Go possesses four properties that make it an ideal domain for tree search:

Finite (though massive) state space: Approximately 10^170 legal board positions—astronomically large but well-defined and bounded.
Unambiguous reward signal: At the end of every game, the outcome is definitively win or lose. There's no subjectivity, no ambiguity.
Immutable rules: The rules of Go have remained essentially unchanged for over 3,000 years. The game environment is perfectly stable.
Perfectly repeatable simulation: Every position can be simulated, every move's consequences can be deterministically evaluated. You can play out the same position a million times and get consistent results.

These four conditions—finite space, clear reward, stable rules, repeatable simulation—are precisely what LLMs lack when facing open-world problems.

II. The Open-World Problem: Where Tree Search Breaks Down

Large language models operate in an open-world environment that is fundamentally different from the closed world of Go. Let's examine each dimension of this difference.

2.1 Search Space: Infinite and Unstructured vs. Finite and Structured

Go's search space, while enormous, is finite and highly structured. The game tree has a clear topology: each node represents a board state, each edge represents a legal move, and the rules define exactly which edges are available from any node. The branching factor is bounded (at most 361 legal moves per position on a 19×19 board).

The open world that LLMs navigate has no such structure. Consider a simple instruction: "Write a persuasive email to your boss requesting a raise." The "search space" for this task includes:

Every possible combination of words in the English language (~1 million words)
Every possible sentence structure (infinitely many)
Every possible rhetorical strategy (unbounded)
Every possible tone, style, and format (unbounded)

There's no function that defines "legal moves" from a given partial text. There's no finite set of successors from any given state. The space is not just large—it's unbounded and unstructured.

LLMs don't search; they associate. Each token generation is a probability sample from a learned distribution, not a decision selected from an enumerated set of alternatives.

This distinction has profound implications. MCTS works because it can enumerate and evaluate alternatives. When alternatives cannot be enumerated—when the "move space" is continuous rather than discrete, when "legal moves" cannot be defined—MCTS has nothing to operate on.

2.2 Reward Signal: Verifiable vs. Unverifiable

In Go, every simulation produces an unambiguous outcome: Black wins or White wins (or, extremely rarely, a draw by agreement). This signal is verifiable by the rules of the game. No human judgment is required. If you play out the simulation and count the territories, the result is deterministic.

In the open world, how do you evaluate whether a generated response is "good"? This is the reward specification problem, and it's unsolved for most real-world tasks:

Fluency: A response can be perfectly fluent while being completely factually wrong. A hallucinated scientific citation reads just as smoothly as a real one.
Factual accuracy: How do you automatically verify whether a generated claim is true? This requires either a reliable external knowledge base (which may not exist for the specific claim) or human evaluation (which doesn't scale).
Helpfulness: Whether a response is helpful depends on the user's intent, context, and expertise level—the same response can be helpful to one user and useless to another.
Safety: A response might be accurate and helpful but still inappropriate in certain contexts.

The absence of verifiable reward signals means LLMs cannot perform the crucial backpropagation step that makes MCTS effective. Without reliable outcome evaluation, there's no signal to guide the search toward better branches.

2.3 The Verification Bottleneck

This deserves deeper examination because it's the core of the problem. In Go, AlphaGo can play a thousand games per second and get a thousand verifiable outcomes. This creates a tight feedback loop: try something → see if it works → adjust.

For LLMs generating text, the equivalent would be: generate a response → verify it's correct → adjust. But the verification step requires:

A ground truth to compare against (often doesn't exist)
A formal specification of correctness (often can't be defined)
An automated verification procedure (often doesn't exist)

In code generation, verification is somewhat tractable—you can run test cases. But for most text generation tasks (summarization, creative writing, advice-giving, analysis), there's no automated verification procedure. You need a human in the loop, and humans are expensive and slow.

The verification bottleneck is why LLMs can't simply "search their way" to better outputs. Search without verification is just random exploration.

III. Scaling Law ≠ Tree Search: Two Different Dimensions of Intelligence

The scaling laws observed in LLMs—performance improves predictably with more parameters, more data, and more compute—have been the dominant paradigm in AI progress. But scaling laws and tree search operate on fundamentally different dimensions.

3.1 What Scaling Laws Actually Improve

Scaling laws improve what we might call pattern matching capacity:

Knowledge breadth: Larger models store more factual knowledge across more domains.
Pattern complexity: Larger models capture more complex statistical relationships in the training data.
Generation fluency: Larger models produce more coherent, natural-sounding text.
In-context learning: Larger models better follow patterns demonstrated in the prompt.

These improvements are real and significant. A 175-billion-parameter model genuinely knows more and generates better text than a 7-billion-parameter model. But all of these improvements are in the dimension of "recognizing and reproducing patterns seen during training."

3.2 What Tree Search Actually Improves

Tree search improves decision optimization capacity:

Exploration: Systematically considering alternatives that might not be the "obvious" choice.
Lookahead: Predicting the consequences of choices several steps ahead.
Backtracking: Abandoning unpromising paths and trying alternatives.
Optimization: Finding the best path through explicit evaluation and comparison.

These capabilities are qualitatively different from pattern matching. A model with perfect pattern matching would always select the statistically most likely next step—the one most similar to its training data. But the optimal decision is often not the most statistically likely one. Sometimes the best move is the surprising one, the unconventional one, the one that doesn't resemble anything in the training data.

Scaling laws make LLMs more "knowledgeable" but not more "strategic." Knowledge and strategy are different dimensions of intelligence, and optimizing one doesn't automatically improve the other.

3.3 The Complementarity Principle

The AlphaGo insight was that pattern matching and tree search are complementary:

Pattern matching (neural networks) provides the intuition to focus search on promising areas
Tree search (MCTS) provides the rigor to verify and refine those intuitions through systematic exploration

Current LLMs have the intuition but lack the systematic search mechanism. They can identify promising directions (because their pattern matching is excellent) but cannot reliably verify whether those directions lead to correct solutions (because they cannot perform systematic search with verifiable outcomes).

This is why LLMs can produce impressively coherent explanations but also confidently hallucinate incorrect facts—their pattern matching is strong enough to generate plausible-sounding content, but they lack the search-and-verify mechanism that would catch errors.

IV. LLM "Reasoning": Pattern Matching Masquerading as Search

When we observe LLMs producing step-by-step reasoning, it's tempting to interpret this as genuine search behavior. But the underlying mechanism is fundamentally different.

4.1 Chain-of-Thought Prompting: Imitation, Not Exploration

Chain-of-Thought (CoT) prompting has been one of the most impactful techniques for improving LLM reasoning. By prompting the model to generate intermediate reasoning steps before giving a final answer, CoT significantly improves performance on mathematical, logical, and multi-step reasoning tasks.

But how does CoT actually work? The model generates intermediate steps by pattern matching against reasoning templates in its training data. When a model sees a math word problem and generates "First, we calculate A. Then, we calculate B. Finally, we compute C," it's not exploring alternative solution paths and selecting the best one. It's retrieving and applying a solution template that resembles the patterns it observed during training.

Evidence for this interpretation comes from several observations:

Brittleness to problem variation: Changing the numbers in a math problem or rephrasing the question can cause dramatic performance drops, even when the underlying logic is identical. If the model were genuinely searching, minor surface variations shouldn't matter.
Error patterns: LLM reasoning errors often show a distinctive pattern—they make a "plausible but wrong" step early in the chain, then faithfully continue the (now-flawed) reasoning to arrive at a confident but incorrect answer. This is exactly what you'd expect from pattern matching without verification.
No backtracking: When a human solves a problem and encounters a dead end, they backtrack and try a different approach. LLMs almost never do this spontaneously. Each "reasoning chain" is a forward-only sequence, never revising earlier steps.

Chain-of-thought improves the surface quality of LLM reasoning, but it doesn't change the underlying mechanism from pattern matching to genuine search. It's better imitation, not genuine exploration.

4.2 The Hallucination Problem as a Structural Consequence

LLM hallucination—generating confident but factually incorrect content—is not a bug that can be patched. It's a structural consequence of the autoregressive generation mechanism.

Every token an LLM generates is sampled from a probability distribution conditioned on all preceding tokens. The model selects the most probable next token (or samples from the distribution with temperature). There's no step in this process where the model checks whether the generated content is factually correct.

AlphaGo doesn't hallucinate because every simulation produces a verifiable game outcome. If a particular move leads to a losing position, MCTS decreases the value estimate for that branch. The verification is built into the loop.

LLMs have no such verification loop. A generated sentence that's grammatically perfect and topically relevant but factually wrong has the same surface features as a correct sentence. The model has no mechanism to distinguish between them during generation.

This is why techniques like retrieval-augmented generation (RAG) are important—they provide an external verification signal. But RAG is a patch, not a solution. It helps with factual accuracy but doesn't enable genuine search-and-verify reasoning.

4.3 The "Confident Wrongness" Phenomenon

Perhaps the most concerning aspect of LLM reasoning is its confidence calibration. LLMs tend to be equally confident whether they're right or wrong. A model will assert an incorrect fact with the same linguistic certainty as a correct one.

This is fundamentally different from human reasoning, where uncertainty typically manifests as hedging language ("I think," "possibly," "I'm not sure but..."). The reason is that human uncertainty often reflects genuine search behavior—we're uncertain when we haven't fully explored the solution space or when multiple paths look promising.

LLMs don't search, so they don't experience exploration-based uncertainty. Their "confidence" is simply a reflection of how strongly the pattern matching activates, which correlates with how well the input matches training patterns—but not with how correct the output is.

V. Implications for AI Agent Development

Understanding the fundamental gap between LLM reasoning and tree search has critical implications for how we build AI agents.

5.1 Agents Need Search, Not Just Language

Current AI agents are predominantly built on LLMs. Their strength is in understanding natural language instructions and generating natural language responses. Their weakness is in planning and decision-making—exactly the capabilities that tree search provides.

This is why frameworks like OpenClaw are introducing tool calling and multi-step planning mechanisms. The idea is to let the LLM handle the "understanding" part (what does the user want? what tools are available?) while delegating the "planning" part to more structured algorithms (what sequence of tool calls will achieve the goal?).

But current implementations are still limited. The "planning" is often just another LLM prompt—ask the model to generate a plan, then execute it. This is pattern-matching-as-planning, not genuine search-based planning. A more principled approach would use actual search algorithms (MCTS, A*, or even simple BFS/DFS) over a defined action space, with the LLM providing heuristics for guiding the search.

5.2 World Models: The Missing Piece

AlphaGo was powerful because it had a world model—a deep understanding of Go's rules and board evaluation. This model enabled it to predict the consequences of different moves and evaluate positions accurately.

LLMs lack such world models. They have vast statistical knowledge about how words relate to each other, but they don't have causal models of how the world works. An LLM knows that "dropping a glass causes it to break" because it's seen this pattern in training data, but it doesn't understand gravity, material strength, or kinetic energy—the underlying causal mechanisms.

Building world models for LLMs is a frontier research problem. Approaches include:

Simulator-based models: Training models to predict the outcomes of physical simulations.
Causal reasoning frameworks: Embedding causal graphs into model architectures.
Interactive learning: Letting models learn world dynamics through interaction (like a child learning physics by playing with blocks).

None of these approaches is mature enough to provide the kind of reliable world model that would enable tree search in open domains. But this is likely the direction that will ultimately close the gap between LLM reasoning and genuine search.

5.3 Hybrid Architectures: The Pragmatic Path Forward

Given the current limitations, the most practical path forward is hybrid architectures that combine LLMs with structured search algorithms:

LLM for understanding: Parse natural language instructions, identify user intent, and generate natural language responses.
Search algorithms for planning: Use MCTS or other search methods over defined action spaces to find optimal action sequences.
World models for prediction: Use learned or engineered models to predict action consequences and provide verifiable reward signals.
Reinforcement learning for optimization: Learn from long-term interaction to improve strategy over time.

This hybrid approach mirrors what made AlphaGo successful—combining the pattern recognition strength of neural networks with the systematic optimization capability of search algorithms. The difference is that AlphaGo operated in a closed world with perfect information, while agents must operate in an open world with imperfect information.

The challenge is significant, but the direction is clear. Pure LLMs will hit a ceiling on reasoning and planning tasks. The next breakthrough will come from architectures that combine LLMs' language understanding with genuine search capabilities.

5.4 What This Means for Agent Infrastructure

For teams building AI agent systems (like OpenClaw, Hermes Agent, or Kaihe A1's agent management platform), this analysis suggests several practical priorities:

Don't rely solely on LLM reasoning for critical decisions. Build in verification steps, human-in-the-loop checkpoints, and fallback mechanisms.
Invest in structured planning layers. Use actual search algorithms for multi-step planning rather than asking the LLM to generate plans via pattern matching.
Build domain-specific world models. In constrained domains (code generation, data analysis, system administration), it's possible to build verifiable models of action consequences.
Design for graceful failure. Since LLM reasoning will sometimes fail, design agent systems that can detect and recover from reasoning errors.

VI. The Bigger Picture: Intelligence as Search

There's a provocative hypothesis in cognitive science: intelligence is search. Herbert Simon, the Nobel Prize-winning economist and AI pioneer, argued that problem-solving is essentially searching through a problem space, and that most of what we call "intelligence" is about searching more efficiently—knowing where to look, recognizing promising directions, and avoiding dead ends.

Under this view, AlphaGo's success makes perfect sense: it got really good at searching Go's problem space. And LLMs' limitations also make sense: they're not searching at all—they're pattern matching.

The implication is that building truly intelligent AI agents requires building systems that can search effectively in open-world problem spaces. LLMs are a crucial component—they provide the language understanding and pattern recognition that guides the search—but they're not sufficient on their own.

The path from current LLM-based agents to truly intelligent agents will likely require:

Better representations of problem spaces (so search algorithms have something to operate on)
More reliable reward signals (so search results can be evaluated)
More efficient search algorithms adapted to open-world settings (where the branching factor is effectively infinite)
Better integration between LLMs and search algorithms (so the strengths of each are fully leveraged)

VII. Conclusion: Understanding Limits Is More Important Than Exaggerating Capabilities

AlphaGo's tree search miracle was built on the ideal conditions of a perfect-information game. The open world that LLMs face is far more complex—and far less amenable to the techniques that made AlphaGo successful.

This doesn't mean LLMs lack value. Their capabilities in understanding, generating, and reasoning about natural language represent one of the most important advances in AI history. But we must be clear-eyed about their limitations: their "reasoning" is pattern matching, their generation is probabilistic sampling, and they lack genuine search and planning capabilities.

The most productive path forward isn't to pretend these limitations don't exist or to assume that scaling alone will overcome them. It's to build systems that combine LLMs' remarkable pattern-matching abilities with genuine search algorithms and world models, creating agents that can both understand language and reason strategically about the world.

Understanding the limitations of LLMs is more valuable than exaggerating their capabilities. Only with honest assessment can we find the right direction for building truly intelligent AI agents.

The AlphaGo miracle showed us what's possible when search and learning work together. The challenge now is to extend that synergy from the closed world of Go to the open world of human problems. That's the frontier—and it's where the next breakthroughs will come from.

VII. Implications for AI Agent Architectures

Why does this matter for AI Agents (like OpenClaw)?

Because an Agent's effectiveness depends on its ability to plan—to search through possible action sequences and pick the best one. Today's Agents (including OpenClaw) mostly use reactive planning (react to each observation), limited lookahead (1-3 steps), and no tree search.

If Agents could use MCTS-like algorithms to plan their actions, they'd be dramatically more capable. Imagine an Agent that, before executing a multi-step task, simulates several possible trajectories and picks the best one.

We're not there yet. But the direction is clear: the next generation of Agents will inherit more from AlphaGo than today's LLMs do.

VII.1 The Agent Planning Problem

Consider a simple agent task: "Research the latest developments in quantum computing and write a summary."

A reactive agent would: 1. Search for "quantum computing latest developments" 2. Read the top result 3. Write a summary based on that one source

A planning agent would: 1. Generate multiple search strategies (academic papers, news, expert opinions) 2. Evaluate which strategy is most promising (MCTS-like selection) 3. Execute the best strategy first, but keep alternatives in reserve 4. After gathering initial results, decide whether to go deeper or broaden 5. Synthesize information from multiple trajectories

The difference is night and day. The reactive agent produces a shallow, single-source summary. The planning agent produces a comprehensive, multi-perspective analysis.

VII.2 Why Hardware Matters for Agent Planning

Agent planning is computationally expensive because it requires running multiple "what-if" scenarios. Each scenario requires an LLM call. MCTS for agents might need 10-100× more LLM calls than reactive approaches.

This is where hardware like KaiheAiBox becomes critical: - 7×24 availability: Planning agents need sustained compute, not burst capacity - Local + cloud hybrid: Quick planning steps can use local models; complex evaluations can use cloud APIs - Memory for context: Planning requires maintaining multiple trajectories in memory simultaneously - Low power for sustained operation: 30W means planning agents can run continuously without breaking the bank

KaiheAiBox is built for this future. As Agent planning algorithms mature, having a 7×24 stable hardware platform—with both local and cloud compute—will be the differentiator. The "Agent Computer" is not just a device; it's the infrastructure for the next paradigm of AI.

VII.3 The Convergence Hypothesis

Here's a prediction: by 2027, the best AI Agents will combine: - LLM-scale language understanding (current generation) - AlphaGo-scale planning and search (next generation) - Real-world tool use and feedback (already emerging)

When this convergence happens, the "Agent Computer" won't be a niche product—it'll be as essential as a laptop. And the hardware that's optimized for sustained, multi-step agent workloads will have a massive advantage over general-purpose computing devices.

VIII. Conclusion: Two Paradigms, One Future

AlphaGo and LLMs represent two different paradigms of AI: search-based and generation-based. AlphaGo searches exhaustively within well-defined rules; LLMs generate fluently within open-ended domains. Neither alone is sufficient for truly intelligent behavior.

The future belongs to systems that combine both: the depth of search with the breadth of generation, the rigor of AlphaGo with the flexibility of LLMs. When that synthesis arrives, it won't just beat humans at games or write convincing text—it will plan, reason, and act with a level of competence that transforms how we work and live.

The hardware that runs these systems will need to be equally versatile: powerful enough for local inference, connected enough for cloud augmentation, and reliable enough for 24/7 operation. That's the Agent Computer, and that's what KaiheAiBox is building toward.

IX. Practical Takeaways for AI Practitioners

For those building AI systems today, the AlphaGo vs. LLM comparison offers concrete guidance:

Use search when the problem space is bounded and well-defined (games, scheduling, route planning)
Use generation when the problem space is open-ended and creative (writing, conversation, design)
Use both when you need reliability in uncertain environments (Agent planning, autonomous systems)
Invest in hardware that supports hybrid architectures — local compute for search loops, cloud APIs for generation quality

The hardware that best serves this hybrid future is the Agent Computer: a device designed for sustained operation with both local and cloud capabilities. This is precisely what KaiheAiBox delivers.

The question is no longer whether search and generation will converge, but when—and the hardware we build today determines whether we're ready when they do. KaiheAiBox's Agent Computer platform is designed for exactly this convergence, providing the reliable infrastructure that next-generation AI systems require.

KaiheAiBox · AI Frontier

VI. Monte Carlo Tree Search in LLMs: Latest Attempts

AlphaGo's success relied heavily on combining Monte Carlo Tree Search (MCTS) with deep neural networks. Researchers naturally wondered: can we apply MCTS to LLMs too?

VI.1 Why MCTS Is Attractive for LLMs

When an LLM generates text, it samples token by token—essentially finding a path in a gigantic search tree. MCTS provides a systematic way to explore this tree:

Selection: Starting from the root, select child nodes according to the UCB formula
Expansion: Upon reaching a leaf node, expand one new child
Simulation: From the new node, roll out to the end of generation
Backpropagation: Update all ancestor nodes with the simulation result

If MCTS could be introduced into LLM token generation, the generation quality would theoretically improve significantly—because the model would "think" about multiple possibilities before choosing the optimal path.

VI.2 Actual Attempts and Challenges

Multiple research teams tried combining MCTS with LLMs in 2024-2025:

DeepMind's AlphaCode: Used MCTS to assist code generation, reaching human-median performance in competitive programming
Anthropic's Constitutional AI: Used search methods during RLHF to optimize response quality
Several startups: Claimed "search-augmented generation" improves accuracy

But reality is harsh. MCTS requires a large number of simulations to converge, and each LLM "simulation" requires a forward pass—extremely computationally expensive. When AlphaGo plays Go, simulating one game is cheap (simple rules); but generating one complete LLM response costs 100-1000× more.

MCTS + LLM is an elegant idea, but until computational costs come down, it remains mostly in papers.

VI.3 Engineering Compromises

Since full MCTS is too expensive, several engineering compromises have emerged:

Beam Search: Only keep top-k paths; don't roll out to completion. Low computational cost, but prone to local optima.
Best-of-N Sampling: Generate N candidate responses, use a reward model to pick the best. Expensive but simple and effective.
Process Reward Models (PRM): Evaluate not just the final answer, but also intermediate reasoning steps. This is exactly AlphaGo's "value network" idea applied to LLMs.

These approaches are all cheaper than full MCTS, but still 2-10× more expensive than pure sampling. For real-time conversation scenarios, the cost is still too high.

VI.4 Reinforcement Learning from Human Feedback: The Closer Cousin

Rather than MCTS, the technique that has actually moved the needle for LLMs is RLHF (Reinforcement Learning from Human Feedback)—and its newer variant, RLAIF (AI Feedback).

RLHF doesn't do tree search. Instead, it: 1. Collects human preferences on pairs of responses 2. Trains a reward model to predict human preferences 3. Uses PPO (Proximal Policy Optimization) to fine-tune the LLM against the reward model

This is more like AlphaGo's "policy network" training than its MCTS search. And it works—RLHF is responsible for the helpfulness and safety improvements in GPT-4, Claude, and Gemini.

The connection: AlphaGo's policy network was trained by self-play (MCTS generating training data). LLMs' policy is trained by human feedback (humans generating training data). Different data sources, same idea: use a weaker signal (game outcomes / human preferences) to train a stronger policy.

VI.5 Search Algorithm Taxonomy for AI Systems

To really understand the landscape, let's taxonomize search algorithms in AI:

Algorithm	Used In	Strength	Weakness
Minimax + Alpha-Beta	Chess engines (Stockfish)	Optimal play against perfect opponent	Requires full game tree; intractable for large spaces
MCTS	AlphaGo, AlphaZero	Handles large search spaces; no need for domain heuristics	Computationally expensive; slow convergence
Beam Search	LLM generation (default)	Fast; good enough for fluent text	Greedy; misses global optimality
Best-of-N	RLHF fine-tuning	Simple; leverages reward model	N× cost for N samples
Guided Beam Search	MedPaLM, GPT-4 with CoT	Improves reasoning	Still fundamentally greedy
MCTS + LLM (research)	AlphaCode, some math reasoning	Highest quality	Prohibitively expensive

AlphaGo sits in the MCTS row. LLMs sit mostly in the Beam Search and Best-of-N rows. The quest to move LLMs to the MCTS row is ongoing—but the computational wall is real.

VI.6 Implications for AI Agent Architectures