Enterprise AI Research Analysis

The Challenge: Structural Blindness in LLM Agents

Modern LLM agents excel at complex tasks but often struggle with long-horizon, sparse-reward environments due to structural blindness. This means they fail to perceive and leverage the underlying topology of their environment, leading to inefficient exploration, imprecise credit assignment, and myopic planning. Standard group-based RL methods, while addressing reward sparsity, do not account for environmental structure, causing agents to get stuck in loops or miss critical pathways. Our research introduces GEPO to directly confront these limitations.

GEPO: Graph-Enhanced Policy Optimization Framework

GEPO dynamically constructs a state-transition graph from agent experience. By leveraging graph-theoretic centrality metrics (e.g., betweenness centrality), it extracts three synergistic learning signals:

• Structured Intrinsic Rewards: Guide exploration towards high-impact states.
• Graph-Enhanced Advantage Function: Enables topology-aware credit assignment.
• Dynamic Discount Factor: Adapts to each state's strategic value, allowing farsighted planning from critical junctures.

This approach transforms sparse-reward problems into dense, informative learning signals without requiring complex GNN architectures or static graph definitions, making it scalable and effective for dynamic textual environments.

Empirical Validation & Robust Performance

GEPO demonstrates strong performance across challenging benchmarks: ALFWorld (multi-room navigation), WebShop (e-commerce browsing), and a proprietary Workbench (procedural workflows). It achieves absolute success rate gains of +4.1% (ALFWorld), +5.3% (WebShop), and a significant +10.9% (Workbench) over competitive baselines. These results highlight that explicitly modeling environmental structure is a robust and generalizable strategy for advancing LLM agent training, especially in tasks requiring precise logical ordering and navigation through critical bottlenecks.

Ablation Study: Synergy of Core Components

Our comprehensive ablation study reveals that each of GEPO's core components—Structured Intrinsic Reward, Topology-Biased Advantage Aggregation, and Dynamic Discount Factor—makes a distinct and necessary contribution. Removing any single component leads to a consistent performance degradation (1-3%). More importantly, pairwise ablations show a strong synergistic and super-additive effect, where combined removals cause significantly larger performance drops (up to double the sum of individual impacts). This validates GEPO's design as a cohesive system, not merely a parallel collection of features, essential for efficiently navigating sparse reward environments.

Further analysis of centrality measures showed betweenness centrality consistently yields the strongest performance across model scales, effectively identifying critical bottleneck states that are crucial for long-horizon tasks.

Conclusion & Future Directions

GEPO successfully mitigates structural unawareness in LLM agents by injecting topological priors into intrinsic rewards, advantage estimation, and the discount factor. The framework's ability to identify and prioritize high-impact bottleneck states is key to guiding exploration and credit assignment in complex tasks.

Future work includes scaling to even larger state spaces with approximate centrality computations, extending to multi-modal or real-world environments, and investigating more sophisticated graph metrics like community detection for highly interconnected subgraphs. These explorations will broaden the scope of LLM-based agents and inspire further innovations in bringing structural priors into RL for real-world applications.