Reinforcement Learning for LLMs

Impact of Reward Function Design & Decay Factor (γ)

Ablation studies confirm the critical role of PORTool's step-wise reward formulation, especially the decay factor γ. An optimal γ (found to be 0.95) effectively balances trajectory efficiency and correctness with formatting precision, leading to superior training and tool-use behaviors. Incorrect settings, such as γ=0 (disregarding correctness) or γ=1 (failing to discriminate efficient steps), lead to degraded performance.

Furthermore, the adaptive aggregation function G(.) (max/avg) and the specific weighting of trajectory-relative (w1) and fork-relative (w2) advantages (derived from Theorem 3.1) are crucial. PORTool's combined approach (w1=1, w2 scaled) significantly outperforms methods relying solely on one advantage type or using equal weighting, which can introduce objective inconsistency.

Frequently Asked Questions

How does PORTool differ from existing RL methods for tool-use LLMs?

Unlike methods that rely on static datasets or assign uniform rewards across trajectories, PORTool employs a tree-like rollout structure to explore diverse solution paths. It assigns step-wise rewards that consider both final answer correctness and formatting compliance, calculating unique fork-relative and trajectory-relative advantages to guide more efficient policy optimization.

What kind of tools can PORTool integrate?

PORTool integrates a comprehensive suite of 17 executable tools, including real-time tools (e.g., weather_search, news_search), factual tools (e.g., math_calculation, knowledge_search), and hybrid tools (e.g., conversion_calculation). This allows it to handle a wide range of time-sensitive and time-invariant queries.

Why is step-wise reward important in PORTool?

Step-wise rewards in PORTool quantify the contribution of each individual step to the final outcome. This granular feedback allows the LLM to learn which intermediate steps are effective, even within trajectories that ultimately lead to incorrect answers, thereby fostering better exploration and the discovery of more robust tool-use strategies.

How does PORTool ensure proper formatting in tool calls?

PORTool's reward function includes a specific formatting reward component. This rubric assigns points for adhering to structural formats (e.g., <think> and <tool_call> blocks, valid JSON, required tool parameters) and penalizes errors. The formatting reward is carefully rescaled to ensure correctness remains the primary objective, but good formatting is strongly encouraged.

What are the practical benefits of using PORTool for LLM training?

Practical benefits include significantly higher accuracy in resolving user queries, increased efficiency due to fewer tool-call steps, and a reduced unanswerable rate. The trained models also exhibit stronger structural compliance and better error-correction capabilities, leading to more reliable and effective tool-integrated reasoning in dynamic, real-world scenarios.