Research Indicates Agents Can Drastically Alter Strategy Even with Identical Inputs, Directly Impairing Their Performance Accuracy
Artificial Intelligence
4
04 Mar 12:47
Modern agents built upon Large Language Models (LLMs), capable of executing complex tasks utilizing tools and sequential reasoning, are increasingly integrated into practical applications. Nevertheless, the degree to which their behavior remains consistent given the exact same input data remains insufficiently examined. This aspect is crucial for the debugging, evaluation, and overall dependability of such systems.
In an effort to address this, a systematic investigation into the behavioral consistency of agents was conducted across three distinct models (Llama 3.1 70B, GPT-4o, Claude Sonnet 4.5) using the HotpotQA dataset. This collection of multiple-choice questions was compiled by a team of natural language processing researchers hailing from Carnegie Mellon University, Stanford University, and the Université de Montréal.
Across 3,000 experimental runs (100 tasks × 10 iterations × 3 models), it was discovered that agents frequently select different action sequences even when presented with identical starting conditions. On average, between 2.0 and 4.2 unique execution paths were logged per 10 trials, with the variance in the number of steps reaching as high as 55%.
The primary finding reveals a direct link: high behavioral consistency (≤2 unique paths) corresponds to an accuracy rate of 80–92%, whereas high variability (≥6 paths) sees accuracy plummet to 25–60%. Furthermore, 69% of these divergences manifest as early as the second step—the very first query made by the agent. Trajectory length also correlates with accuracy: short sequences (3 steps) yield 90% correct answers, while long ones (8+ steps) only achieve 43%.
To boost consistency, the authors suggest reducing the generation temperature (from 0.7 down to 0.0), which curtails the number of unique paths and boosts accuracy by 5.4 percentage points. The researchers advise monitoring behavioral consistency in real-time: if an agent starts diverging early on, the final output is highly likely to be erroneous. This opens the door for automated quality control procedures and the selection of more robust models.
The work underscores that even with a minimal toolkit, agents exhibit considerable fluctuation, a problem that only intensifies as task complexity grows. For real-world deployment, it is paramount not only to push accuracy higher but also to guarantee behavioral stability—this will be the key to developing more reliable and predictable systems.
About the Author
