The platforms assessed were the free-access versions of ChatGPT, Claude, Copilot, DeepSeek, Gemini, Meta AI and Perplexity. Despite receiving the same questions, the tools produced what the authors describe as «substantial variation in guidance» — particularly around emergency savings and asset allocation.
While the platforms generally aligned on broad principles — such as the widely cited 4 percent retirement withdrawal rule — the differences in specific recommendations were significant enough to raise serious concerns about reliability. «Although the tools often produced recommendations that broadly aligned with generic financial planning principles, such as the 4 percent retirement withdrawal rule, there were significant differences across platforms in suggested emergency savings and portfolio allocations,» the authors wrote.
Confident-sounding answers that can still be incomplete, misleading or wrong
One of the sharpest warnings in the study concerns the tone AI tools adopt when answering financial questions. «GenAI-driven responses may sound confident but can still be incomplete, misleading, or incorrect,» the paper states — a combination that researchers argue is particularly dangerous for users who may not have the background to detect errors.
Andrew Lo, director of MIT’s Laboratory for Financial Engineering and principal investigator at its Computer Science and Artificial Intelligence Lab, put it plainly in an interview with CNBC in March. «One of the things about LLMs that I find particularly concerning is that no matter what you ask it, it’ll always come back with an answer that sounds authoritative, even if it’s not,» he said.
Lo identified the highest-risk use case: personalized calculations. «When it comes to very, very specific calculations of your own personal situation, that’s where you have to be very, very careful,» he said. This limitation is compounded by the well-documented problem of algorithmic «hallucination,» where AI models generate plausible-sounding but factually incorrect outputs. AI tools are also sensitive to how prompts are phrased, meaning small differences in how a user words a question can lead to meaningfully different — and potentially conflicting — recommendations.
Demographic bias: recommendations shifted when race and gender changed
Beyond accuracy, the study uncovered a fairness problem. Researchers ran the same financial scenarios a second time, but altered the race and gender of the hypothetical individual in the prompt. The AI platforms’ recommendations changed in response — a finding the authors say raises questions «about the consistency and fairness of GenAI-driven recommendations.»
The study’s authors — Swarn Chatterjee, Brenda Cude and Gianni Nicolini — describe these outputs as «suboptimal» or biased, arguing they could lead different users toward different financial planning outcomes based solely on demographic characteristics rather than their actual financial situation.
This dimension of the research adds a layer of concern beyond simple inaccuracy. If an AI tool systematically provides different investment portfolio or retirement savings guidance depending on a user’s perceived identity, the implications for financial equity are significant — particularly given how many Americans now use these tools as a primary resource for money decisions.
