AI under supervision: why it can’t replace historians

American journalist Matt Novak tested how well modern chatbots handle historical research. The experiment was prompted by a Microsoft study suggesting that the historian profession is among those most likely to be transformed by AI.

Novak asked several bots — GPT-5, Microsoft Copilot, Google Gemini, Perplexity, and Grok — about the movies watched by U.S. presidents, a topic he has researched since 2012. He compared their answers with official White House screening records.

The results showed frequent errors and guesswork. For instance, Copilot and Perplexity wrongly claimed that Dwight Eisenhower watched "Suddenly" on August 11, 1954, while records show he actually saw "River of No Return". Similar mistakes appeared in responses about Nixon, Reagan, Clinton, and others.

According to Novak, AI can't replace the historian’s work, which involves analyzing archives, uncovering rare sources, and interviewing witnesses. While AI can be a helpful tool, it still requires human oversight — especially when accuracy is critical.