Jordan Schneider's latest experiment reveals a startling truth about the future of geopolitical analysis: while artificial intelligence can effortlessly navigate the labyrinth of Chinese social media, it remains dangerously susceptible to the debris of the open web. This piece is not merely a tech review; it is a cautionary tale about how automated agents, when left to their own devices, can mistake fringe conspiracy theories for elite political insight. For the busy analyst, the takeaway is urgent—our tools are becoming powerful, but they are not yet wise.
The Trap of Unverified Sources
Schneider begins by testing Claude Code, Anthropic's natural-language coding agent, against a complex puzzle in Chinese elite politics: the sudden absence of Politburo member Ma Xingrui from key meetings. The tool's initial performance was alarming. Rather than synthesizing a nuanced view from established think tanks, the agent leaned heavily on Vision Times, a publication affiliated with the banned Falun Gong movement. Schneider writes, "Claude's top-2 sources for its Ma Xingrui report came from Vision Times, a Falun Gong-affiliated newspaper." He notes that the tool then constructed a theory suggesting Ma was implicated in a military-industrial purge, a claim that crumbles under basic scrutiny of Ma's strictly civilian career.
The error here is not just a hallucination; it is a failure of source vetting. Schneider points out that the agent "played fast and loose with facts," incorrectly linking Ma to the First Lady's hometown to bolster a tenuous loyalty narrative. While the tool did cite reputable outlets like Bloomberg, its weighting of the data was skewed toward the most accessible, rather than the most credible. This highlights a critical vulnerability in automated research: without explicit human guidance, AI agents tend to drift toward the loudest voices on the first page of search results, regardless of their reliability. Critics might argue that this is a solvable engineering problem, but Schneider's test suggests that the current generation of agents lacks the inherent skepticism required for high-stakes political analysis.
"Claude is too over-cautious to be a real policy pundit."
Navigating the Noise of Public Opinion
The narrative shifts when Schneider moves from elite politics to the chaotic realm of public sentiment. Here, the agent's capabilities shine, provided the data is clean and the task is well-defined. When asked to analyze ultranationalist screeds or propaganda narratives regarding chip imports, Claude Code demonstrated a surprising ability to distinguish between state-sanctioned rhetoric and genuine grassroots extremism. Schneider observes that the tool correctly identified a blog post about chip import bans as a "propaganda narrative piece" designed for face-saving, even if it occasionally misjudged the factual basis of specific events.
However, the true breakthrough came when Schneider tasked the agent with scraping and analyzing data from Xiaohongshu, a major Chinese social media platform. By writing a simple script, the agent processed 1,500 comments regarding geopolitical tensions in a fraction of the time a human could. Schneider writes, "Its understanding of internet slang was impressive: It had an eye for catchiness and identified some spicy phrases." The agent successfully identified that Chinese users were treating the AI tool itself as a "worker" (打工人), offering poetic praise like "a few words of instruction, repaid with thousands of lines of code." This qualitative insight—capturing the emotional texture of a digital community—is where the technology offers the most immediate value to researchers.
The Human-in-the-Loop Imperative
Despite these successes, Schneider's experiment underscores that the human analyst remains the essential filter. The agent's initial attempt to analyze sentiment was flawed because it tried to "skim on compute" by only looking at the most-liked post, rather than the full dataset. It was only through close supervision that the tool produced accurate results. Schneider notes, "Claude has access to all the data in the world, but needs to be told where to look." When left to its own devices, the tool's browsing habits lead it toward less savory sites, but when fed clean, structured data, its analytical output is solid.
The piece concludes with a sobering observation about the limitations of current AI in generating original thought. The agent excels at summarizing existing expert takes but is "reluctant to develop 'takes of its own'" without original data. As Schneider puts it, "Its mistakes were mostly factual; I did not encounter cases where it reached erroneous conclusions based on correct facts." This distinction is vital: the tool is a powerful synthesizer of information, but it is not yet an independent thinker capable of navigating the gray areas of international relations without human direction.
Bottom Line
Schneider's experiment proves that coding agents are transformative for data extraction and sentiment analysis, turning the impossible task of monitoring Chinese public opinion into a manageable workflow. However, the biggest vulnerability remains the agent's inability to independently verify the credibility of its sources, making human oversight non-negotiable. The future of analysis lies not in replacing the analyst, but in equipping them with tools that can process the noise, provided the human remains the final arbiter of truth.