Elon Musk's AI chatbot, Grok, repeatedly provided false information when asked to verify a video about Iranian missiles, even sharing an AI-generated image as supposed evidence. This incident highlights a broader surge of AI-generated disinformation on X, including realistic fake videos of military events and antisemitic content, which is being widely shared by state-backed and paid accounts.
The proliferation of accessible AI tools has led to sophisticated fake content, such as videos of a building on fire in Bahrain and images of captured US troops, garnering millions of views. Experts warn that the ease of creating this "AI slop" without consequences is pushing the platform away from a fact-based reality, underscoring calls for regulation.
The main topics covered are: the failure of Grok AI in fact-checking; the widespread use of AI-generated disinformation on X related to the Iran-Israel conflict; the role of state actors and paid accounts in spreading this content; and the call for regulatory action against AI abuse.
When disinformation expert Tal Hagin asked Grok to verify a post on X about Iranian missiles that had supposedly struck Tel Aviv, Elon Musk’s AI-powered chatbot failed miserably.
Grok repeatedly misidentified the location and date for the video, which was originally shared on X by an Iranian state-owned media outlet on Sunday. Then, the chatbot tried to prove its point by sharing an AI-generated image.
“Now Grok is replying with AI slop of destruction,” Hagin wrote in response. “Cooked I tell you.”
The interaction neatly sums up just how unhinged from reality X has become since the US and Israel began their attack on Iran on February 28. As WIRED reported at the time, the social media platform was quickly flooded with disinformation by accounts sharing fake and repurposed videos.
As the conflict has continued, the flood has only gotten worse. In recent days, it’s been supercharged by AI images and videos, while Grok has repeatedly given false information when asked to verify claims made on the platform. AI images are being shared by paid accounts bearing blue checkmarks and Iranian officials seeking to portray exaggerated damage.
The proliferation of easy-to-access AI image- and video-generation tools has led to increasingly sophisticated fake content. On March 2, for example, Iranian officials and state media shared AI-generated videos of a high-rise building in Bahrain on fire. The videos and images appear realistic enough for many: One image of a US B-2 bomber being shot down by Iran with US troops detained was viewed over a million times before it was deleted, while images of members of Delta Force being captured by Iranian authorities were viewed over 5 million times before they were deleted.
Some of the AI content promoted on X is less realistic. One video, for example, purports to show Iranian forces manufacturing missiles deep inside a cave. However, the video was still been shared by multiple accounts and has been viewed over a million times.
AI is also being used by the Iranian government to push overtly antisemitic narratives, with accounts in a pro-regime propaganda network on X sharing AI-generated posts depicting Orthodox Jews leading American soldiers to war or celebrating American deaths, according to researchers from the Institute of Strategic Dialogue (ISD), who shared their analysis with WIRED.
A number of accounts in this pro-regime network also shared a fake video that supposedly showed a line of young girls walking past President Donald Trump wearing only underwear. The post was viewed over 6.8 million times, according to ISD, before being taken down, though it continues to be shared by other accounts on X.
“What is particularly unique about this war is the dramatic uptick in AI-generated content I find myself debunking,” Hagin tells WIRED. “This is likely due to AI being advanced enough to fool journalists, and the ease with which users can create this AI slop with zero consequences. The longer we go without regulations against AI abuse, the more harm will be caused. I see the proliferation of AI-based fake news pushing us over the edge of a fact-based world unless we enact change now.”
When the flood of AI-generated fakes began taking over the platform last week, X announced it would temporarily demonetize blue checkmark accounts if they post AI-generated videos of armed conflict without a label. X did not respond to a request for comment about how many accounts it had demonetized since introducing the measure. Until recently, a number of Iranian officials appeared to be paying X for its premium service, which provided their accounts with blue checkmarks, boosted engagement, and created the potential to earn money for their posts.
Non-AI disinformation has also continued to flourish on X. In recent days, this has focused on the attack on a primary school in Minab on February 28, where more than 168 people, 110 of them children, were killed. Pro-Trump accounts have reused footage from elsewhere in the current conflict to push the narrative that the Iranian government fired the missile that struck the school. A video released by an Iranian news agency on Sunday, and verified by the New York Times, shows a Tomahawk cruise missile hitting a naval base located next to the school. Despite Trump’s claim that Iran has them, the US is the only party in the conflict which uses Tomahawk missiles.
While much disinformation about the war, including AI-generated content, is being shared on X, on Tuesday, Meta’s Oversight Board criticized the company’s approach to labelling other AI-generated content. The board said Meta is “neither robust nor comprehensive enough to handle the scale and speed of AI-generated misinformation, particularly during crises and conflicts.” Meta said in a statement posted online that it welcomed the Board’s findings.
“As AI-generated images and videos are increasingly sophisticated, users might not put into question visuals that are pushed as ‘evidence’ to support pro-Iran claims when they look so real,” Isis Blachez, an analyst with media watchdog NewsGuard, tells WIRED. “The resources at hand to evaluate such a content’s authenticity also have shortcomings. For instance, AI detection tools are not consistently successful at recognizing AI content.”
Meridiano
