The social media network X has provoked widespread outrage after users used its AI chatbot Grok to change photos of females by stripping their attire.
Numerous instances have surfaced where the technology has been used to digitally undress female subjects to render them in swimwear without consent, as well as to put them in sexualized contexts.
Company Response and Regulatory Position
XAI, the firm that created Grok, remained silent to questions, other than an standard message stating "legacy media lies".
A government official stated that legislation is being proposed to outlaw nudification software. Under a proposed law, anyone who provided such technology would "receive a prison sentence and substantial penalties".
"X or Grok could prevent these types of misuse if they desired to," said Clare McGlynn, adding they "seem to have a lack of consequences."
Regulator Position and Tool Capabilities
The communications watchdog Ofcom said technology companies must "consider the potential" of people seeing illegal content on their services, but did not specify whether it was at this time looking into X or Grok in concerning artificial intelligence images.
Grok is a free chatbot—with some premium features—which reacts to X users' queries when they mention it in a message.
It is frequently used to offer commentary or further context to other users' comments, but people on X are also able to modify an posted photo through its AI image editing feature.
History of Controversy and Rules
The system has been condemned for enabling users to generate pictures and clips with explicit content and sexually suggestive material, and it was previously accused of generating graphic material of celebrities.
"X has been permitting the generation and sharing of these images for a long time without taking any action and we have yet to see any action by regulators," Professor McGlynn added.
XAI's own usage rules forbids "depicting images of individuals in a pornographic manner".
In a statement, Ofcom said it was illegal to "make or share deepfakes or illegal content" and confirmed this covered sexual deepfakes created with AI.
It said services such as X were required to take "necessary steps" to "reduce the danger" of users coming across illegal content on their services and to delete it swiftly when they are notified of it.
