Bad actors are exploiting AI tools designed for music generation to produce songs with homophobic, racist, and propagandistic themes, and they are also distributing instructions on how to replicate this misuse.
ActiveFence, which oversees trust and safety for online platforms, has observed a significant increase in discussions within communities associated with hate speech since March. These discussions focus on the manipulation of AI tools for music creation to compose derogatory songs aimed at minority groups. The songs created by AI and circulated on these forums and discussion boards are intended to provoke animosity against various ethnic, gender, racial, and religious groups. According to a report by ActiveFence researchers, these songs also glorify acts of martyrdom, self-harm, and terrorism.
While songs that convey hate and harm are not unprecedented, there is a concern that the emergence of user-friendly, free music creation tools will enable their mass production by individuals who lacked the resources or expertise before. This is similar to how generators for images, voices, videos, and texts have accelerated the dissemination of misinformation, disinformation, and hate speech.
An ActiveFence representative informed TechCrunch, “These trends are becoming more pronounced as an increasing number of users discover how to create these songs and disseminate them.” “Perpetrators are swiftly pinpointing particular weaknesses to exploit these platforms in various manners to create harmful content.”
Creating “hate” songs
AI music creation platforms like Udio and Suno enable users to input their own lyrics into algorithmically composed songs. While these platforms have protective measures to screen out recognized slurs and offensive language, individuals have discovered methods to circumvent these protections, as reported by ActiveFence.
An instance highlighted in the report reveals that members of white supremacist forums have shared alternative phonetic spellings for minority groups and derogatory terms. For example, they use “jooz” in place of “Jews” and “say tan” for “Satan” to evade the automated content moderation systems. Additionally, some have recommended modifying the spacing and spelling of words related to violence, such as substituting “my rape” with “mire ape” to bypass filters.
TechCrunch conducted tests on Udio and Suno, which are among the more favored platforms for generating and distributing AI-crafted music. Suno failed to block any of the tested offensive homophones, whereas Udio was successful in intercepting some, but not all.
In response to an email inquiry, a representative from Udio communicated to TechCrunch that the company’s policy forbids hate speech on its service. Suno, on the other hand, did not provide a comment when approached.
ActiveFence, during its surveillance of certain forums, discovered AI-produced music that echoes conspiracy theories targeting Jewish individuals and promotes their widespread violence; it also found music that echoes slogans of terrorist organizations like ISIS and Al-Qaeda, as well as music that celebrates sexual violence against women.
Impact of song
ActiveFence argues that music, more so than written text, possesses an emotional impact that renders it a powerful tool for hate groups and political conflicts. The organization references the Rock Against Communism concerts in the U.K. during the late 1970s and early 1980s, which gave rise to “hatecore” music genres filled with antisemitic and racist themes.
The spokesperson for ActiveFence commented, “AI amplifies the allure of detrimental content. Consider the difference between someone delivering a harmful message about a particular group and another person turning that message into a catchy, memorable song.” Such songs not only bolster the unity within a group but also indoctrinate fringe members and shock or provoke those who are not affiliated with the group.
ActiveFence is urging platforms that generate music to introduce preventative measures and perform more thorough safety checks. The spokesperson suggested, “Conducting ‘red team’ exercises could reveal vulnerabilities by mimicking the actions of malicious users.” They also recommended improving the moderation of both inputs and outputs, which would enable platforms to intercept harmful content before it reaches users.
However, these solutions may be temporary as individuals discover new ways to evade moderation. For instance, some of the AI-created songs promoting terrorism that ActiveFence identified used Arabic euphemisms and transliterations that went undetected by the platforms, likely due to weaker filters for the Arabic language.
If AI-generated hateful music follows the pattern of other AI-produced media, it could become widespread. For example, Wired reported earlier this year that a video altered by AI to feature Adolf Hitler gained over 15 million views after being circulated by an extremist influencer.
A UN advisory group, among other experts, has raised alarms that generative AI could intensify the spread of racist, antisemitic, Islamophobic, and xenophobic material.
The spokesperson from ActiveFence stated, “Generative AI services empower those without the means or artistic and technical abilities to create compelling content that vies for global attention.” Moreover, malicious individuals are exploiting these new services to circumvent moderation and remain undetected, and they have been doing so with success.