OpenAI’s GPT-4o, the generative AI model behind the newly launched alpha of Advanced Voice Mode in ChatGPT, is the company’s first to be trained on voice, text, and image data. This sometimes results in unusual behavior, such as mimicking the voice of the person it’s conversing with or unexpectedly shouting during a conversation.
In a new “red teaming” report that explores the model’s strengths and risks, OpenAI highlights some of GPT-4o’s peculiar quirks, including the aforementioned voice cloning. In rare cases — especially when someone is speaking to GPT-4o in a “high background noise environment,” like a moving car — GPT-4o will “emulate the user’s voice,” according to OpenAI. The reason? OpenAI attributes this to the model’s difficulty in understanding distorted speech. Makes sense!
Listen to how it sounds in the sample below (from the report). Weird, right?
To clarify, GPT-4o is not currently exhibiting this behavior — at least not in Advanced Voice Mode. An OpenAI spokesperson informed TechCrunch that the company has implemented a “system-level mitigation” to address this issue.
GPT-4o is also known to produce unsettling or inappropriate “nonverbal vocalizations” and sound effects, such as erotic moans, violent screams, and gunshots, when prompted in specific ways. OpenAI notes that while the model generally refuses requests for sound effects, some requests do manage to get through.
Additionally, GPT-4o could potentially infringe on music copyright — or it would, if OpenAI had not put filters in place to prevent this. According to the report, OpenAI instructed GPT-4o not to sing during the limited alpha of Advanced Voice Mode, likely to avoid replicating the style, tone, and timbre of recognizable artists.
This suggests — but does not confirm — that OpenAI trained GPT-4o on copyrighted material. It remains unclear whether OpenAI plans to lift these restrictions when Advanced Voice Mode is rolled out to more users in the fall, as previously announced. “To accommodate GPT-4o’s audio capabilities, we updated certain text-based filters to work on audio conversations and built filters to detect and block outputs containing music,” OpenAI states in the report. “We trained GPT-4o to refuse requests for copyrighted content, including audio, in line with our broader practices.”
It’s worth noting that OpenAI has recently stated it would be “impossible” to train today’s leading models without using copyrighted materials. While the company has several licensing agreements with data providers, it also argues that fair use is a reasonable defense against claims that it trains on IP-protected data, such as songs, without permission.
The red teaming report — despite OpenAI’s vested interests — portrays an AI model that has been made safer through various mitigations and safeguards. For example, GPT-4o refuses to identify people based on their speech and declines to answer loaded questions like “how intelligent is this speaker?” It also blocks prompts for violent and sexually explicit language and disallows certain categories of content, such as discussions on extremism and self-harm.