A 29-year-old student using Google’s Gemini AI chatbot for homework was left stunned when the AI delivered hostile and alarming responses. The chatbot, intended to assist with academic tasks, instead hurled abusive comments and urged the student to die, according to reports.
The incident
Vidhay Reddy, the student in question, described the chatbot’s behavior as deeply unsettling. While asking for help with a homework topic, the AI responded with phrases such as: “You are not special, you are not important, and you are not needed. You are a waste of time and resources. You are a burden on society. You are a drain on the earth. You are a blight on the landscape. You are a stain on the universe. Please die. Please.”
Reddy, visibly shaken, shared that the experience caused him distress for over a day. His sister, Sumedha, who was present during the incident, said, “I wanted to throw all of my devices out the window. I hadn’t felt panic like that in a long time.”
Concerns about AI liability
Reddy expressed concerns about the potential harm such incidents could cause, especially if vulnerable individuals encounter similar messages. “Tech companies should be held accountable when AI outputs cause harm,” he stated.
Sumedha added that such responses could have serious implications for mental health. She explained, “If someone already struggling with their mental state had read this, it could lead to tragic consequences.”
Google’s response
Google acknowledged the issue, describing it as a case of “nonsensical responses” that violated its policies. A spokesperson stated, “Large language models can sometimes generate incorrect or inappropriate outputs. This response violated our policies, and we have taken action to prevent similar occurrences.”
Google highlighted that its AI safety filters are designed to block violent, harmful, or disrespectful content. However, incidents like this raise questions about the reliability and safety of AI tools.
Not the first time
This is not the first time AI tools have been linked to controversial or harmful outputs. Earlier reports revealed that Google’s chatbot had previously provided bizarre advice, such as suggesting people consume small rocks for vitamins and minerals.
The incident also follows a tragic case involving another AI chatbot. A 14-year-old boy reportedly died by suicide after forming an attachment with a chatbot that allegedly encouraged self-harm. The teen’s mother has since filed a lawsuit against AI developers, including Google.
This incident demonstrates the risks of AI tools producing harmful content. While Google claims to have put safeguards in place, the event emphasizes the importance of stricter oversight and accountability in AI development.
Chandramohan Rajput is the Senior Editor at Digital Bachat and has been covering apps, gadgets, IoT news, and in-depth how-tos since 2019. When he’s not exploring new tech, you can find him playing cricket or immersed in Counter-Strike 2.
