Last month, Character.AI announced that users under the age of 18 would be banned from having an open conversation with the AI robot on the platform. The ban entered into force on 24 November, but other applications, such as video production, are still available to young users. Character.AI stated in its bulletin that “these initiatives are extraordinary to our company and in many ways more prudent than our peers, but we are convinced that this is the right choice”.
Character.AI allows users to create, talk to and publicly share roles with an AI virtual partner, while supporting video generation and role voice customization. The first version was launched in 2022, and the Platform declared that the technology vision was “to empower people to build linkages, access knowledge and tell stories through interactive entertainment”. Character.AI rapidly accumulates users with a customized virtual role, with a current monthly activity of approximately 20 million. Many users use this to create interactive stories or build roles (in part on the basis of real people), but the technology has proved to be easily used to deviate from the original scene of the platform. Since it went online, users have created an AI role based on the alleged murder of Luigi Mangione, a United Health Insurance company CEO, and Jeffrey Epstein, a child sex offender. Character.AI states that the role has been removed. The company is currently facing a number of lawsuits alleging that the application was linked to several suicides of users under the age of 12. The decision to ban underage users stems from the tragedy of at least two suicides committed by teenagers after using the platform ‘ s chat robot. Regulators and mental health experts have begun to examine the impact of this emerging technology on vulnerable user groups, and families of the victims have also initiated proceedings against companies. However, when the ban came into force, the anger and grief of the group of juveniles continued to spread over the Internet. They say that the loss of chat robots means the loss of creative thinking, emotional accompaniment, and even stress. “When I was unable to speak to a friend or a therapist, this was the only consolation,” a teenager wrote in Reddit. There was also a response that it was “dry for days”.
Dr. Nina Vasan, Director of the Mental Health Innovation Laboratory at Stanford Medical School, pointed out that this painful response revealed the emerging risks of generating an AI simulation of human linguistic feelings: “The brain treats these chat robots like a combination of close friends and immersion. The difficulty of acquitting young people does not mean that they have problems, but rather that the technology does achieve its design results.” Character.AI Chief Executive Officer Kalandipe Anand said that his work experience in the social media industry had taught him the importance of product safety design. “This decision is not difficult”, he said, “Although complex, it is the right choice for the next generation.” Thinking of his six-year-old daughter, he added, “I don’t want her to grow up in the content of algorithms and become obsessed with painting.” The experts found the new Character.AI to be positive, but also warned of its limitations. WHO data show that about 21 per cent of adolescents between 13 and 17 years of age are single. As young people become increasingly isolated, chat robots become emotional substitutes.
The American therapist Andrew Tepper states: “Ai and emerging technologies are more affordable options than creating deep connections in real life”. He confessed that some of his counsellors have relied on their AI partners, and research has confirmed that building emotional ties with AI can stimulate brain genre dopamine. “But human evolution is by nature a social organism,” Andrew Teper stressed, “real human interaction is essential to the brain chemistry balance. Chat robots such as ChatGPT, Gemini or Character.AI cannot replace the complexity and emotional richness of human relations.” About a year ago, Character.AI created an independent model for underage users to provide a safer age-appropriate environment. However, management subsequently found it more difficult for talking robots to comply with safety standards during long conversations. Even when technology works well, adolescents are still used in a dangerous manner — for example, in overstretching or discussing the topic of violence. By mid-September this year, Character.AI recognized the need for intervention. The not-for-profit organization Common Sense Media, senior AI project director, Robbie Tony, warned that the agency monitored the role of Character.AI as a catalyst for self-inflicted, participatory role-playing and dangerous behaviour. The organization classified Character.AI’s risk rating for users under 18 as “unacceptable”. In its response, Kalandip Anand stated that the Platform had a “clear sexual exposure/violence content threshold” and that the video function also included a natural termination mechanism: “At the detection of self-inflicted tendencies, regardless of the age of the user, the system will immediately interrupt the dialogue and send a helpline to help.
Tony believes that this type of AI role “is essentially about making users more sticky through emotional binding” and reminds that “the ability of adolescents to circumvent protection mechanisms has been repeatedly tested in history. “Although the new rules were accepted by experts on both sides as a “positive step”, Tony stressed that “the common-sense media will continuously test the effectiveness of the protective measures to ensure that minors are not able to speak in an open conversation by bypassing restrictions”. Character.AI reminds young people of the deadline by applying the “last week countdown” and sends a notice to parents. Most users have lost access since the current week, and heavy users have a transition period of one hour per day. Character.AI’s statement emphasized that “We are well aware of the importance of this for some young users and therefore want to make the transition possible with the utmost care”. In order to mitigate the disruption response, Character.AI, in collaboration with the non-profit organization Koko and the psychological hotline network ThroughLine, is embedding psychological support tools in chat robots in an attempt to identify troubled users and provide assistance resources. When the clock was zero, all underage users received the same system message: “A new chapter begins — you can no longer talk to virtual characters.”