An interactive voice-based chatbot developed by researchers at the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S), the ARC Centre of Excellence for the Digital Child and the QUT Gen AI Lab invites children to explore what happens when AI chatbots agree with their ideas and helps them consider the risks when artificial intelligence prioritises agreement over accuracy.
The chatbot invites users to compare two different chatbot behaviours by switching between an “always agree” mode and a mode designed to respond more carefully and thoughtfully.
The experience explores ‘AI sycophancy’, a known problem in which AI systems flatter or agree with users, prioritising likeability over accuracy, critical thinking and sound judgement.
The interactive experience is based on the voice-based chatbot developed for research with children, extending on the pilot project Making AI Friends? Enhancing Primary Years AI Literacy with a Voice AI Chatbot Experience, which explored how children can develop critical capabilities for understanding AI.
Dr Henry Fraser, a QUT researcher and ADM+S Associate Investigator, says AI safety and governance needs bottom-up approaches, alongside top-down regulation and enforcement.
“Children’s critical AI capabilities are an essential part of AI safety and governance,” Fraser says.
“Regulation, like Australia’s proposed digital duty of care, is important, but it can’t fix everything.
“What we can do is enhance children’s preparedness for and resilience to AI risks in a way that takes their experiences and perspectives seriously.”
The project comes amid growing attention on how young people use AI for personal advice and support.
Recent reporting on research involving Adelaide University and the Australian Federal Police found that 35 per cent of children who had experienced online sexual victimisation either turned to an AI chatbot for help or told no one.
The findings underline the importance of helping young people to understand the limitations of AI systems and recognise when they need support from a trusted person.
Dr Henry Fraser and his research team aim to go beyond addressing potential AI harms and safety risks in isolation and to “leave a space to imagine better forms of ‘children’s AI’ alongside a better ‘children’s internet’”.
Voice interfaces are increasingly appearing in toys and other smart devices used by children and families, making AI literacy an important part of understanding the digital environments children encounter.
The interactive experience encourages users to reflect on the differences between chatbot responses, why a system might be designed to agree with its users, and whether agreement should be mistaken for trustworthy advice.
The project reflects the importance of involving children in conversations about the technologies increasingly shaping their lives. By exploring different chatbot behaviours and discussing what better responses might look like, children can contribute their perspectives to how AI is understood and designed.
The research underpinning the interactive chatbot will be presented at the Association of Internet Researchers (AoIR) Annual Conference in Mexico in October, contributing to international discussions about children’s experiences with AI and the importance of developing AI literacy from an early age.
The researchers hope this work will support conversations among children, families and educators about both the possibilities and limitations of conversational AI, including the importance of questioning responses, recognising when AI may be wrong, and seeking help from a trusted person when needed.
There are trade-offs involved in participatory approaches to exploring the risks of AI with children, the researchers acknowledged in their report.
“In navigating these issues, the pilot deliberately positioned children as autonomous and competent participants in the research process, inviting them to position themselves as “expert informants” – in this case, on their understanding of AI risks, and on the best potential ways to engage their peers.
“Our preliminary observations suggest that children are indeed capable of taking on these roles, and can speak thoughtfully about issues of authenticity, trust and bias in digital interactions,” the researchers state in the report.
This indicates, they say, that there are opportunities to undertake similar activities focusing on other AI chatbot features and problems that have social and technical dimensions.
“These might include misinformation, disinformation, deepfakes and data privacy; as well as the capacity of chatbots to acknowledge uncertainty or appropriately communicate with users about the degree to which they should rely on outputs.”
As the project builds on the foundation of the pilot the research team plans to include children earlier and more fully in the co-design process of artefacts and participatory experiences.
“In doing so, we aim to go beyond addressing potential harms and safety risks in isolation and to leave a space to imagine better forms of ‘children’s AI’ alongside a better ‘children’s internet’,” the report concluded.
To try the interactive voice-enabled chatbot and spot the sycophant on the GenAI Arcade website, click here. (The chatbot is intended for supervised educational use. Children should ask a trusted adult before using it and should not follow chatbot advice without checking with a trusted adult.)
To read the research ‘Should I hide my mum’s slippers? A design-based pilot study of children’s critical engagement with AI sycophancy using a custom AI chatbot’ click here.
