A new study shows that artificial intelligence can accurately understand what smokers are trying to communicate in support group messages, potentially making digital smoking-cessation programs far more effective at keeping people engaged. Researchers at the University of California, Irvine developed an AI system that improved its ability to detect user intent from 38% accuracy to 90% by using specialized training techniques, offering a blueprint for how technology could enhance one of the biggest health challenges facing millions of Americans. Why Accurate Message Understanding Matters for Smoking Cessation? Smoking remains a leading preventable cause of death, yet quit rates stay stubbornly low despite the availability of proven cessation aids like nicotine replacement therapy (NRT) and support groups. Mobile health interventions—including text-based support groups and quit-smoking apps—have emerged as a promising way to overcome barriers like inconvenient meeting times and limited access to in-person groups. However, these digital programs face a critical challenge: they need to understand what users are actually asking for in order to provide helpful, timely responses. This is where chatbots come in. When a chatbot can accurately detect what a person means by their message—whether they're reporting a relapse, asking for motivation, seeking information about nicotine replacement therapy, or simply checking in—it can deliver personalized, evidence-based support without requiring a human moderator to be available 24/7. "Accurate intent detection is crucial for identifying user needs and delivering timely, appropriate chatbot responses," the researchers noted in their study. How Researchers Trained AI to Understand Smokers Better The research team used a large language model called Llama-3 8B (a publicly available AI system from Meta with 8 billion parameters) and trained it specifically on messages from a smoking-cessation support group. The challenge was significant: support group conversations are messy and unpredictable, involving multiple people, interruptions, incomplete thoughts, and 25 different types of user intents—far more complex than typical one-on-one chatbot conversations. The researchers applied three key techniques to improve the AI's performance: - Fine-tuning: Training the AI model on real messages from the smoking-cessation support group so it could learn the specific language and context of people trying to quit smoking. - Data downsampling: Reducing the number of common message types (like off-topic comments) so the AI would learn to recognize rarer but critically important messages, such as reports of relapse or overdose concerns. - Error correction: Having humans review cases where the AI disagreed with human annotators, identifying and fixing labeling mistakes in the training data to improve overall accuracy. The Results: A Dramatic Leap in AI Accuracy Without any specialized training, the AI system performed poorly, achieving only 38% accuracy at understanding user intent. After fine-tuning on the smoking-cessation data, accuracy jumped to 80%. When the researchers combined fine-tuning with data downsampling, the system reached 91% accuracy on the cleaned test dataset. The final method—combining fine-tuning, downsampling, and error correction—achieved 90% weighted accuracy, meaning the AI could reliably identify what users were trying to communicate across all 25 different intent categories. This is a substantial improvement that could translate into chatbots providing more relevant, timely support when smokers need it most. Why This Matters for Digital Health Interventions The implications extend beyond smoking cessation. The research demonstrates that large language models require domain-specific training to work effectively in health settings. Off-the-shelf AI systems, no matter how sophisticated, struggle with the unique language, context, and data imbalances found in real-world health applications. By showing how to overcome these challenges through careful training and data management, the study provides a roadmap for improving chatbots across many health conditions. Mobile health support groups offer considerable promise because active engagement in online communities leads to better outcomes, while low engagement is often linked to dropouts. A chatbot that truly understands what users are communicating can keep people engaged by responding with relevant, personalized support—potentially helping millions of smokers stay on track with their quit attempts. Steps to Improve AI-Powered Health Support Systems - Collect domain-specific data: Train AI systems on real messages from the specific health condition or population you're trying to help, rather than relying on general-purpose models. - Address data imbalance: Ensure the AI learns to recognize rare but critical messages (like overdose reports) by adjusting how common message types are represented in training data. - Validate with human experts: Have healthcare professionals and domain experts review AI predictions to catch errors and improve the training process over time. - Test in real-world conditions: Evaluate the AI system using actual support group conversations with their inherent messiness, interruptions, and incomplete thoughts. The research was published in the Journal of Medical Internet Research (JMIR) in March 2026 and represents a significant step forward in making digital health interventions smarter and more responsive to what people actually need.
