Key contributions
As a core developer for the I.R.I.S. project, my role is pivotal in integrating and enhancing the robot's conversational capabilities, autonomy, and ability to interact meaningfully with older adults. Here’s a detailed breakdown of my contributions:
Python Frameworks & ROS IntegrationDeveloped a Python-based software framework to bridge the high-level conversational AI logic with low-level robotic controls provided by the Robot Operating System (ROS). Engineered custom Python-ROS nodes to enable real-time inter-process communication, thus coordinating complex sequences such as user interaction with robotic responses and movements. Integrated sensor data management, speech recognition, and transcription using Whisper API (speech-to-text service) to feed environmental and user interaction data into the conversational AI logic for context-aware responses.
Natural Language Understanding and GenerationUtilized GPT-3 and GPT-4 Large Language Model, fine-tuning it with a rich dataset comprising dialogues and interactions specific to older adults' well-being, resulting in tailored, context-aware responses. Combing this conditional prompting based on the conversational flow for particular activities. Incorporated Natural Language Understanding (NLU) processes to discern user intent and sentiment, enabling I.R.I.S. to respond empathetically and keep the conversation relevant and engaging.CI/CD Pipelines and Data Flow ArchitectureOrchestrated Continuous Integration/Continuous Deployment (CI/CD) pipelines to automate the testing and deployment of new conversational models and software updates, ensuring the system evolves with user feedback.Designed a robust data flow architecture that allows for seamless data transfer between the robot's subsystems, maintaining the integrity and security of user data.System Design and Conversational AI
Developed the system architecture focusing on modularity and scalability, ensuring that individual components like the NLU module or the dialogue management system can be upgraded without disrupting the entire system architecture.Created activity-context models within the conversational AI to recognize and adapt to the user's current activity, ensuring that dialogue and robot actions are appropriate and timely.LLM Fine-Tuning and Conversational Context
Refined the Large Language Models (LLMs) by incorporating domain-specific knowledge, enabling nuanced conversations about health, daily activities, and personal interests.Implemented a context management system that retains information over the course of a conversation, allowing the robot to make callbacks to previous dialogues for a more natural and human-like interaction experience.
System Evaluation
Conducted comprehensive user experience evaluations with older adults, caregivers, and dementia care staff. Utilized surveys and observational, qualitative, and quantitative methods to assess comfort, enjoyment, and willingness to engage with the robotic system. Contributed to conducting monthly co-design workshops with 12-15 older adult panelist members and their caregivers, who are the main contributors to the design process.