Key learnings
Real-time media transport with LiveKit rooms, agents, and low-latency data tracks
Trade-offs between cascaded STT/LLM/TTS pipelines and streaming speech-to-speech models
Mapping MediaPipe face landmarks and blendshapes onto physical actuators at 25 Hz
Designing and running a counterbalanced within-subjects user study with 41 participants
Structuring a multi-process Python system with typed, centralised configuration
Description
HRI LiveKit is a conversational pipeline for the android robot head at Stuttgart Media University's HRI lab. The existing system ran a cascade (speech-to-text, then a language model, then speech synthesis and lip-sync), which produced carefully authored facial expressions but left participants waiting six to twelve seconds for a reply. We rebuilt the conversation around a real-time speech-to-speech model to see whether responsiveness was worth the expressiveness we would lose.
The system is three processes sharing one LiveKit room. A conversational agent handles the dialogue with OpenAI's realtime model and drives a Simli avatar, which joins the room as a participant and publishes a video stream of a speaking virtual face. A movement extractor subscribes to that video, runs the MediaPipe Face Landmarker on every frame, and converts the resulting blendshapes, head pose, and gaze into 14-channel actuator frames published over a data track. A robot-side client bridges the room to the hardware: it publishes the microphone, plays the agent's audio, and forwards each actuator frame to the controller over serial.
In other words, rather than playing back authored expressions, the physical head mirrors an avatar that is itself generated from the agent's speech.
We then ran a within-subjects study comparing the two systems. Each of the 41 participants held a short German conversation with both robot heads and rated them on the Godspeed Questionnaire Series, plus items on perceived response speed and eeriness.
Reflection
I joined a working proof of concept (one script that got a Simli avatar onto the robot head) and my main contribution was turning it into something three people could develop and actually run in a study. That meant splitting the script into the agent, extractor, and client packages, moving configuration into a single typed Pydantic settings module shared by all three, and adding the pre-commit hooks and lint rules that kept the codebase consistent once we were all committing to it. Unglamorous work, but the study depended on being able to start the whole pipeline reliably in a room full of waiting participants.
The most satisfying bug was an audio one. The agent's speech reached the loudspeaker faster than the corresponding actuator frames reached the head, so the robot's mouth lagged behind its own voice. The fix was to deliberately delay the audio playback to line it back up with the movement: the kind of problem where the correct move is to make one path slower rather than the other faster.
The results supported our hypotheses. The streaming pipeline was rated significantly higher on all four Godspeed subscales (anthropomorphism, animacy, likeability, and perceived intelligence) despite its facial movement being coarser, tracking-derived, and occasionally wrong. It was also rated less eerie than the baseline, which surprised me: coarse, tracking-derived movement on a humanlike face is exactly what I would have expected to land in the uncanny valley. Reading the free-text answers made the pattern clearer: participants complained about the baseline's pauses and its eyes drifting during "thinking" gaps far more than about the fidelity of either face. What felt wrong to them was the rhythm of the conversation, not the animation.
The honest caveat is that we changed two things at once. The streaming pipeline was both faster and better-sounding, and nothing in our data separates those two contributions. Writing that limitation into the paper was a useful corrective to how neatly the numbers had come out.
Running the study itself was the part I had least experience with, and it taught me the most: counterbalancing the order, holding the prompt and hardware constant across conditions, sitting through 42 sessions, and then analysing the results properly rather than eyeballing the means. Watching people talk to the head was also its own kind of feedback: some spoke to it like a person, others fell into the clipped phrasing they use with a voice assistant, and a few interrupted the slow robot the moment they had experienced the fast one.
