The challenge
Some things you can learn from a textbook. Others need time, nuance, and most importantly: practice. One of the hardest skills to master as a future doctor? Making decisions together with patients – also called shared decision making.
Shared decision making is rarely straightforward. Together, patients and doctors must find out what decision is best fit for the situation, taking into account the best scientific evidence as well as what matters to the patient. Yet students only get limited opportunities to practice these complex conversations.
So how do you prepare students for the unpredictable messiness of this human interaction without relying on actors, peers, or one-size-fits-all simulations? That’s the challenge Pieter Van Bostraeten and Bert Aertgeerts from the Academic Center for General Practice (ACHG) at KU Leuven brought to us: can we build a realistic, scalable, and safe training ground where students can practice difficult conversations with realistic patients - anytime, anywhere?
Before we started building, we had to understand the current reality. How are students being trained today? What works? What doesn’t? What kind of feedback actually helps them grow?
We mapped existing training methods through in-depth observations, conversations with professors, and user testing with teachers, doctors and students. We sat in on sessions with actors. Took notes. Asked questions. We examined how students were assessed: what made conversations feel ‘right’? Empathy? Clarity? Confidence?
We also looked into medical cases, patient personalities, and the types of conversational challenges students need to master. Together with ACHG, we drafted character profiles, case prompts, and difficulty levels - the foundation of the narrative engine that would eventually drive each avatar.
With a solid vision in place, we began designing the platform. A virtual consultation room inside a web-based platform where students can interact with digital patients. It had to be seamless, accessible, and fully browser-based. No apps, no downloads, no special hardware. Just log in and start talking.
To make the avatars feel lifelike (think: subtle in expressions, natural in timing), we relied on MetaHuman technology in Unreal Engine. Working with this technology allowed us to design highly realistic patients: diverse in age, background, tone and overall appearance. But a realistic face is only half the story. It also has to move the right way, at the right time, in sync with what’s being said.
That’s where we used Audio2Face: a powerful animation tool that helped us match spoken audio with facial movements, generating natural lip sync and emotion-based microexpressions on the fly, making it all even more realistic.
That realism, however, comes at a technical cost. Rendering MetaHuman in-browser isn’t natively supported. So, we turned to pixel streaming: a technique that lets us run the heavy 3D engine in the cloud, and stream the output to a student’s laptop. This keeps the experience smooth and high-quality - even on basic laptops.
High-quality Dutch AI voices are one thing. But Flemish accents? That’s another thing. We trained our own Flemish voice clones using ElevenLabs, giving each avatar a distinct tone and regional accent. Think Limburg, Antwerp, Flemish Brabant. Sorry, West Flanders - your dialect is one even we don’t dare to recreate.
And then there’s the brains behind it all. A custom orchestrator manages the entire conversation loop: from detecting when a student starts or stops speaking, to interpreting what they said, generating a meaningful response via large language models (GPT-4o), and converting that text back into speech. All within seconds, so the conversation flows naturally.
Each avatar has a deeply defined personality prompt, outlining age, background, tone of voice, emotional baseline, and conversational style. One might be nervous. Another defensive. Some talk a lot, others hold back.
Layered on top of that personality is the case prompt; the reason they’re visiting the doctor. Each case comes with relevant symptoms, medical history, emotional state, and contextual nuance. And because real patients don’t come in easy, medium, or hard modes, we created a difficulty scaling system: at higher levels, avatars might withhold key information or push back when they feel misunderstood.
💡 Take Sanne, a 29-year-old single mother of a toddler. Ear tubes for her three year old son are being discussed, but the idea of general anaesthesia terrifies her – a distant relative once didn’t wake up after surgery, and that fear stuck. On top of that, she’s struggling to organise the hospital visit: no nearby family, a half-time job, and another young child at home. If the student ignores her concerns, she shuts down or gets emotional. But with empathy, clarity, and no medical jargon? She opens up. That’s the kind of layered human interaction students can practice – again and again, until they get it right.
Training without feedback doesn’t stick. That’s why every conversation ends with a structured reflection opportunity. After each session, the full transcript of the interaction is passed to an LLM that acts as an evaluator. It processes the entire exchange and analyses it across a set of predetermined communication objectives.
The result? Instant, personalized feedback. It highlights what went well, how the student handled cues, and whether key questions were missed.
Because the feedback is generated immediately, students can revisit the same case multiple times, adjusting their approach, experimenting with tone, or building confidence gradually. It transforms the experience from a one-off simulation into a repeatable, personal learning loop - with multiple difficulty settings.
In the future, this could even be scaled to include feedback from educators. Conversations can be reviewed asynchronously, with added comments or evaluations by teachers. Everything is logged into the system, so growth can be tracked over time.
We created a full demo video for this platform:
From day one, we ensured GDPR and EU AI Act-compliance. All interactions - including voice data, transcripts, and feedback - are handled securely in the cloud. Conversations are logged and stored safely, with transcripts, summaries, and structured feedback made available to both students and professors.
At this stage, the platform is in a focused proof-of-concept phase. It’s being tested with a small group of students and professors to validate functionality and fine-tune performance.
Even though we are currently in the testing stage, the infrastructure underneath was never designed for a one-off pilot. It is built with scale in mind. Whether it’s adding more cases, switching up avatar personalities, tweaking difficulty levels, or expanding feedback mechanisms: the system can handle it.
Doctors spend years learning how the body works. But when something’s wrong, what matters most is how they make the right decision. How they explain the consequences. How they listen to what matters to who’s in front of them. How they connect.
This platform gives students the freedom to make mistakes, reflect, and try again - in a space that feels safe and real enough to truly prepare them for real-world practice. Without pressure. Without real consequences. And with the ability to stop, reflect, and go again.
Of course, it’s not a real conversation. But it’s close enough to build confidence and competence. And that’s where its value lies: not in replacing what already exists, but extending it. Offering something that fits in between lectures, roleplays, internships and actual application with real patients.
Want to know more?
