Building with NVIDIA Nemotron 3 VoiceChat

Last week, I started exploring NVIDIA’s latest hardware and AI infrastructure after reading about its RTX hardware designed to run AI locally on personal computers.

While browsing NVIDIA’s website, I discovered Nemotron 3 VoiceChat, an early-access, full-duplex voice-to-voice model. I requested access because, well, why not?

A few days later, NVIDIA sent me an invitation.

Naturally, I had to build something with it.

Building the demo

Using my NVIDIA API key, I asked Claude Code to read the documentation and help me build a browser-based evaluation harness. We created several personas with different system prompts, allowing me to select a scenario and speak directly with the model. After that, we also added a custom prompt that can be updated live from the browser.

Running the model requires a GPU, so I used NVIDIA Brev to provision the required infrastructure. Claude Code helped with the setup through NVIDIA’s tools and APIs, and the result was a working web app where I could choose a scenario and have real-time voice conversations with the model.

The model ran on an A100-80GB GPU in Brev, while the web app ran on my computer. I connected them through an SSH tunnel, so the GPU endpoint was never exposed publicly.

The Nemotron family

Nemotron is a family of models designed for different roles:

Nemotron Nano — a smaller, efficient model for specialized agents.
Nemotron Super — a larger model for reasoning, planning, and multi-agent systems.
Nemotron Ultra — the largest model, optimized for difficult reasoning tasks.
Nemotron VoiceChat — specialized for natural, real-time voice conversations.

VoiceChat is not simply Nemotron Ultra with a voice added to it. It is a separate model built specifically for speech-to-speech conversations.

What makes full duplex different

Traditional voice agents usually connect three independent stages:

Speech → ASR → finished text → text LLM → finished response → TTS → speech

ASR means automatic speech recognition, which is basically speech-to-text. Each stage waits for the previous stage, and the application has to connect and coordinate all three services.

A full-duplex model works differently:

Continuous incoming audio → unified streaming speech model → continuous outgoing audio

VoiceChat still contains speech-recognition, language, and speech-generation capabilities. What makes it different is that they work together as one streaming model architecture instead of three separate services passing completed text between them.

The model can process small audio chunks while it is generating speech. This allows it to listen and speak at the same time, making interruptions, overlap, and more natural turn-taking possible.

The experience

Before this, I experimented with the Gemini Live API and OpenAI’s Realtime API.

Gemini Live is pretty good. OpenAI Realtime is especially powerful because it combines natural voice conversations with strong reasoning and tool calling.

Nemotron VoiceChat produced surprisingly fluid conversations. It was interesting to speak with a model running inside a GPU container that I had provisioned myself instead of calling a normal hosted API.

The conversations flowed well, although interruptions occasionally caused mismatches. Handling interruptions and deciding exactly when to speak or stay quiet remain some of the hardest problems in real-time voice AI.

The biggest missing capability is currently tool calling. Without it, using the model in real applications is difficult. NVIDIA has acknowledged that tool calling is not yet available. Since the model is still in early access, I expect this will improve.

What we built

A protocol-compatible mock server, allowing most of the application to be developed without paying for a GPU.
Switching between the mock and real model requires changing only one environment variable.
An evaluation harness that measures first-audio latency, displays both transcripts, records agent audio, and exports session data.
Multiple personas through system prompts: casual conversation, clinic intake, receptionist, interview coach, and strength coach.
A dynamic custom prompt option that can be updated live from the browser.
The real model running on an NVIDIA Brev A100-80GB, costing approximately $1.98 per running hour.
An SSH tunnel between my computer and Brev, so the GPU endpoint was never exposed publicly.

Bugs and challenges

As usual, getting access was the easy part. Making everything work together was more interesting.

Wrong audio format produced garbled speech. The model expected 24kHz audio, and the original configuration was silently ignored.
Chrome suspended its audio context. The microphone appeared active, but no audio reached the model until we explicitly resumed it.
The model cannot greet first. This checkpoint only responds after receiving spoken audio.
Interruptions are difficult. Full duplex makes interruption possible, but it does not make turn-taking automatically perfect.
Silence can be confusing. Continuously streaming the microphone sometimes caused phantom user transcripts from near-silence.

These issues gave me a better understanding of what makes voice models difficult. Generating a good spoken response is only one part of the problem. The model also needs to hear correctly, understand timing, decide when a person has finished, react to interruptions, and know when not to speak.

What I learned

Building the app was a great experience.

It gave me a much better understanding of full-duplex voice models, GPU infrastructure, streaming audio, and the challenges involved in making conversations feel natural.

It also showed me another side of NVIDIA. I started by reading about local AI hardware, then ended up using NVIDIA’s model registry, container infrastructure, Brev GPUs, and an early-access speech model in one project.