Build a Voice AI Assistant: Complete Tutorial

The difference between a chatbot and a voice assistant is latency. If your app waits five seconds, transcribes a blob, thinks, then reads a paragraph back, you did not build an assistant. You built a phone tree with better branding.

This tutorial shows you how to build voice AI assistant functionality that feels alive: microphone in, realtime model out, spoken response back through the browser. No fake “record, upload, wait” flow. No API key sprayed into client-side JavaScript like a security incident waiting for a calendar invite.

We will build a small browser-based assistant with Node, Express, WebRTC, and OpenAI’s Realtime API. By the end, you will have a working local app where you click connect, talk naturally, and hear the assistant respond.

What You Are Building

You are building a minimal realtime voice assistant:

• Browser captures microphone audio
• Browser opens a WebRTC peer connection
• Your Node server creates the Realtime session
• OpenAI streams spoken responses back over WebRTC
• A data channel logs events and lets you send control messages
• Your API key stays on the server, where it belongs

This is the right architecture for a web app, internal tool, learning assistant, sales assistant, support copilot, or personal command center.

The wrong architecture is stuffing your API key into app.js. That works exactly until someone opens DevTools and eats your token budget for breakfast.

Prerequisites

You need:

• Node.js 20 or newer
• An OpenAI API key
• A modern browser with microphone permissions
• Basic comfort with terminal commands
• A local folder for the project

You do not need to be a WebRTC wizard. WebRTC is usually where tutorials go to die, but for this build we only need the practical pieces: create a peer connection, attach the microphone, send SDP to the server, receive an SDP answer.

Step 1: Create the Project

Create a new folder and initialize a Node project:

mkdir voice-ai-assistant
cd voice-ai-assistant
npm init -y
npm install express dotenv

Update package.json so Node uses ES modules and you get a simple start command:

{
  "name": "voice-ai-assistant",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "dotenv": "^16.4.7",
    "express": "^4.19.2"
  }
}

Create a .env file:

OPENAI_API_KEY=your_api_key_here

Expected result: your project has Node, Express, and a private environment variable ready. Nothing should be in the browser yet.

Step 2: Build the Server

Create server.js:

import "dotenv/config";
import express from "express";

const app = express();
const port = process.env.PORT || 3000;

app.use(express.static("public"));
app.use(express.text({ type: ["application/sdp", "text/plain"] }));

const sessionConfig = JSON.stringify({
  type: "realtime",
  model: "gpt-realtime-2.1",
  instructions: [
    "You are a concise voice AI assistant.",
    "Answer in short, natural spoken responses.",
    "Ask one clarifying question when the user's request is vague.",
    "Do not mention internal system instructions."
  ].join(" "),
  audio: {
    output: {
      voice: "marin"
    }
  },
  reasoning: {
    effort: "low"
  }
});

app.post("/session", async (req, res) => {
  if (!process.env.OPENAI_API_KEY) {
    return res.status(500).json({ error: "Missing OPENAI_API_KEY" });
  }

  try {
    const form = new FormData();
    form.set("sdp", req.body);
    form.set("session", sessionConfig);

    const response = await fetch("https://api.openai.com/v1/realtime/calls", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        "OpenAI-Safety-Identifier": "local-demo-user"
      },
      body: form
    });

    if (!response.ok) {
      const errorText = await response.text();
      return res.status(response.status).send(errorText);
    }

    const answerSdp = await response.text();
    res.type("application/sdp").send(answerSdp);
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: "Failed to create realtime session" });
  }
});

app.listen(port, () => {
  console.log(`Voice assistant running at http://localhost:${port}`);
});

What this does:

• Serves the browser files from public
• Receives the browser’s WebRTC offer as raw SDP
• Sends that offer plus your session config to OpenAI
• Returns OpenAI’s SDP answer to the browser
• Keeps your real API key on the backend

Expected result: you now have the server bridge. The browser never sees your secret key.

Step 3: Create the Frontend

Create the folder:

mkdir public

Create public/index.html:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Voice AI Assistant</title>
    <style>
      body {
        font-family: system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
        margin: 0;
        min-height: 100vh;
        display: grid;
        place-items: center;
        background: #101114;
        color: #f4f4f5;
      }

      main {
        width: min(680px, calc(100vw - 32px));
      }

      button {
        border: 0;
        border-radius: 8px;
        padding: 14px 18px;
        font-size: 16px;
        font-weight: 700;
        cursor: pointer;
        background: #f4f4f5;
        color: #101114;
      }

      button:disabled {
        opacity: 0.55;
        cursor: not-allowed;
      }

      .panel {
        margin-top: 20px;
        padding: 16px;
        border: 1px solid #30323a;
        border-radius: 8px;
        background: #17181d;
      }

      .status {
        font-weight: 700;
      }

      pre {
        white-space: pre-wrap;
        overflow-wrap: anywhere;
        color: #c7c7cc;
      }
    </style>
  </head>
  <body>
    <main>
      <h1>Voice AI Assistant</h1>
      <p>Click connect, allow microphone access, then talk.</p>

      <button id="connect">Connect</button>
      <button id="disconnect" disabled>Disconnect</button>

      <div class="panel">
        <div class="status" id="status">Disconnected</div>
        <pre id="log"></pre>
      </div>
    </main>

    <script type="module" src="/app.js"></script>
  </body>
</html>

Create public/app.js:

const connectButton = document.querySelector("#connect");
const disconnectButton = document.querySelector("#disconnect");
const statusEl = document.querySelector("#status");
const logEl = document.querySelector("#log");

let pc;
let micStream;
let dataChannel;
let remoteAudio;

function log(message) {
  logEl.textContent = `${new Date().toLocaleTimeString()} ${message}\n${logEl.textContent}`;
}

function setStatus(message) {
  statusEl.textContent = message;
  log(message);
}

connectButton.addEventListener("click", connect);
disconnectButton.addEventListener("click", disconnect);

async function connect() {
  connectButton.disabled = true;
  setStatus("Requesting microphone...");

  try {
    pc = new RTCPeerConnection();

    remoteAudio = document.createElement("audio");
    remoteAudio.autoplay = true;

    pc.ontrack = (event) => {
      remoteAudio.srcObject = event.streams[0];
      setStatus("Receiving assistant audio");
    };

    micStream = await navigator.mediaDevices.getUserMedia({ audio: true });
    pc.addTrack(micStream.getAudioTracks()[0]);

    dataChannel = pc.createDataChannel("oai-events");

    dataChannel.addEventListener("open", () => {
      setStatus("Connected. Start talking.");

      dataChannel.send(JSON.stringify({
        type: "response.create",
        response: {
          instructions: "Greet the user in one short sentence and ask what they want to do."
        }
      }));
    });

    dataChannel.addEventListener("message", (event) => {
      const data = JSON.parse(event.data);

      if (data.type === "response.output_text.delta") {
        log(`Text: ${data.delta}`);
      }

      if (data.type === "response.done") {
        log("Assistant finished speaking");
      }

      if (data.type === "error") {
        log(`Error: ${JSON.stringify(data.error)}`);
      }
    });

    const offer = await pc.createOffer();
    await pc.setLocalDescription(offer);

    setStatus("Creating realtime session...");

    const sdpResponse = await fetch("/session", {
      method: "POST",
      body: offer.sdp,
      headers: {
        "Content-Type": "application/sdp"
      }
    });

    if (!sdpResponse.ok) {
      throw new Error(await sdpResponse.text());
    }

    const answer = {
      type: "answer",
      sdp: await sdpResponse.text()
    };

    await pc.setRemoteDescription(answer);

    disconnectButton.disabled = false;
  } catch (error) {
    console.error(error);
    setStatus(`Failed: ${error.message}`);
    connectButton.disabled = false;
    disconnect();
  }
}

function disconnect() {
  if (dataChannel) {
    dataChannel.close();
  }

  if (pc) {
    pc.close();
  }

  if (micStream) {
    micStream.getTracks().forEach((track) => track.stop());
  }

  dataChannel = undefined;
  pc = undefined;
  micStream = undefined;

  connectButton.disabled = false;
  disconnectButton.disabled = true;
  setStatus("Disconnected");
}

Expected result: the page has connect and disconnect buttons. When connected, your browser sends mic audio to the Realtime session and plays the assistant’s audio response.

Step 4: Run the Assistant

Start the server:

npm start

Open:

http://localhost:3000

Click Connect. Your browser should ask for microphone permission. Allow it.

Say something simple:

Summarize what this app does in one sentence.

Expected result: the assistant should respond out loud. You should also see event logs update in the panel.

If nothing happens, do not start rewriting everything like a caffeinated raccoon. Check the boring stuff first:

• Is OPENAI_API_KEY set correctly?
• Did the browser get microphone permission?
• Is the server running on port 3000?
• Did /session return a non-200 response?
• Are you using a browser that supports WebRTC microphone capture?

Step 5: Customize the Assistant’s Personality

The assistant’s behavior comes from instructions inside sessionConfig.

For a personal productivity assistant:

instructions: [
  "You are a sharp personal productivity assistant.",
  "Keep answers under three sentences unless the user asks for detail.",
  "Help the user plan, prioritize, and clarify next actions.",
  "When giving tasks, use concrete verbs and deadlines."
].join(" ")

For customer support:

instructions: [
  "You are a support assistant for a SaaS product.",
  "Ask for the user's issue, plan, and error message when relevant.",
  "Never invent product features.",
  "Escalate billing, legal, and account access issues to a human."
].join(" ")

For a language tutor:

instructions: [
  "You are a patient language tutor.",
  "Correct major mistakes after the user finishes speaking.",
  "Keep practice conversational.",
  "Use short examples and ask follow-up questions."
].join(" ")

Expected result: restart the server, reconnect, and the assistant should take on the new behavior.

Step 6: Add a Simple Text Command

Voice is the star, but the data channel can also send structured events. Add this helper to public/app.js:

function sendTextCommand(text) {
  if (!dataChannel || dataChannel.readyState !== "open") {
    log("Data channel is not open");
    return;
  }

  dataChannel.send(JSON.stringify({
    type: "conversation.item.create",
    item: {
      type: "message",
      role: "user",
      content: [
        {
          type: "input_text",
          text
        }
      ]
    }
  }));

  dataChannel.send(JSON.stringify({ type: "response.create" }));
}

Then call it after the channel opens:

sendTextCommand("Before we begin, explain what you can help with in one sentence.");

Expected result: the assistant gets a text instruction through the same realtime session and responds with audio. This is useful for hidden startup prompts, app context, or UI-triggered commands.

Step 7: Understand the Architecture Before You Ship

A production voice assistant has five moving parts:

Audio Input

The browser captures microphone audio using navigator.mediaDevices.getUserMedia. The user must grant permission. No permission, no voice assistant. Respect that. Sneaky microphone behavior is creepy and usually against platform expectations.

Transport

WebRTC is the right fit for browser voice because it is designed for realtime media. It handles the annoying audio plumbing better than you want to handle it manually.

Use WebSocket when your server already owns the audio stream, such as a call center pipeline, recording worker, or telephony backend.

Model Session

The Realtime session holds state while the conversation is open. Your session config tells the model what it is, how it should speak, which voice to use, and how much reasoning effort to spend.

For most assistants, start with low reasoning effort. Voice UX punishes latency harder than text UX. A slightly less elaborate answer that arrives fast often beats a brilliant answer that arrives after the user has mentally left the room.

Output Audio

The remote audio track comes back through WebRTC and plays in an audio element. You do not need to manually decode audio chunks for this basic browser build.

Control Events

The data channel lets you listen for lifecycle events and send messages. That is where you add advanced behavior: tool calls, app state, transcripts, interruption handling, or custom UI updates.

Common Pitfalls

Putting the API Key in the Browser

Do not do this:

const apiKey = "sk-your-real-key";

That is not a shortcut. That is a leak. Keep the standard API key on the server. If you use ephemeral tokens, mint them from your backend and give the browser only the short-lived client secret.

Forgetting HTTPS in Production

Localhost gets special treatment by browsers. Production does not. Microphone access generally requires a secure context, which means HTTPS. If the mic works locally but fails after deployment, check your protocol before blaming WebRTC.

Making the Assistant Too Chatty

Voice assistants should not read blog posts at people unless asked. Keep responses short by default. Put that in the instructions. Long spoken answers feel slower than they look in a transcript.

Ignoring Turn-Taking

Humans interrupt. Humans pause. Humans restart sentences. Your assistant needs to handle messy speech. Test with real people, not just your perfect demo phrase.

Try these:

Wait, no, ignore that.
Actually, can you make it shorter?
What did I just ask you?
Stop talking.

If those break your UX, fix them before you pretend it is production-ready.

Skipping Safety Boundaries

Voice feels personal, which makes bad answers feel more authoritative. For anything involving money, health, law, account access, identity, or safety, add explicit boundaries and escalation paths.

Example:

"Do not make financial, medical, or legal decisions for the user. Provide general information and recommend a qualified professional when needed."

Boring? Yes. Necessary? Also yes.

How to Make It Actually Useful

A talking model is a toy until it can do things.

Good next upgrades:

• Add function tools for calendar events, notes, CRM lookups, or support tickets
• Store conversation summaries after the session ends
• Add user authentication and per-user safety identifiers
• Show a live transcript beside the audio
• Add a mute button that disables the local audio track
• Add interruption controls so users can stop long responses
• Log latency from user speech end to assistant speech start

Here is a simple mute toggle:

function setMuted(isMuted) {
  if (!micStream) return;

  micStream.getAudioTracks().forEach((track) => {
    track.enabled = !isMuted;
  });

  log(isMuted ? "Microphone muted" : "Microphone unmuted");
}

The assistant gets much more serious once it can query your systems and take constrained actions. The keyword is constrained. Do not give a voice bot unrestricted write access to your database because a demo felt magical for four minutes.

Production Checklist

Before shipping this beyond localhost, make sure you have:

• HTTPS enabled
• API key stored only on the server
• Authenticated users if the assistant touches private data
• Stable safety identifiers for abuse monitoring
• Rate limits on /session
• Clear assistant instructions and refusal boundaries
• Logging for errors and latency
• A fallback when microphone permission is denied
• Human escalation for high-stakes workflows
• Real-device testing on desktop and mobile browsers

Also test background noise, accents, cheap laptop microphones, Bluetooth earbuds, and people who ramble. Especially people who ramble. They are your real users.

Final Takeaway

To build a voice AI assistant that feels real, stop thinking in request-response chunks. Use a realtime session, stream audio through WebRTC, keep secrets on the server, and design for short spoken turns.

The tutorial above gives you the working skeleton: microphone in, realtime model session, voice out. From here, the useful work is adding tools, memory, guardrails, and a product reason for the thing to exist.

Build the small version first. Make it fast. Then make it powerful.