AI Phone Assistant — Twilio + OpenAI Realtime on Cloudflare Workers

A working AI phone assistant you can deploy in one sitting. Callers talk, the assistant captures a structured intake, and you get a transcript with an optional email summary.

Built on Twilio voice + OpenAI Realtime + Cloudflare Workers (Durable Objects, D1 or Neon).

You'll need: a Twilio account with a phone number, an OpenAI API key, a Cloudflare account, and Node.js installed.

No subscriptions, no fixed costs. Cloudflare Workers and D1/Neon have free tiers, so the infrastructure side is zero until you're actually handling call volume. You only pay for what you use.

What happens when someone calls

1. Caller dials your Twilio number
2. A Cloudflare Worker answers with a bilingual greeting and opens a media stream
3. Audio flows to OpenAI Realtime, which runs the conversation
4. The assistant captures caller name, reason, and urgency through a structured tool call
5. The intake record and full transcript get saved to D1 or Neon
6. If Resend is configured, your team gets an email summary

flowchart LR
  A["Caller dials your Twilio number"] --> B["Twilio POST /voice/incoming"]
  B --> C["Cloudflare Worker returns TwiML + Media Stream"]
  C --> D["Durable Object bridges Twilio audio"]
  D --> E["OpenAI Realtime"]
  D --> F["D1 or Neon persists intake + transcript"]
  F --> G["Optional email summary"]

Loading

Sample intake record

After a successful call, your database contains something like:

Field	Value
caller_name	Maria Chen
reason_summary	Scheduling a follow-up appointment for next Thursday
preferred_language	English
urgency	routine
transcript_text	(full conversation, both sides)
call_status	completed
email_status	sent

If Resend is configured, the team also gets a formatted email with the structured fields and the full transcript attached.

Why this stack?

• Cloudflare Workers + Durable Objects — WebSocket support at the edge, no origin server. Cold starts haven't been a problem for real-time voice in practice.
• OpenAI Realtime — speech-to-speech, no separate STT/TTS pipeline to stitch together.
• Twilio — one webhook, one phone number, you're live. It just works.
• D1 or Neon — D1 if you want zero external dependencies. Neon if you want Postgres.

Automated setup

If you want the quickest Cloudflare-native setup, run:

./scripts/setup.sh

That script:

• installs dependencies
• creates or reuses a D1 database
• writes wrangler.toml
• applies the schema locally and remotely
• sets Worker secrets from your environment
• deploys to workers.dev or an optional custom domain

Set CUSTOM_DOMAIN=your-subdomain.example.com before running if you want a custom hostname.

Manual setup

If you prefer to wire things up yourself, or you're using Neon instead of D1:

1. Install dependencies.

npm install

2. Pick your storage path.

• D1: docs/setup-d1.md
• Neon: docs/setup-neon.md

3. Copy the example env file for local development.

cp .dev.vars.example .dev.vars

4. Start local development.

npm run dev

5. Deploy the Worker.

npm run deploy

6. In Twilio, set your phone number's inbound voice webhook to:

https://<your-domain>/voice/incoming

7. Place one real call and confirm the saved intake with docs/inbound-smoke-test.md.

Choose your storage

Cloudflare D1

Use D1 if you want the simplest all-in-Cloudflare setup.

• Setup guide: docs/setup-d1.md
• Schema: docs/schema-d1.sql

Neon Postgres

Use Neon if you want Postgres and easier querying outside Cloudflare.

• Setup guide: docs/setup-neon.md
• Schema: docs/schema-neon.sql

Twilio setup details

If you already set the webhook during setup, you're done. For reference:

1. Open the phone number you want to use in the Twilio console.
2. Under Voice, set "A call comes in" to https://<your-domain>/voice/incoming
3. Use HTTP POST.

That is the only Twilio webhook required for inbound calls.

Prompt and behavior customization

The two main files to change are:

• src/prompts.ts for the system prompt and tool schema
• src/realtime-bridge.ts for the first-turn greeting behavior

For current OpenAI guidance on Realtime prompting, use the official docs instead of copying private notes into your repo:

• Realtime API guide
• Realtime prompting guide
• Realtime prompting cookbook

Docs

• docs/index.md: docs home
• docs/setup-d1.md: fastest setup path
• docs/setup-neon.md: Postgres setup
• docs/inbound-smoke-test.md: first-call verification
• docs/customization.md: prompt and greeting changes
• docs/voice-notes.md: Twilio and WebSocket implementation lessons
• docs/outbound-starter.md: minimal outbound route overview
• docs/outbound-patterns.md: production-oriented outbound notes

Cost profile

The Cloudflare side runs on the free Workers plan for most test and small deployments. The real costs come from Twilio and OpenAI:

• Twilio charges per minute of voice usage (inbound + carrier fees)
• OpenAI Realtime charges per token for audio input/output

A short 2-3 minute intake call typically costs a few cents on the Twilio side and roughly $0.10-0.30 in OpenAI Realtime tokens. Check current pricing for Twilio Voice and OpenAI Realtime before going to production.

Environment variables

Required secrets:

• OPENAI_API_KEY
• TWILIO_AUTH_TOKEN
• DATABASE_URL if you use Neon

Optional variables

• RESEND_API_KEY
• NOTIFICATION_EMAIL
• NOTIFICATION_BCC_EMAILS
• RESEND_FROM_EMAIL
• TWILIO_ACCOUNT_SID
• TWILIO_PHONE_NUMBER
• OUTBOUND_API_TOKEN
• STORAGE_BACKEND if you want to force neon or d1

Non-secret runtime values live in wrangler.toml.

Routes

• GET /
• GET /health
• POST /voice/incoming
• POST /voice/outbound
• POST /voice/outbound/twiml
• POST /voice/status
• GET|POST /voice/ws/:callSid

Read these before deploying to production:

Security notes

• Validate X-Twilio-Signature with TWILIO_AUTH_TOKEN
• If you use Cloudflare Access for inbound only, bypass /voice/incoming and /voice/ws/*
• If you also use the outbound starter, bypass /voice/outbound/twiml and /voice/status
• Keep /voice/outbound behind your own auth. Twilio never needs direct access to that route.
• Do not commit .dev.vars

Tooling note (undici override)

The repo currently keeps an undici override in package.json because the current wrangler and miniflare dependency chain can otherwise surface audit findings. That override is a temporary safety pin for this version of the Cloudflare toolchain, and you can remove it later if a newer wrangler release makes it unnecessary.

Key files

• src/index.ts: Worker routes and TwiML responses
• src/realtime-bridge.ts: Twilio to OpenAI Realtime bridge
• src/prompts.ts: system prompt and tool schema
• src/db.ts: Neon and D1 persistence helpers
• src/email.ts: optional Resend email delivery
• src/twilio.ts: Twilio signature validation
• scripts/setup.sh: turnkey D1 setup and deploy
• LICENSE: MIT license

---

MIT licensed.