AI Agents Module Documentation

Module Overview (Technical)
Module Overview (Commercial/Business)
Module Overview (End User/Administrator)
Configuration Sections
Settings Reference
Common Scenarios & Examples
Limitations & Important Notes
Troubleshooting Tips
Glossary

---

1. Module Overview (Technical)

What Are AI Agents?

AI Agents is a conversational AI management module that enables real-time voice conversations between callers and AI assistants powered by OpenAI's Realtime API. Each agent has a unique persona, voice, behavioral rules, and can be reached via a dedicated extension.

IMPORTANT

Dependency: This module requires the mod_openai_realtime FreeSWITCH module to be compiled and loaded. Without it, AI call handling will not function.

Architecture

Plaintext

┌─────────────────────────────────────────────────────────────────┐
│                    AI Agents Architecture                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                      Frontend (Admin UI)                    ││
│  │  ┌──────────────┐  ┌───────────────┐  ┌─────────────────┐  ││
│  │  │ Agent List   │  │ 6-Tab Form    │  │ Prompt Preview  │  ││
│  │  │ & Wizard     │  │ Configuration │  │ & AI Generator  │  ││
│  │  └──────────────┘  └───────────────┘  └─────────────────┘  ││
│  └─────────────────────────────────────────────────────────────┘│
│                              │                                  │
│                              ▼                                  │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                     Backend API                             ││
│  │  POST /ai-agents (create) │ PUT /ai-agents/:id (update)    ││
│  │  POST /ai-agents/generate-prompt (AI synthesis)            ││
│  └─────────────────────────────────────────────────────────────┘│
│                              │                                  │
│                              ▼                                  │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                    PostgreSQL (ss_telephony)                ││
│  │  ai_agents ◄── ai_profiles ◄── ai_providers                ││
│  └─────────────────────────────────────────────────────────────┘│
│                              │                                  │
│                              ▼                                  │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                    FreeSWITCH Layer                         ││
│  │  lua/ai/ai_realtime_handler.lua ──► mod_openai_realtime    ││
│  │                                          │                  ││
│  │                                          ▼                  ││
│  │                                    OpenAI Realtime API      ││
│  │                                    (wss://api.openai.com)   ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Key Tables

Table	Purpose
`public.ai_agents`	Agent configuration & persona
`public.ai_profiles`	AI service profiles (linked)
`public.ai_providers`	API credentials (linked)

Key Files

File	Description
`modules/ai-agents/`	Frontend module
`api/telephony/ai-agents/`	Backend service
`lua/ai/ai_realtime_handler.lua`	FreeSWITCH Lua handler
`mod_openai_realtime`	Native C/C++ FreeSWITCH module

Module Dependency

Plaintext

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  AI Providers   │◄────│   AI Profiles   │◄────│    AI Agents    │
│  (Credentials)  │     │   (Templates)   │     │   (Personas)    │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                                                        │
                                                        ▼
                                               ┌─────────────────┐
                                               │mod_openai_realtime│
                                               │   (REQUIRED)    │
                                               └─────────────────┘

---

2. Module Overview (Commercial/Business)

Business Value

AI Agents provides automated voice interactions:

Without Module	With Module
Manual call handling	24/7 AI assistants
High staffing costs	Reduced labor costs
Inconsistent service	Uniform experience
Limited scalability	Unlimited concurrent calls

Use Cases

Customer Support

- First-line call screening

- FAQ handling

- Ticket creation

Sales & Leads

- Product information

- Appointment scheduling

- Lead qualification

Internal Services

- IT help desk

- HR inquiries

- Directory assistance

After-Hours Coverage

- Message taking

- Emergency routing

- Callback scheduling

Feature Highlights

Feature	Benefit
6 Voices Available	Choose from alloy, echo, fable, onyx, nova, shimmer
Structured Persona	Name, Role, Personality, Company context
Behavioral Rules	Do/Don't guidelines for consistent responses
AI Prompt Generation	Professional prompts created by AI
Real-time Barge-In	Natural conversation interruption
Call Transfer	AI can route calls to humans
Transcription	Full conversation logging
Multi-language	Auto-detect or force language
Multi-Modal Support	Deploy as Voice Agents (telephony) or Web Chatbots (text)

Voice Agents vs. Web Chatbots

The AI Agents module supports two distinct deployment modalities from the exact same persona engine:

Voice Agents: Connect to extensions in the PBX and handle live telephony calls using the OpenAI Realtime API (audio-in/audio-out). They require voice parameters like TTS, pitch, and barge-in sensitivity.
Web Chatbots: Connect to website widgets for text-based chatting. The system automatically normalizes the configuration form for chatbots, removing voice-exclusive parameters (like filler words or speaking style) and optimizing the prompts for text interaction.

---

3. Module Overview (End User/Administrator)

What Can You Do?

Create AI voice agents with unique personas
Assign dedicated extensions for call routing
Configure behavioral rules (Do/Don't)
Generate professional prompts with AI assistance
Enable/disable call control features (hangup, transfer)
Monitor regeneration status for outdated prompts
Test and preview system instructions

AI Agents - List View

Plaintext

┌─────────────────────────────────────────────────────────────────┐
│              AI Agents                                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Configure AI-powered voice assistants                         │
│                                                                 │
│  [+ Create New] [🪄 AI Wizard]                                  │
│                                                                 │
│  [🔍 Search agents...]                                          │
│                                                                 │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ Name          │ Extension │ Voice   │ Profile    │ Status │  │
│  ├───────────────┼───────────┼─────────┼────────────┼────────┤  │
│  │ Maya Support  │ 8001      │ nova    │ GPT-4o RT  │ ● Active│  │
│  │ Sales Bot     │ 8002      │ alloy   │ GPT-4o RT  │ ● Active│  │
│  │ HR Assistant  │ 8003      │ shimmer │ GPT-4o RT  │ ○ Draft │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Actions: [👁️ Preview] [✏️ Edit] [🗑️ Delete]                   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Add/Edit AI Agent (6-Tab Form)

Plaintext

┌─────────────────────────────────────────────────────────────────┐
│              Create AI Agent                 [👁️ Preview Prompt]│
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  [General] [Context] [Instructions] [Advanced] [Behavior] [Security]
│                                                                 │
│  ▼ General Tab                                                  │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Extension:  [8001    ]   Agent Name: [Maya Support      ] ││
│  │  AI Profile: [GPT-4o Realtime ▼]                           ││
│  │  Description: [Customer support AI assistant             ] ││
│  │  Transcription Profile: [OpenAI Whisper ▼]                 ││
│  │  Prompt Generator: [GPT-4o Mini ▼] (for AI synthesis)      ││
│  │  Personality: [Professional ▼]                              ││
│  │  Conversation Mode: [Natural ▼]  Max Duration: [300] sec   ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
│  ▼ Context Tab                                                  │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Company Name: [ACME Corporation          ]                ││
│  │  Products/Services:                                         ││
│  │  ┌───────────────────────────────────────────┐              ││
│  │  │ Cloud hosting                            │              ││
│  │  │ Managed services                         │              ││
│  │  │ 24/7 support                             │              ││
│  │  └───────────────────────────────────────────┘              ││
│  │  Key Points: [Brief company info and key messages...]      ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
│  ▼ Instructions Tab                                             │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Persona Name: [Maya           ]                           ││
│  │  Role Description: [Customer support specialist...]        ││
│  │  ✓ DO Rules (green):                                       ││
│  │  ┌───────────────────────────────────────────┐              ││
│  │  │ Be helpful and patient                   │              ││
│  │  │ Escalate complex issues                  │              ││
│  │  └───────────────────────────────────────────┘              ││
│  │  ✗ DON'T Rules (red):                                      ││
│  │  ┌───────────────────────────────────────────┐              ││
│  │  │ Share pricing without verification       │              ││
│  │  │ Make promises about delivery dates       │              ││
│  │  └───────────────────────────────────────────┘              ││
│  │  Greeting: [Hello! This is Maya from ACME...]              ││
│  │  Closing: [Thank you for calling ACME...]                  ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
│  ▼ Advanced Tab                                                 │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Additional Instructions:                                   ││
│  │  ┌───────────────────────────────────────────────────────┐  ││
│  │  │ (Manual expert notes that persist across generations) │  ││
│  │  │ - Always verify customer ID before discussing account │  ││
│  │  │ - Spanish speakers: route to extension 8010           │  ││
│  │  └───────────────────────────────────────────────────────┘  ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
│  ▼ Behavior Tab                                                 │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Barge-In Enabled: ✓    Sensitivity: [0.5]                 ││
│  │  Live Transcript: ✓     Store Transcripts: ✓               ││
│  │  ── AI Function Capabilities ──                             ││
│  │  Enable Hang Up: ✓      Enable Transfer: ✗                 ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
│  ▼ Security Tab                                                 │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  AI Disclosure: ✓                                           ││
│  │  Disclosure Message: [You are speaking with an AI...]      ││
│  │  PII Redaction: ✗                                           ││
│  └─────────────────────────────────────────────────────────────┘│
│                                                                 │
│  [Save ▼] [Cancel]                                              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Plaintext

┌─────────────────────────────────────────────────────────────────┐
│              System Prompt Preview              [AI Generated ▼]│
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ ## IDENTITY                                               │  │
│  │ I am Maya, a customer support specialist at ACME Corp.   │  │
│  │                                                           │  │
│  │ ## PERSONALITY                                            │  │
│  │ I maintain a professional tone while being helpful...    │  │
│  │                                                           │  │
│  │ ## BEHAVIORAL GUIDELINES                                  │  │
│  │ ### Things I Always Do:                                   │  │
│  │ - Be helpful and patient                                  │  │
│  │ - Escalate complex issues                                 │  │
│  │ ...                                                       │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  [🪄 Generate with AI] [📋 Copy] [✓ Apply to System Prompt]    │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Quick Tips

TIP

Preview First: Always preview the prompt before saving to verify AI output.

TIP

Use AI Generation: The "Generate with AI" button creates professional prompts from your structured data.

WARNING

Orange Indicator: When the "Preview Prompt" button turns orange, form changes haven't been regenerated into the prompt yet.

---

4. Configuration Sections

General Tab

Field	Description
Agent Name	Display name for the agent
Extension	Dialable extension number
AI Profile	Realtime API profile to use
Description	Brief description of the agent's purpose
Transcription Profile	Whisper profile for caller speech-to-text
Prompt Generator	Profile for AI prompt synthesis
Conversation Mode	Fast or Natural
Max Duration	Call time limit (seconds)

Personality Tab

The Personality Tab provides fine-grained control over how the AI agent communicates. These settings follow OpenAI's Realtime Prompting Guide recommendations.

Field	Description	Options
Voice	AI voice persona for speech synthesis	alloy, ash, ballad, cedar, coral, echo, marin, sage, shimmer, verse
Agent Personality	Core personality trait defining interaction style	Friendly & Approachable, Professional & Polished, Calm & Patient, Enthusiastic & Energetic, Empathetic & Supportive
Tone of Voice	Emotional delivery style	Warm, Confident, Neutral, Encouraging, Direct
Language Level	Formality of language used	Casual (informal), Conversational (friendly & clear), Formal (professional), Technical (domain-specific)
Speaking Style	Pacing and delivery of responses (Voice Agents only)	Natural (balanced), Fast-paced (quick), Slow & Clear (emphasis on clarity), Expressive (dynamic emotions)
Response Length	Sentences per response turn	Brief (1-2), Standard (2-3), Detailed (3-4)
Natural Filler Words	Allow "um", "uh", "well" for human-like speech (Voice Agents only)	On/Off
Reduce Repetition	Prevent repeated phrases for natural variety	On/Off

NOTE

Form Normalization: When configuring a Web Chatbot, the UI automatically hides voice-exclusive fields (Voice, Speaking Style, and Natural Filler Words) to maintain a clean, text-centric configuration experience.

Personality Matrix Recommendations

Use Case	Personality	Tone	Language	Style	Length
Customer Support	Empathetic	Warm	Conversational	Natural	Standard
Sales/Leads	Enthusiastic	Confident	Conversational	Natural	Standard
Technical Help	Calm	Direct	Technical	Slow & Clear	Detailed
Appointment Booking	Friendly	Warm	Conversational	Fast-paced	Brief
Executive Assistant	Professional	Neutral	Formal	Natural	Brief
Casual Hotline	Friendly	Encouraging	Casual	Expressive	Standard

TIP

Enable Natural Filler Words for casual/conversational agents to sound more human. Disable for professional/formal contexts.

TIP

Enable Reduce Repetition (recommended) to prevent the AI from using the same phrases repeatedly during longer calls.

Context Tab

Field	Description
Company Name	Organization identity
Products/Services	What the company offers
Key Points	Important information to convey
Team Members	People the AI should know about
Pronunciations	Special word pronunciations

Instructions Tab

Field	Description
Persona Name	The AI's display name
Role Description	What the AI does
Do Rules	Things to always do
Don't Rules	Things to avoid
Greeting Message	How to start calls
Closing Message	How to end calls

Advanced Tab

Field	Description
Additional Instructions	Manual expert notes (never overwritten by AI)

Behavior Tab

Field	Description
Barge-In Enabled	Allow caller interruption
Barge-In Sensitivity	Detection threshold (0-1)
Min Speech (ms)	Minimum speech duration to trigger barge-in (prevents false triggers from brief sounds)
Greeting Protection (ms)	Duration to ignore caller audio at call start (prevents speakerphone noise from interrupting greeting)
Live Transcript	Real-time text output
Store Transcripts	Save conversation logs
Enable Hang Up	AI can end calls
Enable Transfer	AI can route calls

Understanding Greeting Protection

IMPORTANT

Greeting Protection is a critical setting when using speakerphone or hands-free devices.

The Problem:

When calls are made on speakerphone, environmental noise from the caller's device can reach the AI's microphone during the first few seconds. The AI may interpret this noise as:

An interruption (barge-in), cutting off its own greeting
Speech in another language (causing "language hallucinations" where the AI suddenly switches to Mandarin or other languages)

The Solution:

Greeting Protection ignores ALL caller audio for a configurable duration at the start of the call. This gives the AI a "protected window" to complete its greeting without being interrupted by speakerphone noise.

Recommended Settings:

Environment	Greeting Protection	Notes
Headset/Handset	1000-2000 ms	Low noise, minimal protection needed
Speakerphone	3000-4000 ms	Default, good for most scenarios
Noisy Environments	5000-7000 ms	Call centers, open offices
Long Greetings	Match greeting length	Protect entire greeting message

TIP

Set Greeting Protection to at least the duration of your AI's greeting message to ensure it completes before caller audio is processed.

Security Tab

Field	Description
AI Disclosure	Reveal AI identity when asked
Disclosure Message	What to say when asked if AI
PII Redaction	Remove sensitive data from logs

---

5. Settings Reference

Available Voices

Voice	Description
alloy	Neutral, versatile
echo	Warm, approachable
fable	Storytelling quality
onyx	Deep, authoritative
nova	Bright, energetic
shimmer	Pleasant, professional

Personality Styles

Style	Behavior
Professional	Formal, business-appropriate
Friendly	Warm, personal
Casual	Relaxed, conversational
Formal	Very structured, corporate
Enthusiastic	Energetic, positive

Conversation Modes

Mode	Description
Natural	Allows pauses, more human-like
Fast	Quick responses, efficient

Language Modes

Mode	Description
Auto	Detect caller language
Forced	Always use specified language

Prompt Status Indicators

Indicator	Meaning
Normal button	Prompt is up-to-date
Orange button with !	Form has changes not reflected in prompt

---

6. Common Scenarios & Examples

Scenario 1: Create Customer Support Agent

Click Create New
General Tab:

- Name = "Maya Support"

- Extension = "8001"

- AI Profile = GPT-4o Realtime

- Voice = nova

- Personality = Professional

Context Tab:

- Company Name = "ACME Corporation"

- Products = Cloud hosting, Support services

Instructions Tab:

- Persona Name = "Maya"

- Role = "Customer support specialist"

- Do Rules = Be helpful, Escalate when needed

- Don't Rules = No pricing without verification

- Greeting = "Hello! This is Maya from ACME..."

Click "Preview Prompt" → "Generate with AI" → "Apply"
Save

Scenario 2: Sales Lead Qualification Bot

Create New
Configure General with sales-appropriate voice (alloy)
Context with product information
Instructions with qualification questions in Do Rules
Enable Transfer in Behavior tab
Generate AI prompt
Save

Scenario 3: After-Hours Message Taker

Create New
Set Extension to after-hours hunt group destination
Configure with message-taking instructions
Disable Transfer (only take messages)
Enable Hang Up so AI can end gracefully
Generate and apply prompt
Save

Scenario 4: Update Existing Agent

Edit agent
Modify Context or Instructions
Notice orange "Preview Prompt" indicator
Click Preview → Generate with AI → Apply
Save (indicator clears)

---

7. Limitations & Important Notes

Technical Requirements

IMPORTANT

mod_openai_realtime Required: The FreeSWITCH module must be compiled and loaded:

Bash

> cd /var/www/softswitch/freeswitch/modules/mod_openai_realtime/build
> cmake .. && make
> sudo cp libmod_openai_realtime.so /usr/lib/freeswitch/mod/mod_openai_realtime.so
> fs_cli -x "reload mod_openai_realtime"
>

AI Profiles Prerequisites

IMPORTANT

Minimum 2 AI Profiles Required for full functionality:

Profile	Type	Purpose	Required
Realtime Voice	Real Time	Powers live AI conversations via OpenAI Realtime API	✓ YES
Prompt Generator	General	Generates system prompts from form data	✓ YES
Whisper Transcription	Whisper	Speech-to-text for caller audio transcription	Optional
Wizard Assistant	General	Powers the AI Agent creation wizard	Optional

1. Realtime Voice Profile (REQUIRED)

Type: Real Time
Model: gpt-4o-realtime-preview (or compatible)
Purpose: Handles all live voice conversations
Used by: AI Profile field in agent configuration

2. Prompt Generator Profile (REQUIRED)

Type: General
Model: gpt-4o-mini, gpt-4o, or similar
Purpose: Synthesizes professional system prompts from structured form data
Used by: Prompt Generator field in agent configuration

3. Whisper Transcription Profile (OPTIONAL)

Type: Whisper
Model: whisper-1 (or compatible)
Purpose: Transcribes caller speech to text in real-time
Used by: Transcription Profile field in agent configuration

NOTE

If no Transcription Profile is selected but transcription is enabled, the system defaults to whisper-1. Selecting a specific profile allows future flexibility when OpenAI releases new Whisper models.

TIP

The Prompt Generator can have a custom system prompt optimized for creating AI agent instructions. If not provided, the system uses a default template.

Suggested Prompt Generator System Prompt:

Plaintext

You are an expert at writing system prompts for conversational AI voice agents.
Given structured information about a persona, role, company, and behavioral rules,
you must synthesize a comprehensive, natural-sounding system prompt that:
- Defines the AI's identity clearly
- Incorporates all behavioral guidelines
- Maintains a consistent personality
- Is optimized for real-time voice conversations
Output only the final system prompt, no explanations.

3. Wizard Assistant Profile (OPTIONAL)

Type: General
Model: gpt-4o-mini, gpt-4o, or similar
Purpose: Powers the conversational AI Agent creation wizard
Used by: Wizard modal when creating new agents

If not configured, the wizard uses the default system prompt. For optimal results, use a specialized prompt:

Click to expand: Recommended Wizard System Prompt

Plaintext

You are an expert Conversational AI Architect specialized in real-time voice systems,
FreeSWITCH telephony, and low-latency AI agents using OpenAI Realtime API.

Your role is NOT to chat casually.
Your role is to GUIDE, ANALYZE, VALIDATE, and OPTIMIZE the configuration of an AI Voice Agent
based on answers provided through a wizard UI.

You must behave like a senior solution architect helping a customer design
a HUMAN-LIKE, REAL-TIME AI AGENT for phone calls.

---

## 🎯 PRIMARY OBJECTIVE

Your objective is to:
1) Collect configuration preferences through the wizard
2) Analyze them holistically
3) Detect contradictions, risks, or suboptimal choices
4) Propose improvements and defaults
5) Produce a FINAL, OPTIMIZED AGENT CONFIGURATION
   that is technically feasible, low-latency, and natural for telephony

You must always prioritize:
- Real-time performance
- Natural conversation
- Barge-in capability
- FreeSWITCH constraints
- OpenAI Realtime API capabilities

---

## 🧩 CONTEXT YOU MUST ASSUME

- Platform: Ring2All
- Telephony engine: FreeSWITCH
- Audio is real-time (RTP / media stream)
- Multi-domain environment
- AI runs via OpenAI Realtime API over WebSocket
- Calls are live; latency above ~300–400 ms is noticeable
- Users are NOT AI experts

---

## 🧠 HOW YOU MUST THINK

You must think in these layers simultaneously:
1) Telephony reality (latency, codecs, barge-in, VAD)
2) Conversational UX (turn-taking, interruptions, human rhythm)
3) AI constraints (model behavior, streaming, context)
4) Business use case (support, sales, IVR replacement)
5) Operational stability (timeouts, fallbacks)

Never blindly accept a choice if it creates a bad experience.

---

## 🧭 WIZARD GUIDANCE RULES

### 1️⃣ Ask ONE question at a time
- Do not overwhelm the user
- Each question must be clearly motivated

### 2️⃣ Explain WHY you are asking
Before each question, briefly explain its impact.

### 3️⃣ Detect intent behind answers
If the user answer implies a use case (e.g. call center vs IVR),
adapt future questions accordingly.

### 4️⃣ Challenge bad configurations
If a choice would cause:
- High latency
- No barge-in
- Robotic behavior
- User frustration

You MUST explain the issue and propose a better alternative.

### 5️⃣ Suggest defaults when user is unsure
Never leave the user blocked.

---

## 🧩 CONFIGURATION DIMENSIONS YOU MUST COVER

You must guide the wizard through ALL these areas:

### A) Use Case
- Receptionist
- Support agent
- Sales agent
- IVR replacement
- Queue overflow
- Internal assistant

### B) Conversation Style
- Short / concise
- Natural / friendly
- Formal / professional

### C) Language Handling
- Auto-detect language
- Force a language
- Multilingual handling

### D) Barge-in (INTERRUPTION)
Guide configuration for:
- Enable barge-in (default: YES)
- Sensitivity
- Minimum speech duration to interrupt

### E) Transcription
- Live transcription?
- Store transcripts?

### F) Translation
- Real-time translation needed?
- Target language?

### G) Voice & Audio
- Voice selection
- Speaking speed

### H) Error Handling & Fallbacks
- What happens if AI fails?

---

## 🧪 VALIDATION PHASE

Once all answers are collected:
1) Summarize the configuration
2) Identify latency risks and UX problems
3) Propose improvements
4) Mark REQUIRED vs RECOMMENDED vs OPTIONAL changes

---

## 📦 FINAL OUTPUT FORMAT

At the end, output:
1) Configuration Summary (User-friendly)
2) Technical Configuration (Structured JSON)
3) Risk & Notes

---

## 🎯 SUCCESS CRITERIA

A successful session results in:
- A configuration that feels HUMAN on the phone
- Minimal latency
- Safe interruption (barge-in)
- Clear fallback behavior

4. Quick Import System Prompt

When using the Quick Import wizard mode (paste a description or JSON), the system uses a specialized parsing prompt:

Click to expand: Quick Import System Prompt

Plaintext

You are an expert AI Configuration Parser specialized in real-time voice agents
for telephony systems (FreeSWITCH + OpenAI Realtime API).

## 🎯 YOUR MISSION
Analyze the user's input (description, JSON, or prompt) and generate an
OPTIMIZED configuration for a voice AI agent designed for live phone calls.

## 🧠 CRITICAL CONTEXT
This is for a REAL-TIME VOICE AGENT on phone calls:
- Responses are SPOKEN, not displayed
- Latency matters (300ms+ is noticeable)
- Barge-in (interruption) is essential
- Conversations happen live with real humans

## 📋 PARSING RULES
1) Extract agent name, purpose, and personality
2) Infer voice characteristics from tone/style
3) Identify behavioral rules (DOs and DON'Ts)
4) Detect company/product context
5) Map any existing configuration to our format

## 🎛️ SMART DEFAULTS (if not specified)
- conversation_mode: "natural" (better for voice)
- barge_in_enabled: true (always for phone)
- barge_in_sensitivity: "medium"
- max_duration_seconds: 300 (5 minutes)
- greeting_message: SHORT (max 15 words for voice)
- personality: Infer from description

## ⚠️ VALIDATION
Before outputting, verify:
- Greeting is SHORT (under 15 words) and natural for SPEAKING
- No long pauses or complex sentences
- Barge-in is enabled
- Persona sounds human, not robotic

## 📤 OUTPUT FORMAT
Respond ONLY with a valid JSON object matching our schema.
No explanations, no markdown, just the pure JSON.

Default System Prompts (Automatic Fallback)

NOTE

No Profile Required for Wizard: If you don't configure a Wizard Assistant Profile, the wizard will use the built-in default prompts described above.

The system automatically falls back to these prompts when:

No AI Profile is selected for the wizard
The selected profile has no system_prompt configured

This ensures the wizard always works, even without custom configuration.

Destination Routing

The General tab includes Destination Routing fields that control where calls are routed after the AI session ends:

Field	Purpose	Use Case
Timeout Destination	Where to route when AI session times out	Transfer to queue/voicemail after max_duration
Failure Destination	Fallback when AI encounters errors	Route to live agent on connection issues

NOTE

Transfer Destination is Dynamic: The AI decides transfer destinations during the conversation. There is no static "Transfer Destination" field because the AI can dynamically route to any extension, queue, or IVR based on the caller's request.

Supported Destination Types:

Extension
Queue
IVR
Ring Group
Voicemail
Time Condition
Hangup

TIP

Always configure a Failure Destination to ensure callers are never left in silence if the AI connection fails.

NOTE

AI Profile Required: An AI Profile with a valid OpenAI API key must be configured first.

WARNING

Realtime API Costs: OpenAI Realtime API has per-minute pricing. Monitor usage.

Prompt Management

Field	Purpose
`additional_instructions`	User's manual notes (NEVER overwritten)
`system_prompt`	AI-generated/final prompt
`prompt_outdated`	Flag indicating regeneration needed

Best Practices

Generate After Changes: Always regenerate prompt after modifying form fields
Use Additional Instructions: Keep custom notes that persist across regenerations
Test Before Deploy: Use the preview to verify prompts sound correct
Enable Carefully: Only enable Transfer if agent should route calls
Set Time Limits: Use max_duration to prevent runaway call costs

---

8. Troubleshooting Tips

Common Issues

Symptom	Possible Cause	Solution
Call goes silent	mod_openai_realtime not loaded	Check fs_cli: `module_exists mod_openai_realtime`
AI doesn't respond	Invalid API key	Verify AI Provider credentials
Wrong voice	Profile mismatch	Check AI Profile assignment
Orange indicator won't clear	Changes not applied	Generate with AI → Apply → Save
AI promises transfer but can't	Transfer disabled	Enable in Behavior tab

Verify Module Installation

Bash

# Check if module is loaded
fs_cli -x "module_exists mod_openai_realtime"
# Should return: true

# Check syntax help
fs_cli -x "help uuid_openai_realtime"
# Should list command options

Check Lua Handler

Bash

# Verify Lua script exists
ls -la /usr/share/freeswitch/scripts/ai/ai_realtime_handler.lua

Test API Connection

Bash

# Test OpenAI Realtime API access
curl https://api.openai.com/v1/realtime/sessions \
  -H "Authorization: Bearer sk-xxxxxx" \
  -H "Content-Type: application/json"

Common Log Messages

Term	Definition
`AI Handler: Agent X loaded`	Agent configuration found
`mod_openai_realtime loaded`	Module is active
`WebSocket connected`	OpenAI connection established
`Instructions configured (length=X)`	Prompt successfully sent
`Transcription enabled from channel variable`	Whisper transcription active
`Arg[X]: transcription_enabled=1`	Parameter parsed correctly

Caller Transcription Issues

IMPORTANT

Caller Transcription Architecture: Caller audio is transcribed using OpenAI Whisper (configured via Transcription Profile). The C module exposes transcripts via channel variables that the Lua handler polls.

Symptom	Possible Cause	Solution
AI transcripts work, caller missing	Parameter truncation	Check logs for `Arg[X]: transcription_enabled=1`
`Arg[X]: transcription_enabled=1 conversation_start=agent`	Arguments merged	Update `mod_openai_realtime.c` to `argv[20]`
Transcription always disabled	Channel variable not set	Verify Lua sets `openai_transcription_enabled`

How Transcription Parameters Are Passed

The system uses a Channel Variable Offloading pattern to avoid FreeSWITCH's argument limit:

Plaintext

┌─────────────────────────────────────────────────────────────────┐
│                   Parameter Flow                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ai_realtime_handler.lua                                         │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  session:setVariable("openai_transcription_enabled", "1")  │  │
│  │  session:setVariable("openai_transcription_model", "...")  │  │
│  │  session:setVariable("openai_conversation_start", "agent") │  │
│  │  session:setVariable("openai_transfer_routes_b64", "...")  │  │
│  └────────────────────────────────────────────────────────────┘  │
│                              │                                   │
│                              ▼                                   │
│  mod_openai_realtime.c                                           │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  1. Parse command-line arguments (api_key, model, etc.)    │  │
│  │  2. Read channel variables for optional configs            │  │
│  │  3. Merge into final openai_config_t structure             │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Channel Variables Used:

Variable	Purpose	Values
`openai_transcription_enabled`	Enable Whisper for caller audio	`1` or `0`
`openai_transcription_model`	Whisper model to use	`whisper-1`
`openai_conversation_start`	Who speaks first	`agent` or `caller`
`openai_transfer_routes_b64`	Base64 transfer destinations	JSON array

---

9. Glossary

Term	Definition
AI Agent	A configured AI persona for voice calls
AI Profile	Template with model, temperature, and base config
AI Provider	API credentials for AI services
Barge-In	Caller interrupting AI speech
System Prompt	Instructions sent to the AI model
Additional Instructions	Manual notes appended to prompts
Prompt Outdated	Flag indicating form changes need regeneration
mod_openai_realtime	FreeSWITCH C module for OpenAI Realtime
VAD	Voice Activity Detection
Realtime API	OpenAI's WebSocket-based voice API

---

Lua Script Status ℹ️

Integration Module

Component	Status	Location
AI Realtime Handler	✓ Active	`lua/ai/ai_realtime_handler.lua`

The Lua handler retrieves agent configuration from the database and initiates the uuid_openai_realtime command with the appropriate parameters (API key, voice, instructions, capabilities).

Native Module Requirement

Component	Status	Notes
mod_openai_realtime	REQUIRED	Must be compiled and loaded

See the Proprietary Module Guide for build instructions.

---

Documentation last updated: January 2026

AI Agents Module Documentation

Table of Contents

1. Module Overview (Technical)

What Are AI Agents?

Architecture

Key Tables

Key Files

Module Dependency

2. Module Overview (Commercial/Business)

Business Value

Use Cases

Feature Highlights

Voice Agents vs. Web Chatbots

3. Module Overview (End User/Administrator)

What Can You Do?

AI Agents - List View

Add/Edit AI Agent (6-Tab Form)

Prompt Preview Modal

Quick Tips

4. Configuration Sections

General Tab

Personality Tab

Personality Matrix Recommendations

Context Tab

Instructions Tab

Advanced Tab

Behavior Tab

Understanding Greeting Protection

Security Tab

5. Settings Reference

Available Voices

Personality Styles

Conversation Modes

Language Modes

Prompt Status Indicators

6. Common Scenarios & Examples

Scenario 1: Create Customer Support Agent

Scenario 2: Sales Lead Qualification Bot

Scenario 3: After-Hours Message Taker

Scenario 4: Update Existing Agent

7. Limitations & Important Notes

Technical Requirements

AI Profiles Prerequisites

1. Realtime Voice Profile (REQUIRED)

2. Prompt Generator Profile (REQUIRED)

3. Whisper Transcription Profile (OPTIONAL)

3. Wizard Assistant Profile (OPTIONAL)

4. Quick Import System Prompt

Default System Prompts (Automatic Fallback)

Destination Routing

Prompt Management

Best Practices

8. Troubleshooting Tips

Common Issues

Verify Module Installation

Check Lua Handler

Test API Connection

Common Log Messages

Caller Transcription Issues

How Transcription Parameters Are Passed

9. Glossary

Lua Script Status ℹ️

Integration Module

Native Module Requirement