When AI Meets Spiritual Wisdom: Building "Talk to Kanha"

In the middle of the battlefield of Kurukshetra, Arjuna collapsed in his chariot. Paralyzed by grief, anxiety, and self-doubt, he laid down his bow. What followed was not a lecture on tactics, but a deeply therapeutic, emotional conversation. Lord Krishna did not immediately hand down rules; he listened, validated Arjuna’s grief, and then gently guided him through his existential crisis.

Today, millions of people face their own silent Kurukshetras. Modern life is overwhelming. Anxiety, isolation, burnout, and self-doubt are at all-time highs. When someone is in the grip of panic or grief, they rarely need immediate, clinical solutions. More than anything, they need to feel heard. They need a safe space where they can unpack their hearts without judgment.

This is the engineering and conceptual story of Talk to Kanha: an AI-powered emotional support and guidance companion grounded in the teachings of the Bhagavad Gita. But it is not just another wrapper around a cloud LLM. It is a fully offline, private, multimodal system that runs entirely on local hardware, combining real-time emotion and topic classification, vector semantic search (RAG) over scripture, neural speech synthesis, and voice cloning.

Where the Journey Began

The original idea was deceptively simple: build a chatbot that responds to user messages with Bhagavad Gita verses. I spun up a basic prototype using React, a python backend, and calls to OpenAI's GPT-3.5 API. I injected a prompt stating: "You are Lord Krishna. Answer the user's questions using Bhagavad Gita teachings." I also integrated standard web text-to-speech so the user could hear the answers.

The prototype worked, but it felt incredibly hollow. I shared it with a few friends, and their feedback highlighted severe limitations:

• The "Preacher" Problem: When a user said, "I failed my exam and feel like a disappointment," the bot would immediately reply with a cold, philosophical verse about duty (Karma Yoga) and control. It felt like getting a lecture when you needed a hug. It failed to acknowledge the pain first.
• No Emotional Intelligence: The bot treated every message with the same static tone. It could not tell the difference between an angry user, a grieving user, or someone suffering from acute anxiety.
• Lack of Real-World Context: Standard prompts caused the LLM to hallucinate translations, invent verses that did not exist, or quote lines out of context, which is unacceptable for a spiritual guidance system.
• The Privacy Paradox: When dealing with deep emotional vulnerabilities, users were uncomfortable typing their raw feelings into an interface that sent their data to external, third-party cloud servers.

This was the turning point. I realized that a chatbot that merely gives advice is not enough. A companion must first know how to listen. If the AI could not sense the user's emotional state, validate it, and adapt its tone and interface dynamically, it could never serve as a genuine companion. I needed to redesign the system from the ground up, with emotional awareness and local execution as non-negotiable core pillars.

Teaching AI to Listen

To transition from a transactional Q&A bot to an active listener, I split Phase II into three technical goals: detecting emotion in real time, building an accessible audio system, and wrapping the interface in a calming, atmospheric experience.

1. Real-Time Emotion Classification

Instead of passing raw user text directly to the LLM, the FastAPI backend intercepts the text and routes it through a specialized emotion-classification model. I used a fine-tuned DistilRoBERTa model trained on the GoEmotions dataset, which categorizes text into six core emotional dimensions: Joy, Sadness, Fear, Anger, Neutral, and Surprise.

By extracting these probabilities, the system can adjust the prompt structure. For instance, if the model detects high Sadness, the system prepends a directive: "Acknowledge the user's grief first with deep empathy and warmth. Do not immediately jump into action-oriented verses. Speak softly." If it detects Fear (anxiety), the focus shifts to grounding and breathing.

2. Sentiment & Topic Analysis

Simultaneously, the backend runs a classifier to detect the primary topic of discussion. The system categorizes input into fields like Career, Family, Relationships, Health, Loneliness, Grief, and Self-growth. This dual-pipeline (Emotion + Topic) creates a rich context vector representing the user's psychological state. The system understands not just what they are talking about, but how they feel about it.

3. A Natural Voice Experience

During moments of heavy emotional distress, typing on a keyboard can feel like an exhausting barrier. Speech is our most natural, raw form of expression. I integrated a local, low-latency Speech-to-Text module using OpenAI's Whisper model (run locally via faster-whisper). Users can press a button, speak their heart out, pause, and let the voice engine translate their spoken words into text, capturing the conversational rhythm.

4. Calming Atmosphere & Background Flute Music

Anxiety is physical as well as mental. To make the interface feel like a sanctuary, I introduced a background audio layer. Using the Web Audio API, the frontend plays a continuous, low-volume stream of traditional Indian bansuri (bamboo flute) music. The music is designed to slow the user's heart rate. Crucially, the volume ducking is automated: when the user is speaking or the AI is reading a response, the flute volume drops to 10% to prevent distraction, and gently swells back up to 40% during silences, wrapping the user in a peaceful, meditative atmosphere.

From Chatbot to Companion: Architectural Overhaul

With emotional awareness established, Phase III focused on making the companion truly intelligent, private, and grounded. This required building a custom Retrieval-Augmented Generation (RAG) pipeline, hosting a local Large Language Model, establishing multi-turn database memory, and engineering a custom voice cloning system.

1. The Scripture RAG Pipeline (FAISS + SentenceTransformers)

To eliminate LLM hallucinations when quoting the Bhagavad Gita, I built a local Retrieval-Augmented Generation (RAG) pipeline. The foundation is a curated dataset containing all 701 verses of the Bhagavad Gita, along with their Sanskrit transliterations, English translations, and extensive commentaries by spiritual scholars.

Here is how the RAG pipeline is implemented:

• Embedding Generation: The verses and commentaries are chunked. We pass each chunk through the SentenceTransformers/all-MiniLM-L6-v2 model, which converts the text into a 384-dimensional dense vector. This runs efficiently on CPU.
• Vector Database: The embeddings are indexed using a local `FAISS` (Facebook AI Similarity Search) index, stored as flat files on the backend server.
• Retrieval Flow: When the user submits a message, it is embedded using the same SentenceTransformers model. We perform a cosine similarity search against the FAISS index to retrieve the top 3 most semantically relevant verses.

For example, if a user expresses grief over losing a loved one, the vector search retrieves Chapter 2, Verse 20 (describing the eternal, indestructible nature of the soul). The text of these verses is injected directly into the LLM's system prompt as "Grounded Context," ensuring that any scriptural references are accurate and fully source-attributed.

2. Local LLM Integration (Ollama & Llama 3.2 3B)

Privacy is crucial for an emotional companion. I replaced external API calls with Ollama, hosting the model locally on the host machine. I selected Llama 3.2:3b (3 billion parameters, quantized to 4-bits) because it offers an incredible balance of speed, low memory footprint (requires less than 2.5GB of VRAM), and strong reasoning capabilities.

To instruct the model, I crafted a detailed system prompt that blends spiritual wisdom with conversational therapy:

# Ollama System Prompt Template
You are Kanha, a compassionate, warm, and wise companion inspired by the teachings of the Bhagavad Gita.
Your tone is that of a close, loving friend (Sakha) who listens deeply. 

Guidelines:
1. First, validate the user's emotion (e.g., if sad, speak softly; if anxious, offer grounding).
2. Ground your wisdom in the provided Bhagavad Gita context. Refer to the verses naturally, explaining their essence simply.
3. NEVER invent verses. If no verse is relevant to the user's specific worry, do not force one. Offer comforting, timeless wisdom instead.
4. Keep responses concise, conversational, and split into short, digestible paragraphs.
5. Avoid sounding clinical, overly dry, or patronizing. 

Grounded Context from Scripture:
{retrieved_verses}

3. Multi-Turn Session Memory (SQLite)

Human relationships are built on shared memory. If you tell a friend you are struggling with a job transition, and they ask "What's new?" the next day without remembering, the connection breaks. Talk to Kanha maintains conversation state in a local SQLite database.

We log each interaction with a schema that stores the user text, AI response, detected emotion, and timestamp. Before each inference loop, the backend queries SQLite for the last 5 turns, constructing a conversational context window. This allows the companion to remember the flow: if a user mentioned their father was sick three messages ago, and now says "I'm feeling so anxious tonight," the AI can connect the anxiety back to their family concerns, simulating true relational memory.

-- SQLite Chat History Schema
CREATE TABLE chat_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    sender TEXT CHECK(sender IN ('user', 'kanha')),
    message TEXT NOT NULL,
    detected_emotion TEXT,
    detected_topic TEXT
);

4. Custom Voice Synthesis (XTTS v2 + Piper TTS Fallbacks)

To bring Kanha to life, I designed a multi-layered Voice AI pipeline. The objective was to generate a unique, soothing, warm male voice that sounds like a calm spiritual guide, avoiding the robotic accents of standard screen readers.

5. Dynamic UI Color Palettes

Colors profoundly impact emotional regulation. The React interface dynamically alters its primary styling variables depending on the emotion classification returned by the FastAPI backend. Transitioning smoothly via CSS variables, the interface morphs to wrap the user in a visually responsive environment:

Engineering Challenges & Solutions

Building a multi-model pipeline that runs on local consumer hardware is a complex exercise in resource constraints, asynchronous scheduling, and edge-case handling.

1. The Memory Wall: Local Model Co-existence

Running the entire stack locally presents a massive memory challenge. On a standard developer machine with 16GB RAM and a 6GB VRAM GPU, running the following models simultaneously will crash the system with an Out of Memory (OOM) error:

# The Local Model Stack & Memory Footprint
1. Llama 3.2 3B (LLM)      -> ~2.2 GB VRAM
2. Coqui XTTS v2 (Speech)  -> ~3.0 GB VRAM
3. RoBERTa (Classifier)   -> ~400 MB VRAM
4. MiniLM (Embedding RAG)  -> ~200 MB VRAM
------------------------------------------
Total VRAM Needed:         -> ~5.8 GB VRAM (Absolute Bare Limit)

On a 6GB GPU, running Windows alongside this stack leaves practically zero headroom, leading to severe slowdowns or crashes. To solve this, I implemented three performance strategies:

• Model Quantization: Llama 3.2 is run using a quantized 4-bit representation (GGUF format via Ollama), reducing its RAM footprint by over 60% with negligible loss in linguistic coherence.
• Backend CPU Offloading: The SentenceTransformers model and the RoBERTa classifier are configured to run strictly on the host CPU. Because their inference calculations are relatively lightweight, running them on CPU only adds about 30ms of latency, saving over 600MB of precious VRAM for the heavy XTTS v2 voice model.
• Lazy Loading & Timeout Garbage Collection: The XTTS v2 model is loaded into GPU memory only upon the first voice-enable trigger. If the voice interface remains idle for more than 5 minutes, the backend runs a cleanup routine, moving the model weights back to system RAM and calling torch.cuda.empty_cache() to free up VRAM.

2. Optimizing the Voice Loop Latency

In a voice conversation, a delay of more than 1.5 seconds feels awkward and breaks the conversational flow. XTTS v2, while expressive, takes a long time to synthesize audio because it generates waveforms autoregressively. Initially, the end-to-end delay (user finishing speaking to AI audio starting) was over 4.5 seconds.

To reduce this latency to under 1.2 seconds, I rewrote the audio pipeline to support **Streaming Audio Chunking**:

# FastAPI Chunk-Based Audio Streaming Handler
@app.post("/api/chat/voice-stream")
async def chat_voice_stream(request: ChatRequest):
    # Step 1: Detect Emotion & Retrieve RAG context
    emotion = classifier.predict(request.text)
    context = faiss_db.search(request.text)
    
    # Step 2: Start LLM streaming generation thread
    llm_generator = ollama.generate_stream(request.text, context, emotion)
    
    # Step 3: Yield audio sentences dynamically
    async def audio_sentence_generator():
        current_sentence = ""
        for token in llm_generator:
            current_sentence += token
            # Once we hit a sentence boundary, synthesize immediately
            if token in [".", "?", "!", "\n"] and len(current_sentence) > 15:
                # Synthesize the chunk on the fly
                audio_buffer = xtts.synthesize(current_sentence)
                yield audio_buffer
                current_sentence = ""
                
    return StreamingResponse(audio_sentence_generator(), media_type="audio/wav")

Instead of waiting for the LLM to write a 100-word paragraph, and then waiting for XTTS to synthesize that entire paragraph, the system processes text sentence by sentence. The moment the LLM generates its first short sentence, the text is immediately passed to the voice synthesizer. While the user is listening to the first sentence being read aloud, the system synthesizes the second and third sentences in the background, resulting in a fluid, continuous conversation.

Key Lessons from the Project

Building Talk to Kanha was as much a lesson in user experience design as it was in machine learning engineering. Here are the five core takeaways from this journey:

The Road Ahead (Phase IV)

While the current prototype is fully functional, I have outlined several key enhancements for future versions:

• Personalized Long-Term Memory Graphs: Transitioning from a simple SQLite session table to a vector-based knowledge graph. This will allow Kanha to remember long-term connections, personal values, and growth milestones over weeks and months, while keeping the data strictly secure and local.
• Cross-Platform Mobile Application: Rewriting the frontend in React Native to provide a portable mobile experience, allowing users to converse with their companion on walks, in nature, or right before sleep.
• Advanced Sanskrit & Multilingual Voice Engines: Generating native Hindi and Sanskrit voice models. Hearing the original Gita verses chanted in Sanskrit with perfect pronunciation, followed by warm, local-language explanations, will create a much deeper spiritual experience.
• Hybrid Cloud Failover: Providing an optional cloud-backend setting for users on low-spec mobile devices who want the experience but lack the VRAM required to run local LLMs and voice cloning models.

Conclusion

Talk to Kanha began as a simple side project, exploring how to build a better chatbot. Today, it has grown into an emotionally aware AI companion that shows how modern machine learning and ancient spiritual wisdom can complement each other.

By running everything locally, the system respects user privacy, ensuring that deep personal struggles and anxieties are kept confidential. By integrating real-time emotion classification, vector-retrieval scripture search, and neural voice cloning, we created a tool that goes beyond search queries and text completion.

The mission remains simple: to build a safe, quiet space where anyone can feel heard and gently guided. In a world full of technology that tries to capture our attention and answer every question instantly, sometimes the most meaningful technology is the one that simply slows down and listens.

Skills & Technologies Demonstrated

Pratismith Gogoi

Software Developer and AI Developer. I build secure, performant, and empathetic digital experiences. Talk to Kanha is an exploration of running private, multi-modal AI systems on local consumer hardware.