Build a local RAG chatbot with Ollama, Qwen, BGE-M3 and Postgres PGVector

Build a fully local RAG chatbot using Ollama that works without tool calling — ideal for smaller open-source models like Qwen that don't support native function calls. This template lets you run a private, self-hosted AI assistant with retrieval-augmented generation using only your own hardware.

How it works

1. A Webhook receives the user's chat message 2. A small classifier LLM (Qwen 7B) analyzes the input and decides: is this small talk, or a real question that needs the knowledge base? 3. For small talk, a dedicated AI agent responds conversationally with chat memory 4. For real questions, the classifier generates focused sub-queries, which are sent through a loop-based RAG pipeline: - Each sub-query is embedded using BGE-M3 and matched against a Postgres PGVector store - Results are filtered by a relevance score threshold (>0.4) - Chunks are aggregated and deduplicated across all sub-queries 5. An Answer Generator agent (Qwen 14B) produces a sourced answer using a strict 3-step format: short answer → sources → follow-up question 6. Both paths use Postgres-backed chat memory for multi-turn conversations 7. A post-processing step removes <think> tags that some reasoning models produce

Set up steps

1. Install Ollama and pull the required models: - ollama pull qwen2.5:7b (classifier + small talk) - ollama pull qwen3:14b (answer generation) - ollama pull bge-m3 (embeddings) 2. Set up PostgreSQL with the pgvector extension enabled 3. Create your vector store — ingest your documents into the PGVector store using BGE-M3 embeddings (you can use n8n's built-in document loaders for this) 4. Configure credentials in n8n: - Ollama connection (default: http://localhost:11434) - PostgreSQL connection for both chat memory and vector store 5. Customize the webhook path and connect it to your frontend or API client 6. Optional: Adjust the relevance score threshold, swap models for larger/smaller ones, or modify the system prompts to match your use case