A/B test AI prompts with Supabase, Langchain Agent & OpenAI GPT-4o

 Split Test AI Prompts Using Supabase & Langchain Agent

This workflow allows you to A/B test different prompts for an AI chatbot powered by Langchain and OpenAI. It uses Supabase to persist session state and randomly assigns users to either a baseline or alternative prompt, ensuring consistent prompt usage across the conversation.

🧠 Use Case

Prompt optimization is crucial for maximizing the performance of AI assistants. This workflow helps you run controlled experiments on different prompt versions, giving you a reliable way to compare performance over time.

⚙️ How It Works

1. When a message is received, the system checks whether the session already exists in the Supabase table. 2. If not, it randomly assigns the session to either the baseline or alternative prompt. 3. The selected prompt is passed into a Langchain Agent using the OpenAI Chat Model. 4. Postgres is used as chat memory for multi-turn conversation support.

🧪 Features

- Randomized A/B split test per session - Supabase database for session persistence - Langchain Agent + OpenAI GPT-4o integration - PostgreSQL memory for maintaining chat context - Fully documented with sticky notes