Scrape and ingest web content into Supabase pgvector with Firecrawl

What this does

Receives a URL via webhook, uses Firecrawl to scrape the page into clean markdown, and stores it as vector embeddings in Supabase pgvector. A visual, self-hosted ingestion pipeline for RAG knowledge bases. Adding a new source is as simple as sending a URL.

The second part of the workflow exposes a chat interface where an AI Agent queries the stored knowledge base to answer questions, with Cohere reranking for better retrieval quality.

How it works

Part 1: Ingestion Pipeline

1. Webhook receives a POST request with a url field 2. Verify URL validates and normalizes the domain 3. Supabase checks if the URL was already ingested (deduplication) 4. If the URL already exists, ingestion is skipped; otherwise it continues 5. Firecrawl fetches the page and converts it to clean markdown 6. OpenAI generates vector embeddings from the scraped content 7. Default Data Loader attaches the source URL as metadata 8. Supabase Vector Store inserts the content and embeddings into pgvector 9. Respond to Webhook confirms how many items were added

Part 2: RAG Chat Agent

1. Chat trigger receives a user question 2. AI Agent (OpenRouter) queries the Supabase vector store filtered by URL 3. Cohere Reranker improves retrieval quality before the agent responds 4. Agent answers based solely on the ingested knowledge base