An intelligent conversational chatbot for the SRI Congress

The first version of the retrieval system used Azure AI Search as the vector database. At the contracted tier, that service imposed a 60 MB limit on indexes. When converting the SRI Congress's historical content into embeddings, the data volume consistently exceeded that limit, which blocked full content uploads and forced information to be truncated.

Operating cost was the second factor behind the change. Azure AI Search carried a fixed cost from the very start of the project, at a stage when the product and the data volume were not yet stable. Supabase, by contrast, offered a generous free tier that covered the project's needs at that early stage, with the option to scale up to a paid plan later, once the product and the data had reached a more consistent volume. That flexibility — starting at no cost and growing only once the project justified it — weighed as much as the technical 60 MB limitation.

The team evaluated Supabase as an alternative: beyond offering a full Postgres database, it includes the pgvector extension to store and query embeddings directly within the same relational database, with no size ceiling like Azure's. The migration eliminated the capacity limitation entirely, reduced operating costs in the project's early phase, and, in the process, simplified the architecture by unifying both relational and vector data in a single database.

Before responding, every user question is evaluated by an OpenAI model whose sole job is to determine the intent behind the message. The system recognizes four categories:

• Database queryQuestions that can be answered with a precise query over structured data, such as the number of sessions in a specific month and year.
• Event informationQuestions about what happened in a past session, answered with a summary generated from stored content.
• Website informationQuestions about current content published on the official page, such as the date of the next event, resolved through the scraping service.
• Greeting or general conversationMessages such as greetings, which get a direct conversational reply without triggering any of the routes above.

This upfront classification keeps the bot from having to "guess" the right source, and lets each type of question be resolved with the most appropriate and most response-time-efficient method.

To answer questions about the website, the team first evaluated n8n's native scraping node, but ruled it out for two concrete reasons.

First, the native node didn't navigate on its own across the site's different menu sections (home, program, etc.): it only pulled content from the URL explicitly given to it. If the answer lived on another page of the same site, the bot couldn't find it, and it kept returning the same link or a "no relevant information found" message.

Second, a large share of the site's relevant content was embedded inside <script> tags, and the native node's HTML-to-markdown sanitization step discarded exactly those tags, losing information that was actually relevant to the user.

The solution was to build a custom Python microservice, exposed as an API, that uses Scrapling — an open-source scraping library available on GitHub — instead of the native node. The service works in four steps: first it extracts all the links available on the page; then it asks an OpenAI model to pick which of those links is most likely to contain the answer to the user's question; next it automatically navigates to that page and extracts its full text, excluding scripts, styles, menus, and footers while keeping the actual content; and finally, the same OpenAI model analyzes that text and drafts the response. The result is cached for 24 hours to avoid repeating navigation unnecessarily.

This approach solved both the cross-page navigation problem and the loss of content embedded in scripts — two limitations the native node didn't cover.