Background:
In my previous article I explained how Plaudere's support agent was built using Qwen (Alibaba) and the fuse library. That design worked, but it also showed me its limits very quickly. The fuse search is still a lexical approximation of meaning. When a user wrote something with different words than the text, the retrieval layer either missed it or returned something loosely related, and the Small Language Model (SLM), trying to be helpful, filled the gap with an invention. I used to blame those answers on the model. After spending some time reading logs I now think that a very large share of the "hallucinations" were not generation problems at all: they were routing problems. The right text existed, but the system sent the question to the wrong source, or to no source, and the SLM improvised. This article is about how I replaced the old fuse approach with MiniLM.
About MiniLM:
Paraphrase Multilingual MiniLM is a compact sentence embedding model from Microsoft. It does not write text. It converts a sentence into a vector so that sentences with a similar meaning end up close to each other, even if they share no words. That is exactly what the old fuse with some semantic algorithms were trying to imitate by hand. The model is small enough to run inside the same Node.js environment, but I had to implement a FastAPI within the infrastructure to ensure scalability. The division of work is now very clear. MiniLM understands the question and decides where the answer should come from. Qwen only writes the final sentence, and only when there is real evidence to write it from. Each model does the single thing it is good at.
Intent detection in two stages:
Every question goes through two MiniLM assessments before the SLM sees anything. Stage 1 compares the question against a set of prototype phrases written for each language (Spanish, English and Catalan). Each group of phrases represents one intent: greeting, farewell, thanks, about the assistant, conversation history, explicit web search, and the three summaries (website for any page within Plaudere, article and user profile). The most similar phrase wins and gives a fixed score. Stage 2 asks a different question: which source actually contains the answer? The four candidates are the article, the user profile, the website information and the general context. MiniLM scores the question against the first sentences of each source, and the final score of a source is a weighted mix of both stages, one for fixed intent and another for content search. The article receives a small bias, because a question asked while reading an article is usually about that article. If nothing reaches a minimum score, the question is treated as noise. Some cases never reach Stage 2. A clear greeting, a thank you or a request to summarise the article are answered through fast paths, which saves time and removes the risk of the SLM inventing something for a trivial request. Speed mattered a lot here: only a handful of sentences per source are embedded, and conversational intents are not even assessed when the question contains real content.
Refining the Routing and Context:
The architecture looked sound in theory, but operational deployment brought subtle edge cases that required architectural refinement. For instance, distinguishing between a broad request to summarise an article and a specific question containing similar terminology led to the concept of pure commands. A query is treated as a pure command only when its components strictly map to the language's native intent vocabulary, which is generated automatically from string assets rather than hardcoded rules. If a user introduces external terms, the system seamlessly shifts to targeted content retrieval.
Similarly, external web contexts required robust sanitisation. External sources often introduce formatting artifacts or numeric references, which are neutralised through dedicated cleaning steps and stem-based filtering to ensure only relevant sentences are retained. Conversation history management was also optimised to prevent auxiliary system messages, such as error states or timeouts, from contaminating the dialogue context. Furthermore, lexical scoring was transitioned from a binary bonus to a proportional metric based on token frequency, providing a more balanced evaluation of source relevance. Finally, routing fallback logic was refined so that default paths are only triggered when valid fallback text is genuinely present, ensuring the agent consistently leverages the highest-quality source available.
Helping an SLM model behave:
Even with better routing, a model of this size needs guardrails. The prompt asks for a single complete sentence that includes the subject, which stopped the loose fragments. After generation, a grounding check compares the answer with the evidence: any number that does not appear in the evidence or the question is considered invented, and so are words echoed from the question that the evidence never states. When the check fails and evidence exists, the agent returns the best evidence sentence instead of a blank apology. For web snippets, which are already factual sentences, it is safer to return the best one directly than to let a small model rewrite and corrupt it.
Conclusions and Future Improvements:
My conclusion is that the quality of a small RAG agent depends less on the generator than on everything that happens before it. Understanding the intent, choosing the right source and cleaning the context did more for accuracy than any prompt change. MiniLM gave Plaudere a real semantic layer, and Qwen now has a much smaller chance to invent, because it is rarely asked to answer without evidence. Future goals include, among others:
-Discarding irrelevant web results earlier, before they reach the context.
-Caching embeddings of the static intent phrases to reduce latency even more.
-Testing newer Qwen versions now that the retrieval side is stable.
-Keeping the three languages consistent in both intent phrases and answers.
This project keeps confirming the same idea: with clever engineering and a lot of patient log reading, small models can deliver useful and reliable AI on very limited hardware.
