Your Semantic Layer Has Descriptions. Now Give It Memory.

Pretty much all semantic layers allow to specify a “description” string for each entity (such as dimension or measure), giving extra context on what the entity means and what it’s to be used for. However, most of them don’t have great answers to two questions: firstly, how to store business rules or other context that doesn’t map neatly onto a single entity (for example, rules on when to use one of the two available distinct “revenue” definitions); and secondly, on how to leverage those descriptions, for example by discovering entities based on the descriptions.

‍

SLayer release 0.6 addresses exactly those two issues, by introducing a memories mechanism and a search tool.

‍

The memories mechanism stores an arbitrary string (such as a business rule on which one of the two available revenue definitions you are supposed to use when) and a list of fully qualified entity ids (such as datasource_name.model_name.measure_name). To store a memory, you can also supply raw dimension names or even a full query, and the tool will resolve/convert these to fully qualified names of relevant entities. You can also store a complete query definition along with the memory - sometimes an example works best.

‍

Now suppose you have great descriptions for individual columns, metrics, etc, and great memories linked to the right entities - how do you find the ones that matter for your particular question?

‍

There are two ways. Firstly, all memories linked to entities from a particular model are automatically included when you inspect that model. But what if you just have a question to answer, and don’t know the model you need?

‍

That’s where the search tool comes in. It retrieves both memories and entities, using no less than three different kinds of search and merging the results. You can supply to it a freeform question as well as a list of names of entities such as models, columns or measures (or only one of the two). If entity names are supplied, it looks for memories referencing the same entities; if a text string is supplied, it uses both embedding-based retrieval and full-text-search (BM25 via tantivy) to look for the memories and entities with the most relevant text representations. It then merges the retrieval results from the different methods using Reciprocal Rank Fusion.

‍

This gives us the best of all three worlds: retrieval by exact entity name, by exact keyword shared between the question and a memory or description, and by semantic similarity using embeddings.

‍

By the way, before the term RAG became popular, embedding retrieval was often referred to as “semantic search” - does that make SLayer the Doubly Semantic Layer? :)

‍