The SAP HANA Cloud vector engine for custom retrieval

Managed document grounding covers many cases. Sometimes you need more control: your own ranking, joins between vectors and business data, or filters on columns such as plant or release. For that, SAP HANA Cloud has a built-in vector engine. You store embeddings in table columns, query them with SQL, and combine similarity with ordinary WHERE clauses. This lesson covers the pieces you need and the gotchas that SAP's Vector Engine Guide calls out.

Vector data types

SAP HANA Cloud has two vector types. REAL_VECTOR(n) stores single-precision (32-bit) floating-point elements; HALF_VECTOR(n) stores half-precision elements, which halves storage at the cost of precision and range. The optional dimension n can be anything from 1 to 65,000. If you leave it out, the column accepts vectors of any valid dimension, which is convenient for experiments and risky in production, because vectors from two different embedding models can't be compared meaningfully.

Creating embeddings inside the database

You can create embeddings outside HANA (for example, with an embedding model deployed in the generative AI hub) and insert them, or you can let HANA do it with the VECTOR_EMBEDDING function. The function takes the text, a text type and a model with version. The text type matters: use 'DOCUMENT' for stored passages and 'QUERY' for questions, because the model encodes them differently.

Without a remote source, the function uses HANA's internal natural language processing (NLP) service, which must be enabled. SAP lists models such as SAP_NEB.20240715 and SAP_GXY.20250407, both with 768 dimensions; query the NLP_VECTOR_EMBEDDING_MODELS system view for the current list. With a remote source that points to SAP AI Core, the same function calls an embedding model in the generative AI hub; the user needs EXECUTE on that remote source.

Here is the shape of Contoso's chunk table and a top-k query, written as plain SQL:
CREATE COLUMN TABLE CMC_RAG.DOC_CHUNKS (CHUNK_ID INTEGER, DOC_ID NVARCHAR(40), DOC_TYPE NVARCHAR(20), PROCESS NVARCHAR(20), SECURITY_GROUP NVARCHAR(40), CHUNK_TEXT NCLOB, EMB REAL_VECTOR(768));
SELECT TOP 5 DOC_ID, CHUNK_TEXT, COSINE_SIMILARITY(EMB, VECTOR_EMBEDDING(:question, 'QUERY', 'SAP_NEB.20240715')) AS SIM FROM CMC_RAG.DOC_CHUNKS WHERE PROCESS = 'O2C' AND SECURITY_GROUP IN (:user_groups) ORDER BY SIM DESC;

Similarity functions and indexes

Two similarity measures are built in. COSINE_SIMILARITY returns higher values for more similar vectors, so you sort descending. L2DISTANCE returns lower values for more similar vectors, so you sort ascending. Use the measure your embedding model was designed for; cosine is the usual choice for text.

Without an index, HANA compares the question vector with every row after applying your filters, which is exact but grows with the table. For large tables, create an HNSW (hierarchical navigable small world) index: CREATE HNSW VECTOR INDEX CMC_CHUNK_COS ON CMC_RAG.DOC_CHUNKS (EMB) SIMILARITY FUNCTION COSINE_SIMILARITY, optionally with a build configuration (such as M and efConstruction) and a search configuration (efSearch). SAP's guide stresses three points: the index is approximate and may not return the very best matches; each index serves one similarity measure, and a column can have at most one index per measure; and you can force an exact search with the NO_VECTOR_INDEX hint.

Managed grounding or your own HANA tables?