From a9a3f4a0650950ef4a28f598fcd4bf292399fd14 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 15 May 2025 10:08:51 +0200 Subject: [PATCH 1/9] Initial reference docs about the vector index --- .../features/community-edition.md | 2 +- site/content/3.13/aql/functions/vector.md | 95 +++++++++++ .../3.13/index-and-search/indexing/basics.md | 10 ++ .../indexing/which-index-to-use-when.md | 12 ++ .../working-with-indexes/vector-indexes.md | 156 ++++++++++++++++++ 5 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 site/content/3.13/aql/functions/vector.md create mode 100644 site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md diff --git a/site/content/3.13/about-arangodb/features/community-edition.md b/site/content/3.13/about-arangodb/features/community-edition.md index 9ee6ac647d..1d8f118af9 100644 --- a/site/content/3.13/about-arangodb/features/community-edition.md +++ b/site/content/3.13/about-arangodb/features/community-edition.md @@ -149,7 +149,7 @@ see [arangodb.com/community-server/](https://www.arangodb.com/community-server/) {{% /comment %}} {{% comment %}} Experimental feature -- [**Vector search**](#TODO): +- [**Vector search**](../../index-and-search/indexing/working-with-indexes/vector-indexes.md): Find items with similar properties by comparing vector embeddings generated by machine learning models. {{% /comment %}} diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md new file mode 100644 index 0000000000..a34f54d0da --- /dev/null +++ b/site/content/3.13/aql/functions/vector.md @@ -0,0 +1,95 @@ +--- +title: Vector search functions in AQL +menuTitle: Vector +weight: 60 +description: >- + The functions for vector search let you utilize indexed vector embeddings to + quickly find semantically similar documents +--- +To use vector search, you need to have vector embeddings stored in documents +and the attribute that stores them needs to be indexed by a +[vector index](../../index-and-search/indexing/working-with-indexes/vector-indexes.md). + +{{< warning >}} +The vector index is an experimental feature that you need to enable for the +ArangoDB server with the `--experimental-vector-index` startup option. +Once enabled for a deployment, it cannot be disabled anymore because it +permanently changes how the data is managed by the RocksDB storage engine +(it adds an additional column family). +{{< /warning >}} + +{{< comment >}}TODO: Add DSS docs or already mention because of ArangoGraph with ML? +You can calculate vector embeddings using ArangoDB's GraphML capabilities or +external tools. +{{< /comment >}} + +## Distance functions + +In order to utilize a vector index, you need to use one of the following +vector distance functions in a query, sort by this distance, and specify the +maximum number of similar documents to retrieve with a `LIMIT` operation. +Example: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_L2(doc.vector, @q) + LIMIT 5 + RETURN doc +``` + +The `@q` bind variable needs to be vector (array of numbers) with the dimension +as specified in the vector index. It defines the point at which to look for +neighbors (`5` in this case). + +The sorting order needs to be **ascending for the L2 metric** (shown above) and +**descending for the cosine metric**: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC + LIMIT 5 + RETURN doc +``` + +### APPROX_NEAR_COSINE() + +`APPROX_NEAR_COSINE(vector1, vector2, options) → dist` + +Retrieve the approximate distance using the cosine metric, accelerated by a +matching vector index. + +- **vector1** (array of numbers): The first vector. Either this parameter or + `vector2` needs to reference a stored attribute holding the vector embedding. + attribute of a stored document that stores a vector, like `doc.vector` +- **vector2** (array of numbers): The second vector. Either this parameter or + `vector1` needs to reference a stored attribute holding the vector embedding. +- **options** (object, _optional_): + - **nProbe** (number, _optional_): How many neighboring centroids to consider + for the search results. The larger the number, the slower the search but the + better the search results. If not specified, the `defaultNProbe` value of + the vector index is used. +- returns **dist** (number): The approximate cosine distance between both vectors. + + + +### APPROX_NEAR_L2() + +`APPROX_NEAR_L2(vector1, vector2, options) → dist` + +Retrieve the approximate distance using the L2 (Euclidean) metric, accelerated +by a matching vector index. + +- **vector1** (array of numbers): The first vector. Either this parameter or + `vector2` needs to reference a stored attribute holding the vector embedding. + attribute of a stored document that stores a vector, like `doc.vector` +- **vector2** (array of numbers): The second vector. Either this parameter or + `vector1` needs to reference a stored attribute holding the vector embedding. +- **options** (object, _optional_): + - **nProbe** (number, _optional_): How many neighboring centroids to consider + for the search results. The larger the number, the slower the search but the + better the search results. If not specified, the `defaultNProbe` value of + the vector index is used. +- returns **dist** (number): The approximate L2 (Euclidean) distance between + both vectors. + + diff --git a/site/content/3.13/index-and-search/indexing/basics.md b/site/content/3.13/index-and-search/indexing/basics.md index d43d950350..afb4b86925 100644 --- a/site/content/3.13/index-and-search/indexing/basics.md +++ b/site/content/3.13/index-and-search/indexing/basics.md @@ -369,6 +369,16 @@ the `GEO_DISTANCE()` function, or if `FILTER` conditions with `GEO_CONTAINS()` or `GEO_INTERSECTS()` are used. It will not be used for other types of queries or conditions. +## Vector Index + +Vector indexes let you index vector embeddings stored in documents. Such +vectors are arrays of numbers that represent the meaning and relationships of +data numerically. You can you quickly find a given number of semantically +similar documents by searching for close neighbors in a high-dimensional +vector space. + +See [Vector Indexes](working-with-indexes/vector-indexes.md) for details. + ## Fulltext Index {{< warning >}} diff --git a/site/content/3.13/index-and-search/indexing/which-index-to-use-when.md b/site/content/3.13/index-and-search/indexing/which-index-to-use-when.md index fc97fc3c92..97f3d8206f 100644 --- a/site/content/3.13/index-and-search/indexing/which-index-to-use-when.md +++ b/site/content/3.13/index-and-search/indexing/which-index-to-use-when.md @@ -106,6 +106,18 @@ different usage scenarios: of the Earth. It supports points, lines, and polygons. See [Geo-Spatial Indexes](working-with-indexes/geo-spatial-indexes.md). +- **Vector index**: You can find semantically similar documents quickly with + vector indexes. It is required to calculate and store vector embeddings first, + and you may need to update the embeddings when adding new documents. + Vector indexes cannot be used for other types of searches, like equality and + range queries or full-text search. + + Vector indexes are utilized via special distance functions, in combination with + a `SORT` operation to sort by the distance, and a `LIMIT` operation to define + how many similar documents to retrieve. + + See [Vector indexes](working-with-indexes/vector-indexes.md) for details. + - **fulltext index**: a fulltext index can be used to index all words contained in a specific attribute of all documents in a collection. Only words with a (specifiable) minimum length are indexed. Word tokenization is done using diff --git a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md new file mode 100644 index 0000000000..aa503fc882 --- /dev/null +++ b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -0,0 +1,156 @@ +--- +title: Vector indexes +menuTitle: Vector Indexes +weight: 40 +description: >- + You can index vector embeddings to allow queries to quickly find semantically + similar documents +--- +Vector indexes let you index vector embeddings stored in documents. Such +vectors are arrays of numbers that represent the meaning and relationships of +data numerically. You can you quickly find a given number of semantically +similar documents by searching for close neighbors in a high-dimensional +vector space. + +The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/) +to support L2 and cosine metrics. The index used is IndexIVFFlat, the quantizer +for L2 is IndexFlatL2, and the cosine uses IndexFlatIP, where vectors are +normalized before insertion and search. + +Sometimes, if there is no relevant data found in the list, the faiss might not +produce the top K requested results. Therefore, only the found results is provided. + +{{< warning >}} +The vector index is an experimental feature that you need to enable for the +ArangoDB server with the `--experimental-vector-index` startup option. +Once enabled for a deployment, it cannot be disabled anymore because it +permanently changes how the data is managed by the RocksDB storage engine +(it adds an additional column family). +{{< /warning >}} + +### How to use vector indexes + +Creating an index triggers training the index on top of real data, which is a limitation that assumes the data already exists for the vector field upon which the index is created. +The number of training points depends on the nLists parameter; a bigger nLists will produce more correct results but will increase the training time necessary to build the index. + + +## Vector index properties + +- **name** (_optional_): A user-defined name for the index for easier + identification. If not specified, a name is automatically generated. +- **type**: The index type. Needs to be `"vector"`. +- **fields** (array of strings): A list with a single attribute path to specify + where the vector embedding is stored in each document. The vector data needs + to be populated before creating the index. + + If you want to index another vector embedding attribute, you need to create a + separate vector index. +- **params**: The parameters as used by the Faiss library. + - **metric** (string): Whether to use `cosine` or `l2` (Euclidean) distance calculation. + - **dimension** (number): The vector dimension. The attribute to index needs to + have this many elements in the array that stores the vector embedding. + - **nLists** (number): The number of centroids in the index. What value to choose + depends on the data distribution and chosen metric. According to + [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be + around `15 * N` where `N` is the number of documents in the collection, + respectively the number of documents in the shard for cluster deployments. + - **defaultNProbe** (number, _optional_): How many neighboring centroids to + consider for the search results by default. The larger the number, the slower + the search but the better the search results. The default is `1`. + - **trainingIterations** (number, _optional_): The number of iterations in the + training process. The default is `25`. Smaller values lead to a faster index + creation but may yield worse search results. + - **factory** (string, _optional_): You can specify a factory string to pass + through to the underlying Faiss library, allowing you to combine different + options, for example: + - `"IVF100_HNSW10,Flat"` + - `"IVF100,SQ4"` + - `"IVF10_HNSW5,Flat"` + - `"IVF100_HNSW5,PQ256x16"` + The base index must be an IVF to work with ArangoDB. For more information on + how to create these custom indexes, see the + [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + +## Interfaces + +### Create a vector index + +{{< tabs "interfaces" >}} + +{{< tab "Web interface" >}} +1. In the **Collections** section, click the name or row of the desired collection. +2. Go to the **Indexes** tab. +3. Click **Add index**. +4. Select **Vector** as the **Type**. +5. Enter the name of the attribute that holds the vector embeddings into **Fields**. +6. Set the parameters for the vector index, see [Vector index parameters](#vector-index-parameters). +7. Optionally give the index a user-defined name. +8. Click **Create**. +{{< /tab >}} + +{{< tab "arangosh" >}} +```js +db.coll.ensureIndex({ + name: "vector_l2", + type: "vector", + fields: ["embedding"], + params: { + metric: "l2", + dimension: 544, + nLists: 100, + defaultNProbe: 1, + trainingIterations: 25 + } +}); +``` +{{< /tab >}} + +{{< tab "cURL" >}} +```sh +curl -d '{"name":"vector_l2","type":"vector","fields":["embedding"],"params":{"metric":"l2","dimension":544,"nLists":100,"defaultNProbe":1,"trainingIterations":25}}' http://localhost:8529/_db/mydb/_api/index?collection=coll +``` +{{< /tab >}} + +{{< tab "JavaScript" >}} +```js +const info = await coll.ensureIndex({ + name: "vector_l2", + type: "vector", + fields: ["embedding"], + params: { + metric: "l2", + dimension: 544, + nLists: 100, + defaultNProbe: 1, + trainingIterations: 25 + } +}); +``` +{{< /tab >}} + +{{< tab "Go" >}} +The Go driver does not support vector indexes yet. +{{< /tab >}} + +{{< tab "Java" >}} +The Java driver does not support vector indexes yet. +{{< /tab >}} + +{{< tab "Python" >}} +```py +info = coll.add_index({ + "name": "vector_l2", + "type": "vector", + "fields": ["embedding"], + "params": { + "metric": "l2", + "dimension": 544 + "nLists": 100, + "defaultNProbe": 1, + "trainingIterations": 25 + } +}) +``` +{{< /tab >}} + +{{< /tabs >}} From 604f1a0cc0dc3d3ace13816428b016f0231afd13 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 5 Jun 2025 17:39:11 +0200 Subject: [PATCH 2/9] HTTP API docs and refinements --- .../features/community-edition.md | 2 +- site/content/3.12/aql/functions/vector.md | 191 ++++++++++++++++++ .../3.12/develop/http-api/indexes/fulltext.md | 9 +- .../develop/http-api/indexes/geo-spatial.md | 10 +- .../3.12/develop/http-api/indexes/inverted.md | 8 +- .../http-api/indexes/multi-dimensional.md | 18 +- .../develop/http-api/indexes/persistent.md | 17 +- .../3.12/develop/http-api/indexes/ttl.md | 13 +- .../3.12/develop/http-api/indexes/vector.md | 135 +++++++++++++ .../3.12/index-and-search/indexing/basics.md | 10 + .../indexing/which-index-to-use-when.md | 12 ++ .../working-with-indexes/vector-indexes.md | 183 +++++++++++++++++ site/content/3.13/aql/functions/vector.md | 156 +++++++++++--- .../3.13/develop/http-api/indexes/fulltext.md | 9 +- .../develop/http-api/indexes/geo-spatial.md | 10 +- .../3.13/develop/http-api/indexes/inverted.md | 8 +- .../http-api/indexes/multi-dimensional.md | 18 +- .../develop/http-api/indexes/persistent.md | 17 +- .../3.13/develop/http-api/indexes/ttl.md | 13 +- .../3.13/develop/http-api/indexes/vector.md | 135 +++++++++++++ .../working-with-indexes/vector-indexes.md | 53 +++-- 21 files changed, 899 insertions(+), 128 deletions(-) create mode 100644 site/content/3.12/aql/functions/vector.md create mode 100644 site/content/3.12/develop/http-api/indexes/vector.md create mode 100644 site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md create mode 100644 site/content/3.13/develop/http-api/indexes/vector.md diff --git a/site/content/3.12/about-arangodb/features/community-edition.md b/site/content/3.12/about-arangodb/features/community-edition.md index 9ee6ac647d..1d8f118af9 100644 --- a/site/content/3.12/about-arangodb/features/community-edition.md +++ b/site/content/3.12/about-arangodb/features/community-edition.md @@ -149,7 +149,7 @@ see [arangodb.com/community-server/](https://www.arangodb.com/community-server/) {{% /comment %}} {{% comment %}} Experimental feature -- [**Vector search**](#TODO): +- [**Vector search**](../../index-and-search/indexing/working-with-indexes/vector-indexes.md): Find items with similar properties by comparing vector embeddings generated by machine learning models. {{% /comment %}} diff --git a/site/content/3.12/aql/functions/vector.md b/site/content/3.12/aql/functions/vector.md new file mode 100644 index 0000000000..680ece5eea --- /dev/null +++ b/site/content/3.12/aql/functions/vector.md @@ -0,0 +1,191 @@ +--- +title: Vector search functions in AQL +menuTitle: Vector +weight: 60 +description: >- + The functions for vector search let you quickly find semantically similar + documents utilizing indexed vector embeddings +--- +To use vector search, you need to have vector embeddings stored in documents +and the attribute that stores them needs to be indexed by a +[vector index](../../index-and-search/indexing/working-with-indexes/vector-indexes.md). + +You can calculate vector embeddings using [ArangoDB's GraphML](../../data-science/arangographml/_index.md) +capabilities (available in ArangoGraph) or using external tools. + +{{< warning >}} +The vector index is an experimental feature that you need to enable for the +ArangoDB server with the `--experimental-vector-index` startup option. +Once enabled for a deployment, it cannot be disabled anymore because it +permanently changes how the data is managed by the RocksDB storage engine +(it adds an additional column family). + +To restore a dump that contains vector indexes, the `--experimental-vector-index` +startup option needs to be enabled on the deployment you want to restore to. +{{< /warning >}} + +## Vector similarity functions + +In order to utilize a vector index, you need to do the following in an AQL query: + +- Use one of the following vector similarity functions in a query. +- `SORT` by the similarity so that the most similar documents come first. +- Specify the maximum number of documents to retrieve with a `LIMIT` operation. + +As a result, you get up to the specified number of documents whose vector embeddings +are the most similar to the reference vector embedding you provided in the query, +as approximated by the vector index. + +Example: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_L2(doc.vector, @q) + LIMIT 5 + RETURN doc +``` + +For this query, a vector index over the `vector` attribute and with the `l2` +metric is required. The `@q` bind variable needs to be a vector (array of numbers) +with the dimension as specified in the vector index. It defines the point at +which to look for neighbors (up to `5` in this case). How many neighbors can be +found depends on the data as well as the search effort (see the `nProbe` option). + +{{< info >}} +- If there is more than one suitable vector index over the same attribute, it is + undefined which one is selected. +- You cannot have any `FILTER` operation between `FOR` and `LIMIT` for + pre-filtering. +{{< /info >}} + +### APPROX_NEAR_COSINE() + +`APPROX_NEAR_COSINE(vector1, vector2, options) → similarity` + +Retrieve the approximate angular similarity using the cosine metric, accelerated +by a matching vector index. + +The closer the cosine similarity value is to 1, the more similar the two vectors +are. The closer it is to 0, the more different they are. The value can also +be up to -1, indicating that the vectors are not similar and point in opposite +directions. You need to sort in descending order so that the most similar +documents come first, which is what a vector index using the `cosine` metric +can provide. + +- **vector1** (array of numbers): The first vector. Either this parameter or + `vector2` needs to reference a stored attribute holding the vector embedding. + attribute of a stored document that stores a vector, like `doc.vector` +- **vector2** (array of numbers): The second vector. Either this parameter or + `vector1` needs to reference a stored attribute holding the vector embedding. +- **options** (object, _optional_): + - **nProbe** (number, _optional_): How many neighboring centroids to consider + for the search results. The larger the number, the slower the search but the + better the search results. If not specified, the `defaultNProbe` value of + the vector index is used. +- returns **similarity** (number): The approximate angular similarity between + both vectors. + +**Examples** + +Return the documents of up to `10` neighbors that are the closest to the vector +`@q` according to the cosine metric: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC + LIMIT 10 + RETURN doc +``` + +Return the similarity value and the documents of up to `5` close neighbors, +considering `20` neighboring centroids: + +```aql +FOR doc IN coll + LET similarity = APPROX_NEAR_COSINE(doc.vector, @q, { nProbe: 20 }) + SORT similarity DESC + LIMIT 5 + RETURN MERGE( { similarity }, doc) +``` + +Return the similarity and the document keys of up to `3` neighbors for multiple +vectors using a subquery, here taking from ten random documents of the same +collection: + +```aql +FOR docOuter IN coll + LIMIT 10 + LET neighbors = ( + FOR docInner IN coll + LET similarity = APPROX_NEAR_COSINE(docInner.vector, docOuter.vector) + SORT similarity DESC + LIMIT 3 + RETURN { key: docInner._key, similarity } + ) + RETURN { key: docOuter._key, neighbors } +``` + +### APPROX_NEAR_L2() + +`APPROX_NEAR_L2(vector1, vector2, options) → similarity` + +Retrieve the approximate distance using the L2 (Euclidean) metric, accelerated +by a matching vector index. + +The closer the distance is to 0, the more similar the two vectors are. The higher +the value, the more different the they are. You need to sort in ascending order +so that the most similar documents come first, which is what a vector index using +the `l2` metric can provide. + +- **vector1** (array of numbers): The first vector. Either this parameter or + `vector2` needs to reference a stored attribute holding the vector embedding. + attribute of a stored document that stores a vector, like `doc.vector` +- **vector2** (array of numbers): The second vector. Either this parameter or + `vector1` needs to reference a stored attribute holding the vector embedding. +- **options** (object, _optional_): + - **nProbe** (number, _optional_): How many neighboring centroids to consider + for the search results. The larger the number, the slower the search but the + better the search results. If not specified, the `defaultNProbe` value of + the vector index is used. +- returns **similarity** (number): The approximate L2 (Euclidean) distance between + both vectors. + +**Examples** + +Return the documents of up to `10` neighbors that are the closest to the vector +`@q` according to the L2 (Euclidean) metric: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_L2(doc.vector, @q) + LIMIT 10 + RETURN doc +``` + +Return the similarity value and the documents of up to `5` close neighbors, +considering `20` neighboring centroids: + +```aql +FOR doc IN coll + LET similarity = APPROX_NEAR_L2(doc.vector, @q, { nProbe: 20 }) + SORT similarity + LIMIT 5 + RETURN MERGE( { similarity }, doc) +``` + +Return the similarity and the document keys of up to `3` close neighbors for +multiple vectors using a subquery, here taking from ten random documents of the +same collection: + +```aql +FOR docOuter IN coll + LIMIT 10 + LET neighbors = ( + FOR docInner IN coll + LET similarity = APPROX_NEAR_L2(docInner.vector, docOuter.vector) + SORT similarity + LIMIT 3 + RETURN { key: docInner._key, similarity } + ) + RETURN { key: docOuter._key, neighbors } +``` diff --git a/site/content/3.12/develop/http-api/indexes/fulltext.md b/site/content/3.12/develop/http-api/indexes/fulltext.md index a782084656..1d958b463f 100644 --- a/site/content/3.12/develop/http-api/indexes/fulltext.md +++ b/site/content/3.12/develop/http-api/indexes/fulltext.md @@ -49,6 +49,7 @@ paths: description: | Must be equal to `"fulltext"`. type: string + example: fulltext name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -78,15 +79,13 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is - returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.12/develop/http-api/indexes/geo-spatial.md b/site/content/3.12/develop/http-api/indexes/geo-spatial.md index a910a3a39f..16b4b3e020 100644 --- a/site/content/3.12/develop/http-api/indexes/geo-spatial.md +++ b/site/content/3.12/develop/http-api/indexes/geo-spatial.md @@ -13,7 +13,7 @@ paths: operationId: createIndexGeo description: | Creates a geo-spatial index in the collection `collection`, if - it does not already exist. Expects an object containing the index details. + it does not already exist. Geo indexes are always sparse, meaning that documents that do not contain the index attributes or have non-numeric values in the index attributes @@ -47,6 +47,7 @@ paths: description: | Must be equal to `"geo"`. type: string + example: geo name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -105,14 +106,13 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '404': description: | - If the `collection` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.12/develop/http-api/indexes/inverted.md b/site/content/3.12/develop/http-api/indexes/inverted.md index 4a1fc22261..fe016f8436 100644 --- a/site/content/3.12/develop/http-api/indexes/inverted.md +++ b/site/content/3.12/develop/http-api/indexes/inverted.md @@ -44,6 +44,7 @@ paths: description: | Must be equal to `"inverted"`. type: string + example: inverted name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -625,14 +626,13 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is returned. + The index exists already. '201': description: | - If the index does not already exist and can be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.12/develop/http-api/indexes/multi-dimensional.md b/site/content/3.12/develop/http-api/indexes/multi-dimensional.md index d6693a6900..f80f2722da 100644 --- a/site/content/3.12/develop/http-api/indexes/multi-dimensional.md +++ b/site/content/3.12/develop/http-api/indexes/multi-dimensional.md @@ -13,8 +13,7 @@ paths: operationId: createIndexMdi description: | Creates a multi-dimensional index for the collection `collection-name`, if - it does not already exist. The call expects an object containing the index - details. + it does not already exist. parameters: - name: database-name in: path @@ -45,6 +44,7 @@ paths: description: | Must be equal to `"mdi"` or `"mdi-prefixed"`. type: string + example: mdi name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -136,18 +136,16 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is - returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. - '404': - description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The index is created as there is no such existing index. '400': description: | - If the index definition is invalid, then a *HTTP 400* is returned. + The index definition is invalid. + '404': + description: | + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.12/develop/http-api/indexes/persistent.md b/site/content/3.12/develop/http-api/indexes/persistent.md index 765589debc..15d9efafce 100644 --- a/site/content/3.12/develop/http-api/indexes/persistent.md +++ b/site/content/3.12/develop/http-api/indexes/persistent.md @@ -19,8 +19,7 @@ paths: operationId: createIndexPersistent description: | Creates a persistent index for the collection `collection-name`, if - it does not already exist. The call expects an object containing the index - details. + it does not already exist. In a sparse index all documents will be excluded from the index that do not contain at least one of the specified index attributes (i.e. `fields`) or that @@ -64,6 +63,7 @@ paths: description: | Must be equal to `"persistent"`. type: string + example: persistent name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -165,20 +165,17 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is - returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '400': description: | - If the collection already contains documents and you try to create a unique - persistent index in such a way that there are documents violating the - uniqueness, then a *HTTP 400* is returned. + You try to create a unique persistent index but there are already + documents in the collection that violate the uniqueness requirement. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.12/develop/http-api/indexes/ttl.md b/site/content/3.12/develop/http-api/indexes/ttl.md index 97c0642271..54eeae5c40 100644 --- a/site/content/3.12/develop/http-api/indexes/ttl.md +++ b/site/content/3.12/develop/http-api/indexes/ttl.md @@ -13,8 +13,7 @@ paths: operationId: createIndexTtl description: | Creates a time-to-live (TTL) index for the collection `collection-name` if it - does not already exist. The call expects an object containing the index - details. + does not already exist. parameters: - name: database-name in: path @@ -74,18 +73,16 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '400': description: | - If the collection already contains another TTL index, then an *HTTP 400* is - returned, as there can be at most one TTL index per collection. + There is already a TTL index for the collection but there can only be one. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.12/develop/http-api/indexes/vector.md b/site/content/3.12/develop/http-api/indexes/vector.md new file mode 100644 index 0000000000..1472bdd4c2 --- /dev/null +++ b/site/content/3.12/develop/http-api/indexes/vector.md @@ -0,0 +1,135 @@ +--- +title: HTTP interface for vector indexes +menuTitle: Vector +weight: 35 +description: '' +--- +## Create a vector index + +```openapi +paths: + /_db/{database-name}/_api/index#vector: + post: + operationId: createIndexVector + description: | + Creates a vector index for the collection `collection-name`, if + it does not already exist. + parameters: + - name: database-name + in: path + required: true + example: _system + description: | + The name of the database. + schema: + type: string + - name: collection + in: query + required: true + description: | + The collection name. + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + required: + - type + - fields + - params + properties: + type: + description: | + The index type. Needs to be `"vector"`. + type: string + example: vector + name: + description: | + A user-defined name for the index for easier + identification. If not specified, a name is automatically generated. + type: string + fields: + description: | + A list with a single attribute path to specify + where the vector embedding is stored in each document. The vector data needs + to be populated before creating the index. + + If you want to index another vector embedding attribute, you need to create a + separate vector index. + type: array + items: + type: object + params: + description: | + The parameters as used by the Faiss library. + type: object + required: + - metric + - dimension + - nLists + properties: + metric: + description: | + Whether to use `cosine` or `l2` (Euclidean) distance calculation. + type: string + enum: ["cosine", "l2"] + dimension: + description: | + The vector dimension. The attribute to index needs to + have this many elements in the array that stores the vector embedding. + type: integer + nLists: + description: | + The number of centroids in the index. What value to choose + depends on the data distribution and chosen metric. According to + [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be + around `N / 15` where `N` is the number of documents in the collection, + respectively the number of documents in the shard for cluster deployments. + A bigger value produces more correct results but increases the training time + and thus how long it takes to build the index. It cannot be bigger than the + number of documents. + type: integer + defaultNProbe: + description: | + How many neighboring centroids to + consider for the search results by default. The larger the number, the slower + the search but the better the search results. The default is `1`. You should + generally use a higher value here or per query via the `nProbe` option of + the vector similarity functions. + type: integer + default: 1 + trainingIterations: + description: | + The number of iterations in the + training process. The default is `25`. Smaller values lead to a faster index + creation but may yield worse search results. + type: integer + default: 25 + factory: + description: | + You can specify a factory string to pass + through to the underlying Faiss library, allowing you to combine different + options, for example: + - `"IVF100_HNSW10,Flat"` + - `"IVF100,SQ4"` + - `"IVF10_HNSW5,Flat"` + - `"IVF100_HNSW5,PQ256x16"` + The base index must be an IVF to work with ArangoDB. For more information on + how to create these custom indexes, see the + [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + type: string + responses: + '200': + description: | + The index exists already. + '201': + description: | + The index is created as there is no such existing index. + '404': + description: | + The collection is unknown. + tags: + - Indexes +``` diff --git a/site/content/3.12/index-and-search/indexing/basics.md b/site/content/3.12/index-and-search/indexing/basics.md index d43d950350..afb4b86925 100644 --- a/site/content/3.12/index-and-search/indexing/basics.md +++ b/site/content/3.12/index-and-search/indexing/basics.md @@ -369,6 +369,16 @@ the `GEO_DISTANCE()` function, or if `FILTER` conditions with `GEO_CONTAINS()` or `GEO_INTERSECTS()` are used. It will not be used for other types of queries or conditions. +## Vector Index + +Vector indexes let you index vector embeddings stored in documents. Such +vectors are arrays of numbers that represent the meaning and relationships of +data numerically. You can you quickly find a given number of semantically +similar documents by searching for close neighbors in a high-dimensional +vector space. + +See [Vector Indexes](working-with-indexes/vector-indexes.md) for details. + ## Fulltext Index {{< warning >}} diff --git a/site/content/3.12/index-and-search/indexing/which-index-to-use-when.md b/site/content/3.12/index-and-search/indexing/which-index-to-use-when.md index fc97fc3c92..97f3d8206f 100644 --- a/site/content/3.12/index-and-search/indexing/which-index-to-use-when.md +++ b/site/content/3.12/index-and-search/indexing/which-index-to-use-when.md @@ -106,6 +106,18 @@ different usage scenarios: of the Earth. It supports points, lines, and polygons. See [Geo-Spatial Indexes](working-with-indexes/geo-spatial-indexes.md). +- **Vector index**: You can find semantically similar documents quickly with + vector indexes. It is required to calculate and store vector embeddings first, + and you may need to update the embeddings when adding new documents. + Vector indexes cannot be used for other types of searches, like equality and + range queries or full-text search. + + Vector indexes are utilized via special distance functions, in combination with + a `SORT` operation to sort by the distance, and a `LIMIT` operation to define + how many similar documents to retrieve. + + See [Vector indexes](working-with-indexes/vector-indexes.md) for details. + - **fulltext index**: a fulltext index can be used to index all words contained in a specific attribute of all documents in a collection. Only words with a (specifiable) minimum length are indexed. Word tokenization is done using diff --git a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md new file mode 100644 index 0000000000..5aaafd55a4 --- /dev/null +++ b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -0,0 +1,183 @@ +--- +title: Vector indexes +menuTitle: Vector Indexes +weight: 40 +description: >- + You can index vector embeddings to allow queries to quickly find semantically + similar documents +--- +Vector indexes let you index vector embeddings stored in documents. Such +vectors are arrays of numbers that represent the meaning and relationships of +data numerically and can be generated with machine learning models. +You can then quickly find a given number of semantically similar documents by +searching for close neighbors in a high-dimensional vector space. + +The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/) +to support L2 and cosine metrics. The index used is IndexIVFFlat, the quantizer +for L2 is IndexFlatL2, and the cosine uses IndexFlatIP, where vectors are +normalized before insertion and search. + +## How to use vector indexes + +{{< warning >}} +The vector index is an experimental feature that you need to enable for the +ArangoDB server with the `--experimental-vector-index` startup option. +Once enabled for a deployment, it cannot be disabled anymore because it +permanently changes how the data is managed by the RocksDB storage engine +(it adds an additional column family). + +To restore a dump that contains vector indexes, the `--experimental-vector-index` +startup option needs to be enabled on the deployment you want to restore to. +{{< /warning >}} + +1. Enable the experimental vector index feature. +2. Calculate vector embeddings using [ArangoDB's GraphML](../../../data-science/arangographml/_index.md) + capabilities (available in ArangoGraph) or using external tools. + Store each vector as an attribute in the respective document. +3. Create a vector index over this attribute. You need to choose which + similarity metric you want to use later for querying. See + [Vector index properties](#vector-index-properties) for all available + configuration options. +4. Run AQL queries that use [Vector similarity functions](../../../aql/functions/vector.md) + to retrieve a given number of similar documents relative to a vector embedding + you provide. + +Creating a vector index triggers training the index on top of real data and it +cannot be done incrementally without affecting the quality of the computation. +A vector index expects the data to already exist in the specified attribute. +This means **you cannot create a vector index for a collection upfront**, unlike +with all other index types. The documents need to already have vector embeddings +stored in an attribute that you then create the index over and train on. + +While it is possible to add more documents with vector embeddings over time, +they can only be assigned to existing clusters in the high-dimensional vector +space as determined by the original vector index training. This can be suboptimal +as the new data points might warrant a different clustering with different +centroids and the quality of vector search thus degrades. + +## Vector index properties + +- **name** (_optional_): A user-defined name for the index for easier + identification. If not specified, a name is automatically generated. +- **type**: The index type. Needs to be `"vector"`. +- **fields** (array of strings): A list with a single attribute path to specify + where the vector embedding is stored in each document. The vector data needs + to be populated before creating the index. + + If you want to index another vector embedding attribute, you need to create a + separate vector index. +- **params**: The parameters as used by the Faiss library. + - **metric** (string): Whether to use `cosine` or `l2` (Euclidean) distance calculation. + - **dimension** (number): The vector dimension. The attribute to index needs to + have this many elements in the array that stores the vector embedding. + - **nLists** (number): The number of centroids in the index. What value to choose + depends on the data distribution and chosen metric. According to + [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be + around `N / 15` where `N` is the number of documents in the collection, + respectively the number of documents in the shard for cluster deployments. + A bigger value produces more correct results but increases the training time + and thus how long it takes to build the index. It cannot be bigger than the + number of documents. + - **defaultNProbe** (number, _optional_): How many neighboring centroids to + consider for the search results by default. The larger the number, the slower + the search but the better the search results. The default is `1`. You should + generally use a higher value here or per query via the `nProbe` option of + the vector similarity functions. + - **trainingIterations** (number, _optional_): The number of iterations in the + training process. The default is `25`. Smaller values lead to a faster index + creation but may yield worse search results. + - **factory** (string, _optional_): You can specify a factory string to pass + through to the underlying Faiss library, allowing you to combine different + options, for example: + - `"IVF100_HNSW10,Flat"` + - `"IVF100,SQ4"` + - `"IVF10_HNSW5,Flat"` + - `"IVF100_HNSW5,PQ256x16"` + The base index must be an IVF to work with ArangoDB. For more information on + how to create these custom indexes, see the + [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + +## Interfaces + +### Create a vector index + +{{< tabs "interfaces" >}} + +{{< tab "Web interface" >}} +1. In the **Collections** section, click the name or row of the desired collection. +2. Go to the **Indexes** tab. +3. Click **Add index**. +4. Select **Vector** as the **Type**. +5. Enter the name of the attribute that holds the vector embeddings into **Fields**. +6. Set the parameters for the vector index. See [Vector index properties](#vector-index-properties) + under `param`. +7. Optionally give the index a user-defined name. +8. Click **Create**. +{{< /tab >}} + +{{< tab "arangosh" >}} +```js +db.coll.ensureIndex({ + name: "vector_l2", + type: "vector", + fields: ["embedding"], + params: { + metric: "l2", + dimension: 544, + nLists: 100, + defaultNProbe: 1, + trainingIterations: 25 + } +}); +``` +{{< /tab >}} + +{{< tab "cURL" >}} +```sh +curl -d '{"name":"vector_l2","type":"vector","fields":["embedding"],"params":{"metric":"l2","dimension":544,"nLists":100,"defaultNProbe":1,"trainingIterations":25}}' http://localhost:8529/_db/mydb/_api/index?collection=coll +``` +{{< /tab >}} + +{{< tab "JavaScript" >}} +```js +const info = await coll.ensureIndex({ + name: "vector_l2", + type: "vector", + fields: ["embedding"], + params: { + metric: "l2", + dimension: 544, + nLists: 100, + defaultNProbe: 1, + trainingIterations: 25 + } +}); +``` +{{< /tab >}} + +{{< tab "Go" >}} +The Go driver does not support vector indexes yet. +{{< /tab >}} + +{{< tab "Java" >}} +The Java driver does not support vector indexes yet. +{{< /tab >}} + +{{< tab "Python" >}} +```py +info = coll.add_index({ + "name": "vector_l2", + "type": "vector", + "fields": ["embedding"], + "params": { + "metric": "l2", + "dimension": 544 + "nLists": 100, + "defaultNProbe": 1, + "trainingIterations": 25 + } +}) +``` +{{< /tab >}} + +{{< /tabs >}} diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md index a34f54d0da..680ece5eea 100644 --- a/site/content/3.13/aql/functions/vector.md +++ b/site/content/3.13/aql/functions/vector.md @@ -3,31 +3,39 @@ title: Vector search functions in AQL menuTitle: Vector weight: 60 description: >- - The functions for vector search let you utilize indexed vector embeddings to - quickly find semantically similar documents + The functions for vector search let you quickly find semantically similar + documents utilizing indexed vector embeddings --- To use vector search, you need to have vector embeddings stored in documents and the attribute that stores them needs to be indexed by a [vector index](../../index-and-search/indexing/working-with-indexes/vector-indexes.md). +You can calculate vector embeddings using [ArangoDB's GraphML](../../data-science/arangographml/_index.md) +capabilities (available in ArangoGraph) or using external tools. + {{< warning >}} The vector index is an experimental feature that you need to enable for the ArangoDB server with the `--experimental-vector-index` startup option. Once enabled for a deployment, it cannot be disabled anymore because it permanently changes how the data is managed by the RocksDB storage engine (it adds an additional column family). + +To restore a dump that contains vector indexes, the `--experimental-vector-index` +startup option needs to be enabled on the deployment you want to restore to. {{< /warning >}} -{{< comment >}}TODO: Add DSS docs or already mention because of ArangoGraph with ML? -You can calculate vector embeddings using ArangoDB's GraphML capabilities or -external tools. -{{< /comment >}} +## Vector similarity functions + +In order to utilize a vector index, you need to do the following in an AQL query: + +- Use one of the following vector similarity functions in a query. +- `SORT` by the similarity so that the most similar documents come first. +- Specify the maximum number of documents to retrieve with a `LIMIT` operation. -## Distance functions +As a result, you get up to the specified number of documents whose vector embeddings +are the most similar to the reference vector embedding you provided in the query, +as approximated by the vector index. -In order to utilize a vector index, you need to use one of the following -vector distance functions in a query, sort by this distance, and specify the -maximum number of similar documents to retrieve with a `LIMIT` operation. Example: ```aql @@ -37,26 +45,32 @@ FOR doc IN coll RETURN doc ``` -The `@q` bind variable needs to be vector (array of numbers) with the dimension -as specified in the vector index. It defines the point at which to look for -neighbors (`5` in this case). +For this query, a vector index over the `vector` attribute and with the `l2` +metric is required. The `@q` bind variable needs to be a vector (array of numbers) +with the dimension as specified in the vector index. It defines the point at +which to look for neighbors (up to `5` in this case). How many neighbors can be +found depends on the data as well as the search effort (see the `nProbe` option). -The sorting order needs to be **ascending for the L2 metric** (shown above) and -**descending for the cosine metric**: - -```aql -FOR doc IN coll - SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC - LIMIT 5 - RETURN doc -``` +{{< info >}} +- If there is more than one suitable vector index over the same attribute, it is + undefined which one is selected. +- You cannot have any `FILTER` operation between `FOR` and `LIMIT` for + pre-filtering. +{{< /info >}} ### APPROX_NEAR_COSINE() -`APPROX_NEAR_COSINE(vector1, vector2, options) → dist` +`APPROX_NEAR_COSINE(vector1, vector2, options) → similarity` + +Retrieve the approximate angular similarity using the cosine metric, accelerated +by a matching vector index. -Retrieve the approximate distance using the cosine metric, accelerated by a -matching vector index. +The closer the cosine similarity value is to 1, the more similar the two vectors +are. The closer it is to 0, the more different they are. The value can also +be up to -1, indicating that the vectors are not similar and point in opposite +directions. You need to sort in descending order so that the most similar +documents come first, which is what a vector index using the `cosine` metric +can provide. - **vector1** (array of numbers): The first vector. Either this parameter or `vector2` needs to reference a stored attribute holding the vector embedding. @@ -68,17 +82,61 @@ matching vector index. for the search results. The larger the number, the slower the search but the better the search results. If not specified, the `defaultNProbe` value of the vector index is used. -- returns **dist** (number): The approximate cosine distance between both vectors. +- returns **similarity** (number): The approximate angular similarity between + both vectors. + +**Examples** + +Return the documents of up to `10` neighbors that are the closest to the vector +`@q` according to the cosine metric: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC + LIMIT 10 + RETURN doc +``` - +Return the similarity value and the documents of up to `5` close neighbors, +considering `20` neighboring centroids: + +```aql +FOR doc IN coll + LET similarity = APPROX_NEAR_COSINE(doc.vector, @q, { nProbe: 20 }) + SORT similarity DESC + LIMIT 5 + RETURN MERGE( { similarity }, doc) +``` + +Return the similarity and the document keys of up to `3` neighbors for multiple +vectors using a subquery, here taking from ten random documents of the same +collection: + +```aql +FOR docOuter IN coll + LIMIT 10 + LET neighbors = ( + FOR docInner IN coll + LET similarity = APPROX_NEAR_COSINE(docInner.vector, docOuter.vector) + SORT similarity DESC + LIMIT 3 + RETURN { key: docInner._key, similarity } + ) + RETURN { key: docOuter._key, neighbors } +``` ### APPROX_NEAR_L2() -`APPROX_NEAR_L2(vector1, vector2, options) → dist` +`APPROX_NEAR_L2(vector1, vector2, options) → similarity` Retrieve the approximate distance using the L2 (Euclidean) metric, accelerated by a matching vector index. +The closer the distance is to 0, the more similar the two vectors are. The higher +the value, the more different the they are. You need to sort in ascending order +so that the most similar documents come first, which is what a vector index using +the `l2` metric can provide. + - **vector1** (array of numbers): The first vector. Either this parameter or `vector2` needs to reference a stored attribute holding the vector embedding. attribute of a stored document that stores a vector, like `doc.vector` @@ -89,7 +147,45 @@ by a matching vector index. for the search results. The larger the number, the slower the search but the better the search results. If not specified, the `defaultNProbe` value of the vector index is used. -- returns **dist** (number): The approximate L2 (Euclidean) distance between +- returns **similarity** (number): The approximate L2 (Euclidean) distance between both vectors. - +**Examples** + +Return the documents of up to `10` neighbors that are the closest to the vector +`@q` according to the L2 (Euclidean) metric: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_L2(doc.vector, @q) + LIMIT 10 + RETURN doc +``` + +Return the similarity value and the documents of up to `5` close neighbors, +considering `20` neighboring centroids: + +```aql +FOR doc IN coll + LET similarity = APPROX_NEAR_L2(doc.vector, @q, { nProbe: 20 }) + SORT similarity + LIMIT 5 + RETURN MERGE( { similarity }, doc) +``` + +Return the similarity and the document keys of up to `3` close neighbors for +multiple vectors using a subquery, here taking from ten random documents of the +same collection: + +```aql +FOR docOuter IN coll + LIMIT 10 + LET neighbors = ( + FOR docInner IN coll + LET similarity = APPROX_NEAR_L2(docInner.vector, docOuter.vector) + SORT similarity + LIMIT 3 + RETURN { key: docInner._key, similarity } + ) + RETURN { key: docOuter._key, neighbors } +``` diff --git a/site/content/3.13/develop/http-api/indexes/fulltext.md b/site/content/3.13/develop/http-api/indexes/fulltext.md index a782084656..1d958b463f 100644 --- a/site/content/3.13/develop/http-api/indexes/fulltext.md +++ b/site/content/3.13/develop/http-api/indexes/fulltext.md @@ -49,6 +49,7 @@ paths: description: | Must be equal to `"fulltext"`. type: string + example: fulltext name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -78,15 +79,13 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is - returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.13/develop/http-api/indexes/geo-spatial.md b/site/content/3.13/develop/http-api/indexes/geo-spatial.md index a910a3a39f..16b4b3e020 100644 --- a/site/content/3.13/develop/http-api/indexes/geo-spatial.md +++ b/site/content/3.13/develop/http-api/indexes/geo-spatial.md @@ -13,7 +13,7 @@ paths: operationId: createIndexGeo description: | Creates a geo-spatial index in the collection `collection`, if - it does not already exist. Expects an object containing the index details. + it does not already exist. Geo indexes are always sparse, meaning that documents that do not contain the index attributes or have non-numeric values in the index attributes @@ -47,6 +47,7 @@ paths: description: | Must be equal to `"geo"`. type: string + example: geo name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -105,14 +106,13 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '404': description: | - If the `collection` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.13/develop/http-api/indexes/inverted.md b/site/content/3.13/develop/http-api/indexes/inverted.md index 4a1fc22261..fe016f8436 100644 --- a/site/content/3.13/develop/http-api/indexes/inverted.md +++ b/site/content/3.13/develop/http-api/indexes/inverted.md @@ -44,6 +44,7 @@ paths: description: | Must be equal to `"inverted"`. type: string + example: inverted name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -625,14 +626,13 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is returned. + The index exists already. '201': description: | - If the index does not already exist and can be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.13/develop/http-api/indexes/multi-dimensional.md b/site/content/3.13/develop/http-api/indexes/multi-dimensional.md index d6693a6900..f80f2722da 100644 --- a/site/content/3.13/develop/http-api/indexes/multi-dimensional.md +++ b/site/content/3.13/develop/http-api/indexes/multi-dimensional.md @@ -13,8 +13,7 @@ paths: operationId: createIndexMdi description: | Creates a multi-dimensional index for the collection `collection-name`, if - it does not already exist. The call expects an object containing the index - details. + it does not already exist. parameters: - name: database-name in: path @@ -45,6 +44,7 @@ paths: description: | Must be equal to `"mdi"` or `"mdi-prefixed"`. type: string + example: mdi name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -136,18 +136,16 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is - returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. - '404': - description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The index is created as there is no such existing index. '400': description: | - If the index definition is invalid, then a *HTTP 400* is returned. + The index definition is invalid. + '404': + description: | + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.13/develop/http-api/indexes/persistent.md b/site/content/3.13/develop/http-api/indexes/persistent.md index 765589debc..15d9efafce 100644 --- a/site/content/3.13/develop/http-api/indexes/persistent.md +++ b/site/content/3.13/develop/http-api/indexes/persistent.md @@ -19,8 +19,7 @@ paths: operationId: createIndexPersistent description: | Creates a persistent index for the collection `collection-name`, if - it does not already exist. The call expects an object containing the index - details. + it does not already exist. In a sparse index all documents will be excluded from the index that do not contain at least one of the specified index attributes (i.e. `fields`) or that @@ -64,6 +63,7 @@ paths: description: | Must be equal to `"persistent"`. type: string + example: persistent name: description: | An easy-to-remember name for the index to look it up or refer to it in index hints. @@ -165,20 +165,17 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is - returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '400': description: | - If the collection already contains documents and you try to create a unique - persistent index in such a way that there are documents violating the - uniqueness, then a *HTTP 400* is returned. + You try to create a unique persistent index but there are already + documents in the collection that violate the uniqueness requirement. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.13/develop/http-api/indexes/ttl.md b/site/content/3.13/develop/http-api/indexes/ttl.md index 97c0642271..54eeae5c40 100644 --- a/site/content/3.13/develop/http-api/indexes/ttl.md +++ b/site/content/3.13/develop/http-api/indexes/ttl.md @@ -13,8 +13,7 @@ paths: operationId: createIndexTtl description: | Creates a time-to-live (TTL) index for the collection `collection-name` if it - does not already exist. The call expects an object containing the index - details. + does not already exist. parameters: - name: database-name in: path @@ -74,18 +73,16 @@ paths: responses: '200': description: | - If the index already exists, then a *HTTP 200* is returned. + The index exists already. '201': description: | - If the index does not already exist and could be created, then a *HTTP 201* - is returned. + The index is created as there is no such existing index. '400': description: | - If the collection already contains another TTL index, then an *HTTP 400* is - returned, as there can be at most one TTL index per collection. + There is already a TTL index for the collection but there can only be one. '404': description: | - If the `collection-name` is unknown, then a *HTTP 404* is returned. + The collection is unknown. tags: - Indexes ``` diff --git a/site/content/3.13/develop/http-api/indexes/vector.md b/site/content/3.13/develop/http-api/indexes/vector.md new file mode 100644 index 0000000000..1472bdd4c2 --- /dev/null +++ b/site/content/3.13/develop/http-api/indexes/vector.md @@ -0,0 +1,135 @@ +--- +title: HTTP interface for vector indexes +menuTitle: Vector +weight: 35 +description: '' +--- +## Create a vector index + +```openapi +paths: + /_db/{database-name}/_api/index#vector: + post: + operationId: createIndexVector + description: | + Creates a vector index for the collection `collection-name`, if + it does not already exist. + parameters: + - name: database-name + in: path + required: true + example: _system + description: | + The name of the database. + schema: + type: string + - name: collection + in: query + required: true + description: | + The collection name. + schema: + type: string + requestBody: + content: + application/json: + schema: + type: object + required: + - type + - fields + - params + properties: + type: + description: | + The index type. Needs to be `"vector"`. + type: string + example: vector + name: + description: | + A user-defined name for the index for easier + identification. If not specified, a name is automatically generated. + type: string + fields: + description: | + A list with a single attribute path to specify + where the vector embedding is stored in each document. The vector data needs + to be populated before creating the index. + + If you want to index another vector embedding attribute, you need to create a + separate vector index. + type: array + items: + type: object + params: + description: | + The parameters as used by the Faiss library. + type: object + required: + - metric + - dimension + - nLists + properties: + metric: + description: | + Whether to use `cosine` or `l2` (Euclidean) distance calculation. + type: string + enum: ["cosine", "l2"] + dimension: + description: | + The vector dimension. The attribute to index needs to + have this many elements in the array that stores the vector embedding. + type: integer + nLists: + description: | + The number of centroids in the index. What value to choose + depends on the data distribution and chosen metric. According to + [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be + around `N / 15` where `N` is the number of documents in the collection, + respectively the number of documents in the shard for cluster deployments. + A bigger value produces more correct results but increases the training time + and thus how long it takes to build the index. It cannot be bigger than the + number of documents. + type: integer + defaultNProbe: + description: | + How many neighboring centroids to + consider for the search results by default. The larger the number, the slower + the search but the better the search results. The default is `1`. You should + generally use a higher value here or per query via the `nProbe` option of + the vector similarity functions. + type: integer + default: 1 + trainingIterations: + description: | + The number of iterations in the + training process. The default is `25`. Smaller values lead to a faster index + creation but may yield worse search results. + type: integer + default: 25 + factory: + description: | + You can specify a factory string to pass + through to the underlying Faiss library, allowing you to combine different + options, for example: + - `"IVF100_HNSW10,Flat"` + - `"IVF100,SQ4"` + - `"IVF10_HNSW5,Flat"` + - `"IVF100_HNSW5,PQ256x16"` + The base index must be an IVF to work with ArangoDB. For more information on + how to create these custom indexes, see the + [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + type: string + responses: + '200': + description: | + The index exists already. + '201': + description: | + The index is created as there is no such existing index. + '404': + description: | + The collection is unknown. + tags: + - Indexes +``` diff --git a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md index aa503fc882..5aaafd55a4 100644 --- a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -8,17 +8,16 @@ description: >- --- Vector indexes let you index vector embeddings stored in documents. Such vectors are arrays of numbers that represent the meaning and relationships of -data numerically. You can you quickly find a given number of semantically -similar documents by searching for close neighbors in a high-dimensional -vector space. +data numerically and can be generated with machine learning models. +You can then quickly find a given number of semantically similar documents by +searching for close neighbors in a high-dimensional vector space. The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/) to support L2 and cosine metrics. The index used is IndexIVFFlat, the quantizer for L2 is IndexFlatL2, and the cosine uses IndexFlatIP, where vectors are normalized before insertion and search. -Sometimes, if there is no relevant data found in the list, the faiss might not -produce the top K requested results. Therefore, only the found results is provided. +## How to use vector indexes {{< warning >}} The vector index is an experimental feature that you need to enable for the @@ -26,13 +25,35 @@ ArangoDB server with the `--experimental-vector-index` startup option. Once enabled for a deployment, it cannot be disabled anymore because it permanently changes how the data is managed by the RocksDB storage engine (it adds an additional column family). -{{< /warning >}} - -### How to use vector indexes -Creating an index triggers training the index on top of real data, which is a limitation that assumes the data already exists for the vector field upon which the index is created. -The number of training points depends on the nLists parameter; a bigger nLists will produce more correct results but will increase the training time necessary to build the index. +To restore a dump that contains vector indexes, the `--experimental-vector-index` +startup option needs to be enabled on the deployment you want to restore to. +{{< /warning >}} +1. Enable the experimental vector index feature. +2. Calculate vector embeddings using [ArangoDB's GraphML](../../../data-science/arangographml/_index.md) + capabilities (available in ArangoGraph) or using external tools. + Store each vector as an attribute in the respective document. +3. Create a vector index over this attribute. You need to choose which + similarity metric you want to use later for querying. See + [Vector index properties](#vector-index-properties) for all available + configuration options. +4. Run AQL queries that use [Vector similarity functions](../../../aql/functions/vector.md) + to retrieve a given number of similar documents relative to a vector embedding + you provide. + +Creating a vector index triggers training the index on top of real data and it +cannot be done incrementally without affecting the quality of the computation. +A vector index expects the data to already exist in the specified attribute. +This means **you cannot create a vector index for a collection upfront**, unlike +with all other index types. The documents need to already have vector embeddings +stored in an attribute that you then create the index over and train on. + +While it is possible to add more documents with vector embeddings over time, +they can only be assigned to existing clusters in the high-dimensional vector +space as determined by the original vector index training. This can be suboptimal +as the new data points might warrant a different clustering with different +centroids and the quality of vector search thus degrades. ## Vector index properties @@ -52,11 +73,16 @@ The number of training points depends on the nLists parameter; a bigger nLists w - **nLists** (number): The number of centroids in the index. What value to choose depends on the data distribution and chosen metric. According to [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be - around `15 * N` where `N` is the number of documents in the collection, + around `N / 15` where `N` is the number of documents in the collection, respectively the number of documents in the shard for cluster deployments. + A bigger value produces more correct results but increases the training time + and thus how long it takes to build the index. It cannot be bigger than the + number of documents. - **defaultNProbe** (number, _optional_): How many neighboring centroids to consider for the search results by default. The larger the number, the slower - the search but the better the search results. The default is `1`. + the search but the better the search results. The default is `1`. You should + generally use a higher value here or per query via the `nProbe` option of + the vector similarity functions. - **trainingIterations** (number, _optional_): The number of iterations in the training process. The default is `25`. Smaller values lead to a faster index creation but may yield worse search results. @@ -83,7 +109,8 @@ The number of training points depends on the nLists parameter; a bigger nLists w 3. Click **Add index**. 4. Select **Vector** as the **Type**. 5. Enter the name of the attribute that holds the vector embeddings into **Fields**. -6. Set the parameters for the vector index, see [Vector index parameters](#vector-index-parameters). +6. Set the parameters for the vector index. See [Vector index properties](#vector-index-properties) + under `param`. 7. Optionally give the index a user-defined name. 8. Click **Create**. {{< /tab >}} From 4270384a743cd1b4cec42c1b17e780de987a383d Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 5 Jun 2025 17:57:53 +0200 Subject: [PATCH 3/9] Version remark, OpenAPI minItems/maxItems and fix a type --- site/content/3.12/aql/functions/vector.md | 2 ++ site/content/3.12/develop/http-api/indexes/fulltext.md | 5 +++-- site/content/3.12/develop/http-api/indexes/geo-spatial.md | 2 ++ site/content/3.12/develop/http-api/indexes/inverted.md | 1 + .../3.12/develop/http-api/indexes/multi-dimensional.md | 1 + site/content/3.12/develop/http-api/indexes/persistent.md | 1 + site/content/3.12/develop/http-api/indexes/ttl.md | 4 +++- site/content/3.12/develop/http-api/indexes/vector.md | 8 ++++++-- .../indexing/working-with-indexes/vector-indexes.md | 2 ++ site/content/3.13/aql/functions/vector.md | 2 ++ site/content/3.13/develop/http-api/indexes/fulltext.md | 5 +++-- site/content/3.13/develop/http-api/indexes/geo-spatial.md | 2 ++ site/content/3.13/develop/http-api/indexes/inverted.md | 1 + .../3.13/develop/http-api/indexes/multi-dimensional.md | 1 + site/content/3.13/develop/http-api/indexes/persistent.md | 1 + site/content/3.13/develop/http-api/indexes/ttl.md | 4 +++- site/content/3.13/develop/http-api/indexes/vector.md | 8 ++++++-- .../indexing/working-with-indexes/vector-indexes.md | 2 ++ 18 files changed, 42 insertions(+), 10 deletions(-) diff --git a/site/content/3.12/aql/functions/vector.md b/site/content/3.12/aql/functions/vector.md index 680ece5eea..62fdbab5c5 100644 --- a/site/content/3.12/aql/functions/vector.md +++ b/site/content/3.12/aql/functions/vector.md @@ -6,6 +6,8 @@ description: >- The functions for vector search let you quickly find semantically similar documents utilizing indexed vector embeddings --- +Introduced in: v3.12.4 + To use vector search, you need to have vector embeddings stored in documents and the attribute that stores them needs to be indexed by a [vector index](../../index-and-search/indexing/working-with-indexes/vector-indexes.md). diff --git a/site/content/3.12/develop/http-api/indexes/fulltext.md b/site/content/3.12/develop/http-api/indexes/fulltext.md index 1d958b463f..cdcb37c915 100644 --- a/site/content/3.12/develop/http-api/indexes/fulltext.md +++ b/site/content/3.12/develop/http-api/indexes/fulltext.md @@ -59,9 +59,10 @@ paths: type: string fields: description: | - an array of attribute names. Currently, the array is limited - to exactly one attribute. + A list with exactly one attribute path. type: array + minItems: 1 + maxItems: 1 items: type: string minLength: diff --git a/site/content/3.12/develop/http-api/indexes/geo-spatial.md b/site/content/3.12/develop/http-api/indexes/geo-spatial.md index 16b4b3e020..960ae7de51 100644 --- a/site/content/3.12/develop/http-api/indexes/geo-spatial.md +++ b/site/content/3.12/develop/http-api/indexes/geo-spatial.md @@ -73,6 +73,8 @@ paths: All documents which do not have the attribute paths or which have values that are not suitable are ignored. type: array + minItems: 1 + maxItems: 2 items: type: string geoJson: diff --git a/site/content/3.12/develop/http-api/indexes/inverted.md b/site/content/3.12/develop/http-api/indexes/inverted.md index fe016f8436..cf4f2dc956 100644 --- a/site/content/3.12/develop/http-api/indexes/inverted.md +++ b/site/content/3.12/develop/http-api/indexes/inverted.md @@ -58,6 +58,7 @@ paths: default options, or objects to specify options for the fields (with the attribute path in the `name` property), or a mix of both. type: array + minItems: 1 items: type: object required: diff --git a/site/content/3.12/develop/http-api/indexes/multi-dimensional.md b/site/content/3.12/develop/http-api/indexes/multi-dimensional.md index f80f2722da..d0ddac4fce 100644 --- a/site/content/3.12/develop/http-api/indexes/multi-dimensional.md +++ b/site/content/3.12/develop/http-api/indexes/multi-dimensional.md @@ -56,6 +56,7 @@ paths: description: | An array of attribute names used for each dimension. Array expansions are not allowed. type: array + minItems: 1 items: type: string fieldValueTypes: diff --git a/site/content/3.12/develop/http-api/indexes/persistent.md b/site/content/3.12/develop/http-api/indexes/persistent.md index 15d9efafce..690ef723f5 100644 --- a/site/content/3.12/develop/http-api/indexes/persistent.md +++ b/site/content/3.12/develop/http-api/indexes/persistent.md @@ -81,6 +81,7 @@ paths: You can expand one array attribute with `[*]`. type: array + minItems: 1 items: type: string storedValues: diff --git a/site/content/3.12/develop/http-api/indexes/ttl.md b/site/content/3.12/develop/http-api/indexes/ttl.md index 54eeae5c40..b42d24426b 100644 --- a/site/content/3.12/develop/http-api/indexes/ttl.md +++ b/site/content/3.12/develop/http-api/indexes/ttl.md @@ -53,8 +53,10 @@ paths: type: string fields: description: | - an array with exactly one attribute path. + A list with exactly one attribute path. type: array + minItems: 1 + maxItems: 1 items: type: string expireAfter: diff --git a/site/content/3.12/develop/http-api/indexes/vector.md b/site/content/3.12/develop/http-api/indexes/vector.md index 1472bdd4c2..f41a023d5d 100644 --- a/site/content/3.12/develop/http-api/indexes/vector.md +++ b/site/content/3.12/develop/http-api/indexes/vector.md @@ -4,6 +4,8 @@ menuTitle: Vector weight: 35 description: '' --- +Introduced in: v3.12.4 + ## Create a vector index ```openapi @@ -52,15 +54,17 @@ paths: type: string fields: description: | - A list with a single attribute path to specify + A list with exactly one attribute path to specify where the vector embedding is stored in each document. The vector data needs to be populated before creating the index. If you want to index another vector embedding attribute, you need to create a separate vector index. type: array + minItems: 1 + maxItems: 1 items: - type: object + type: string params: description: | The parameters as used by the Faiss library. diff --git a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md index 5aaafd55a4..589186e9a0 100644 --- a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -6,6 +6,8 @@ description: >- You can index vector embeddings to allow queries to quickly find semantically similar documents --- +Introduced in: v3.12.4 + Vector indexes let you index vector embeddings stored in documents. Such vectors are arrays of numbers that represent the meaning and relationships of data numerically and can be generated with machine learning models. diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md index 680ece5eea..62fdbab5c5 100644 --- a/site/content/3.13/aql/functions/vector.md +++ b/site/content/3.13/aql/functions/vector.md @@ -6,6 +6,8 @@ description: >- The functions for vector search let you quickly find semantically similar documents utilizing indexed vector embeddings --- +Introduced in: v3.12.4 + To use vector search, you need to have vector embeddings stored in documents and the attribute that stores them needs to be indexed by a [vector index](../../index-and-search/indexing/working-with-indexes/vector-indexes.md). diff --git a/site/content/3.13/develop/http-api/indexes/fulltext.md b/site/content/3.13/develop/http-api/indexes/fulltext.md index 1d958b463f..cdcb37c915 100644 --- a/site/content/3.13/develop/http-api/indexes/fulltext.md +++ b/site/content/3.13/develop/http-api/indexes/fulltext.md @@ -59,9 +59,10 @@ paths: type: string fields: description: | - an array of attribute names. Currently, the array is limited - to exactly one attribute. + A list with exactly one attribute path. type: array + minItems: 1 + maxItems: 1 items: type: string minLength: diff --git a/site/content/3.13/develop/http-api/indexes/geo-spatial.md b/site/content/3.13/develop/http-api/indexes/geo-spatial.md index 16b4b3e020..960ae7de51 100644 --- a/site/content/3.13/develop/http-api/indexes/geo-spatial.md +++ b/site/content/3.13/develop/http-api/indexes/geo-spatial.md @@ -73,6 +73,8 @@ paths: All documents which do not have the attribute paths or which have values that are not suitable are ignored. type: array + minItems: 1 + maxItems: 2 items: type: string geoJson: diff --git a/site/content/3.13/develop/http-api/indexes/inverted.md b/site/content/3.13/develop/http-api/indexes/inverted.md index fe016f8436..cf4f2dc956 100644 --- a/site/content/3.13/develop/http-api/indexes/inverted.md +++ b/site/content/3.13/develop/http-api/indexes/inverted.md @@ -58,6 +58,7 @@ paths: default options, or objects to specify options for the fields (with the attribute path in the `name` property), or a mix of both. type: array + minItems: 1 items: type: object required: diff --git a/site/content/3.13/develop/http-api/indexes/multi-dimensional.md b/site/content/3.13/develop/http-api/indexes/multi-dimensional.md index f80f2722da..d0ddac4fce 100644 --- a/site/content/3.13/develop/http-api/indexes/multi-dimensional.md +++ b/site/content/3.13/develop/http-api/indexes/multi-dimensional.md @@ -56,6 +56,7 @@ paths: description: | An array of attribute names used for each dimension. Array expansions are not allowed. type: array + minItems: 1 items: type: string fieldValueTypes: diff --git a/site/content/3.13/develop/http-api/indexes/persistent.md b/site/content/3.13/develop/http-api/indexes/persistent.md index 15d9efafce..690ef723f5 100644 --- a/site/content/3.13/develop/http-api/indexes/persistent.md +++ b/site/content/3.13/develop/http-api/indexes/persistent.md @@ -81,6 +81,7 @@ paths: You can expand one array attribute with `[*]`. type: array + minItems: 1 items: type: string storedValues: diff --git a/site/content/3.13/develop/http-api/indexes/ttl.md b/site/content/3.13/develop/http-api/indexes/ttl.md index 54eeae5c40..b42d24426b 100644 --- a/site/content/3.13/develop/http-api/indexes/ttl.md +++ b/site/content/3.13/develop/http-api/indexes/ttl.md @@ -53,8 +53,10 @@ paths: type: string fields: description: | - an array with exactly one attribute path. + A list with exactly one attribute path. type: array + minItems: 1 + maxItems: 1 items: type: string expireAfter: diff --git a/site/content/3.13/develop/http-api/indexes/vector.md b/site/content/3.13/develop/http-api/indexes/vector.md index 1472bdd4c2..f41a023d5d 100644 --- a/site/content/3.13/develop/http-api/indexes/vector.md +++ b/site/content/3.13/develop/http-api/indexes/vector.md @@ -4,6 +4,8 @@ menuTitle: Vector weight: 35 description: '' --- +Introduced in: v3.12.4 + ## Create a vector index ```openapi @@ -52,15 +54,17 @@ paths: type: string fields: description: | - A list with a single attribute path to specify + A list with exactly one attribute path to specify where the vector embedding is stored in each document. The vector data needs to be populated before creating the index. If you want to index another vector embedding attribute, you need to create a separate vector index. type: array + minItems: 1 + maxItems: 1 items: - type: object + type: string params: description: | The parameters as used by the Faiss library. diff --git a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md index 5aaafd55a4..589186e9a0 100644 --- a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -6,6 +6,8 @@ description: >- You can index vector embeddings to allow queries to quickly find semantically similar documents --- +Introduced in: v3.12.4 + Vector indexes let you index vector embeddings stored in documents. Such vectors are arrays of numbers that represent the meaning and relationships of data numerically and can be generated with machine learning models. From 209675ab37540cb9bf1a41247bc285f8efac738a Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 5 Jun 2025 17:59:36 +0200 Subject: [PATCH 4/9] inBackground and parallelism are supported --- site/content/3.12/develop/http-api/indexes/vector.md | 10 ++++++++++ site/content/3.13/develop/http-api/indexes/vector.md | 10 ++++++++++ 2 files changed, 20 insertions(+) diff --git a/site/content/3.12/develop/http-api/indexes/vector.md b/site/content/3.12/develop/http-api/indexes/vector.md index f41a023d5d..905a8fb06b 100644 --- a/site/content/3.12/develop/http-api/indexes/vector.md +++ b/site/content/3.12/develop/http-api/indexes/vector.md @@ -65,6 +65,16 @@ paths: maxItems: 1 items: type: string + parallelism: + description: | + The number of threads to use for indexing the fields. Default: `2` + type: integer + inBackground: + description: | + This attribute can be set to `true` to create the index + in the background, not write-locking the underlying collection for + as long as if the index is built in the foreground. The default value is `false`. + type: boolean params: description: | The parameters as used by the Faiss library. diff --git a/site/content/3.13/develop/http-api/indexes/vector.md b/site/content/3.13/develop/http-api/indexes/vector.md index f41a023d5d..905a8fb06b 100644 --- a/site/content/3.13/develop/http-api/indexes/vector.md +++ b/site/content/3.13/develop/http-api/indexes/vector.md @@ -65,6 +65,16 @@ paths: maxItems: 1 items: type: string + parallelism: + description: | + The number of threads to use for indexing the fields. Default: `2` + type: integer + inBackground: + description: | + This attribute can be set to `true` to create the index + in the background, not write-locking the underlying collection for + as long as if the index is built in the foreground. The default value is `false`. + type: boolean params: description: | The parameters as used by the Faiss library. From 03050a5c026381f4f7326e9461fd6bb57135dfeb Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Tue, 10 Jun 2025 12:38:11 +0200 Subject: [PATCH 5/9] Review feedback, address cosine metric issue, reword inBackground, add parallelism --- site/content/3.12/aql/functions/vector.md | 44 +++++++++---------- .../3.12/develop/http-api/indexes/_index.md | 6 +-- .../3.12/develop/http-api/indexes/fulltext.md | 7 +-- .../develop/http-api/indexes/geo-spatial.md | 7 +-- .../3.12/develop/http-api/indexes/inverted.md | 7 +-- .../http-api/indexes/multi-dimensional.md | 7 +-- .../develop/http-api/indexes/persistent.md | 7 +-- .../3.12/develop/http-api/indexes/ttl.md | 7 +-- .../3.12/develop/http-api/indexes/vector.md | 27 +++++++----- .../working-with-indexes/vector-indexes.md | 22 +++++++--- site/content/3.13/aql/functions/vector.md | 44 +++++++++---------- .../3.13/develop/http-api/indexes/_index.md | 6 +-- .../3.13/develop/http-api/indexes/fulltext.md | 7 +-- .../develop/http-api/indexes/geo-spatial.md | 7 +-- .../3.13/develop/http-api/indexes/inverted.md | 7 +-- .../http-api/indexes/multi-dimensional.md | 7 +-- .../develop/http-api/indexes/persistent.md | 7 +-- .../3.13/develop/http-api/indexes/ttl.md | 7 +-- .../3.13/develop/http-api/indexes/vector.md | 27 +++++++----- .../working-with-indexes/vector-indexes.md | 22 +++++++--- 20 files changed, 158 insertions(+), 124 deletions(-) diff --git a/site/content/3.12/aql/functions/vector.md b/site/content/3.12/aql/functions/vector.md index 62fdbab5c5..ddbcd3c906 100644 --- a/site/content/3.12/aql/functions/vector.md +++ b/site/content/3.12/aql/functions/vector.md @@ -42,16 +42,16 @@ Example: ```aql FOR doc IN coll - SORT APPROX_NEAR_L2(doc.vector, @q) + SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC LIMIT 5 RETURN doc ``` -For this query, a vector index over the `vector` attribute and with the `l2` +For this query, a vector index over the `vector` attribute and with the `cosine` metric is required. The `@q` bind variable needs to be a vector (array of numbers) with the dimension as specified in the vector index. It defines the point at -which to look for neighbors (up to `5` in this case). How many neighbors can be -found depends on the data as well as the search effort (see the `nProbe` option). +which to look for similar documents (up to `5` in this case). How many documents can +be found depends on the data as well as the search effort (see the `nProbe` option). {{< info >}} - If there is more than one suitable vector index over the same attribute, it is @@ -67,9 +67,9 @@ found depends on the data as well as the search effort (see the `nProbe` option) Retrieve the approximate angular similarity using the cosine metric, accelerated by a matching vector index. -The closer the cosine similarity value is to 1, the more similar the two vectors +The higher the cosine similarity value is, the more similar the two vectors are. The closer it is to 0, the more different they are. The value can also -be up to -1, indicating that the vectors are not similar and point in opposite +be negative, indicating that the vectors are not similar and point in opposite directions. You need to sort in descending order so that the most similar documents come first, which is what a vector index using the `cosine` metric can provide. @@ -80,16 +80,16 @@ can provide. - **vector2** (array of numbers): The second vector. Either this parameter or `vector1` needs to reference a stored attribute holding the vector embedding. - **options** (object, _optional_): - - **nProbe** (number, _optional_): How many neighboring centroids to consider - for the search results. The larger the number, the slower the search but the - better the search results. If not specified, the `defaultNProbe` value of - the vector index is used. + - **nProbe** (number, _optional_): How many neighboring centroids respectively + closest Voronoi cells to consider for the search results. The larger the number, + the slower the search but the better the search results. If not specified, the + `defaultNProbe` value of the vector index is used. - returns **similarity** (number): The approximate angular similarity between both vectors. **Examples** -Return the documents of up to `10` neighbors that are the closest to the vector +Return up to `10` similar documents based on their closeness to the vector `@q` according to the cosine metric: ```aql @@ -99,8 +99,8 @@ FOR doc IN coll RETURN doc ``` -Return the similarity value and the documents of up to `5` close neighbors, -considering `20` neighboring centroids: +Return up to `5` similar documents as well as the similarity value, +considering `20` neighboring centroids respectively closest Voronoi cells: ```aql FOR doc IN coll @@ -110,9 +110,9 @@ FOR doc IN coll RETURN MERGE( { similarity }, doc) ``` -Return the similarity and the document keys of up to `3` neighbors for multiple -vectors using a subquery, here taking from ten random documents of the same -collection: +Return the similarity value and the document keys of up to `3` similar documents +for multiple input vectors using a subquery. In this example, the input vectors +are taken from ten random documents of the same collection: ```aql FOR docOuter IN coll @@ -154,7 +154,7 @@ the `l2` metric can provide. **Examples** -Return the documents of up to `10` neighbors that are the closest to the vector +Return up to `10` similar documents based on their closeness to the vector `@q` according to the L2 (Euclidean) metric: ```aql @@ -164,8 +164,8 @@ FOR doc IN coll RETURN doc ``` -Return the similarity value and the documents of up to `5` close neighbors, -considering `20` neighboring centroids: +Return up to `5` similar documents as well as the similarity value, +considering `20` neighboring centroids respectively closest Voronoi cells: ```aql FOR doc IN coll @@ -175,9 +175,9 @@ FOR doc IN coll RETURN MERGE( { similarity }, doc) ``` -Return the similarity and the document keys of up to `3` close neighbors for -multiple vectors using a subquery, here taking from ten random documents of the -same collection: +Return the similarity value and the document keys of up to `3` similar documents +for multiple input vectors using a subquery. In this example, the input vectors +are taken from ten random documents of the same collection: ```aql FOR docOuter IN coll diff --git a/site/content/3.12/develop/http-api/indexes/_index.md b/site/content/3.12/develop/http-api/indexes/_index.md index f43b17fcb5..479dfd2604 100644 --- a/site/content/3.12/develop/http-api/indexes/_index.md +++ b/site/content/3.12/develop/http-api/indexes/_index.md @@ -247,9 +247,9 @@ paths: `cacheEnabled` defaults to `false` and should only be used for indexes that are known to benefit from an extra layer of caching. - The optional attribute **inBackground** can be set to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. + The optional attribute **inBackground** can be set to `true` to keep the + collection/shards available for write operations by not using an exclusive + write lock for the duration of the index creation. parameters: - name: database-name in: path diff --git a/site/content/3.12/develop/http-api/indexes/fulltext.md b/site/content/3.12/develop/http-api/indexes/fulltext.md index cdcb37c915..687ca29230 100644 --- a/site/content/3.12/develop/http-api/indexes/fulltext.md +++ b/site/content/3.12/develop/http-api/indexes/fulltext.md @@ -73,10 +73,11 @@ paths: type: integer inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.12/develop/http-api/indexes/geo-spatial.md b/site/content/3.12/develop/http-api/indexes/geo-spatial.md index 960ae7de51..ad51599a94 100644 --- a/site/content/3.12/develop/http-api/indexes/geo-spatial.md +++ b/site/content/3.12/develop/http-api/indexes/geo-spatial.md @@ -101,10 +101,11 @@ paths: type: boolean inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.12/develop/http-api/indexes/inverted.md b/site/content/3.12/develop/http-api/indexes/inverted.md index cf4f2dc956..4826aaae55 100644 --- a/site/content/3.12/develop/http-api/indexes/inverted.md +++ b/site/content/3.12/develop/http-api/indexes/inverted.md @@ -489,10 +489,11 @@ paths: type: integer inBackground: description: | - This attribute can be set to `true` to create the index - in the background, not write-locking the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false cleanupIntervalStep: description: | Wait at least this many commits between removing unused files in the diff --git a/site/content/3.12/develop/http-api/indexes/multi-dimensional.md b/site/content/3.12/develop/http-api/indexes/multi-dimensional.md index d0ddac4fce..53aba9abe6 100644 --- a/site/content/3.12/develop/http-api/indexes/multi-dimensional.md +++ b/site/content/3.12/develop/http-api/indexes/multi-dimensional.md @@ -130,10 +130,11 @@ paths: default: true inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.12/develop/http-api/indexes/persistent.md b/site/content/3.12/develop/http-api/indexes/persistent.md index 690ef723f5..99be09a69f 100644 --- a/site/content/3.12/develop/http-api/indexes/persistent.md +++ b/site/content/3.12/develop/http-api/indexes/persistent.md @@ -159,10 +159,11 @@ paths: type: boolean inBackground: description: | - This attribute can be set to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.12/develop/http-api/indexes/ttl.md b/site/content/3.12/develop/http-api/indexes/ttl.md index b42d24426b..11fb88d96a 100644 --- a/site/content/3.12/develop/http-api/indexes/ttl.md +++ b/site/content/3.12/develop/http-api/indexes/ttl.md @@ -68,10 +68,11 @@ paths: type: number inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.12/develop/http-api/indexes/vector.md b/site/content/3.12/develop/http-api/indexes/vector.md index 905a8fb06b..a98bd79166 100644 --- a/site/content/3.12/develop/http-api/indexes/vector.md +++ b/site/content/3.12/develop/http-api/indexes/vector.md @@ -67,14 +67,15 @@ paths: type: string parallelism: description: | - The number of threads to use for indexing the fields. Default: `2` + The number of threads to use for indexing. Default: `2` type: integer inBackground: description: | - This attribute can be set to `true` to create the index - in the background, not write-locking the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false params: description: | The parameters as used by the Faiss library. @@ -96,10 +97,11 @@ paths: type: integer nLists: description: | - The number of centroids in the index. What value to choose + The number of Voronoi cells to partition the vector space + into, respectively the number of centroids in the index. What value to choose depends on the data distribution and chosen metric. According to [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be - around `N / 15` where `N` is the number of documents in the collection, + around `15 * sqrt(N)` where `N` is the number of documents in the collection, respectively the number of documents in the shard for cluster deployments. A bigger value produces more correct results but increases the training time and thus how long it takes to build the index. It cannot be bigger than the @@ -123,16 +125,17 @@ paths: default: 25 factory: description: | - You can specify a factory string to pass - through to the underlying Faiss library, allowing you to combine different - options, for example: + You can specify an index factory string that is + forwarded to the underlying Faiss library, allowing you to combine different + advanced options. Examples: - `"IVF100_HNSW10,Flat"` - `"IVF100,SQ4"` - `"IVF10_HNSW5,Flat"` - `"IVF100_HNSW5,PQ256x16"` - The base index must be an IVF to work with ArangoDB. For more information on - how to create these custom indexes, see the - [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + The base index must be an inverted file (IVF) to work with ArangoDB. + If you don't specify an index factory, the value is equivalent to + `IVF,Flat`. For more information on how to create these custom + indexes, see the [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). type: string responses: '200': diff --git a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md index 589186e9a0..90b82edbf9 100644 --- a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -68,11 +68,18 @@ centroids and the quality of vector search thus degrades. If you want to index another vector embedding attribute, you need to create a separate vector index. +- **parallelism** (number): + The number of threads to use for indexing. The default is `2`. +- **inBackground** (boolean): + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. The default is `false`. - **params**: The parameters as used by the Faiss library. - **metric** (string): Whether to use `cosine` or `l2` (Euclidean) distance calculation. - **dimension** (number): The vector dimension. The attribute to index needs to have this many elements in the array that stores the vector embedding. - - **nLists** (number): The number of centroids in the index. What value to choose + - **nLists** (number): The number of Voronoi cells to partition the vector space + into, respectively the number of centroids in the index. What value to choose depends on the data distribution and chosen metric. According to [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be around `N / 15` where `N` is the number of documents in the collection, @@ -88,16 +95,17 @@ centroids and the quality of vector search thus degrades. - **trainingIterations** (number, _optional_): The number of iterations in the training process. The default is `25`. Smaller values lead to a faster index creation but may yield worse search results. - - **factory** (string, _optional_): You can specify a factory string to pass - through to the underlying Faiss library, allowing you to combine different - options, for example: + - **factory** (string, _optional_): You can specify an index factory string that is + forwarded to the underlying Faiss library, allowing you to combine different + advanced options. Examples: - `"IVF100_HNSW10,Flat"` - `"IVF100,SQ4"` - `"IVF10_HNSW5,Flat"` - `"IVF100_HNSW5,PQ256x16"` - The base index must be an IVF to work with ArangoDB. For more information on - how to create these custom indexes, see the - [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + The base index must be an inverted file (IVF) to work with ArangoDB. + If you don't specify an index factory, the value is equivalent to + `IVF,Flat`. For more information on how to create these custom + indexes, see the [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). ## Interfaces diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md index 62fdbab5c5..ddbcd3c906 100644 --- a/site/content/3.13/aql/functions/vector.md +++ b/site/content/3.13/aql/functions/vector.md @@ -42,16 +42,16 @@ Example: ```aql FOR doc IN coll - SORT APPROX_NEAR_L2(doc.vector, @q) + SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC LIMIT 5 RETURN doc ``` -For this query, a vector index over the `vector` attribute and with the `l2` +For this query, a vector index over the `vector` attribute and with the `cosine` metric is required. The `@q` bind variable needs to be a vector (array of numbers) with the dimension as specified in the vector index. It defines the point at -which to look for neighbors (up to `5` in this case). How many neighbors can be -found depends on the data as well as the search effort (see the `nProbe` option). +which to look for similar documents (up to `5` in this case). How many documents can +be found depends on the data as well as the search effort (see the `nProbe` option). {{< info >}} - If there is more than one suitable vector index over the same attribute, it is @@ -67,9 +67,9 @@ found depends on the data as well as the search effort (see the `nProbe` option) Retrieve the approximate angular similarity using the cosine metric, accelerated by a matching vector index. -The closer the cosine similarity value is to 1, the more similar the two vectors +The higher the cosine similarity value is, the more similar the two vectors are. The closer it is to 0, the more different they are. The value can also -be up to -1, indicating that the vectors are not similar and point in opposite +be negative, indicating that the vectors are not similar and point in opposite directions. You need to sort in descending order so that the most similar documents come first, which is what a vector index using the `cosine` metric can provide. @@ -80,16 +80,16 @@ can provide. - **vector2** (array of numbers): The second vector. Either this parameter or `vector1` needs to reference a stored attribute holding the vector embedding. - **options** (object, _optional_): - - **nProbe** (number, _optional_): How many neighboring centroids to consider - for the search results. The larger the number, the slower the search but the - better the search results. If not specified, the `defaultNProbe` value of - the vector index is used. + - **nProbe** (number, _optional_): How many neighboring centroids respectively + closest Voronoi cells to consider for the search results. The larger the number, + the slower the search but the better the search results. If not specified, the + `defaultNProbe` value of the vector index is used. - returns **similarity** (number): The approximate angular similarity between both vectors. **Examples** -Return the documents of up to `10` neighbors that are the closest to the vector +Return up to `10` similar documents based on their closeness to the vector `@q` according to the cosine metric: ```aql @@ -99,8 +99,8 @@ FOR doc IN coll RETURN doc ``` -Return the similarity value and the documents of up to `5` close neighbors, -considering `20` neighboring centroids: +Return up to `5` similar documents as well as the similarity value, +considering `20` neighboring centroids respectively closest Voronoi cells: ```aql FOR doc IN coll @@ -110,9 +110,9 @@ FOR doc IN coll RETURN MERGE( { similarity }, doc) ``` -Return the similarity and the document keys of up to `3` neighbors for multiple -vectors using a subquery, here taking from ten random documents of the same -collection: +Return the similarity value and the document keys of up to `3` similar documents +for multiple input vectors using a subquery. In this example, the input vectors +are taken from ten random documents of the same collection: ```aql FOR docOuter IN coll @@ -154,7 +154,7 @@ the `l2` metric can provide. **Examples** -Return the documents of up to `10` neighbors that are the closest to the vector +Return up to `10` similar documents based on their closeness to the vector `@q` according to the L2 (Euclidean) metric: ```aql @@ -164,8 +164,8 @@ FOR doc IN coll RETURN doc ``` -Return the similarity value and the documents of up to `5` close neighbors, -considering `20` neighboring centroids: +Return up to `5` similar documents as well as the similarity value, +considering `20` neighboring centroids respectively closest Voronoi cells: ```aql FOR doc IN coll @@ -175,9 +175,9 @@ FOR doc IN coll RETURN MERGE( { similarity }, doc) ``` -Return the similarity and the document keys of up to `3` close neighbors for -multiple vectors using a subquery, here taking from ten random documents of the -same collection: +Return the similarity value and the document keys of up to `3` similar documents +for multiple input vectors using a subquery. In this example, the input vectors +are taken from ten random documents of the same collection: ```aql FOR docOuter IN coll diff --git a/site/content/3.13/develop/http-api/indexes/_index.md b/site/content/3.13/develop/http-api/indexes/_index.md index f43b17fcb5..479dfd2604 100644 --- a/site/content/3.13/develop/http-api/indexes/_index.md +++ b/site/content/3.13/develop/http-api/indexes/_index.md @@ -247,9 +247,9 @@ paths: `cacheEnabled` defaults to `false` and should only be used for indexes that are known to benefit from an extra layer of caching. - The optional attribute **inBackground** can be set to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. + The optional attribute **inBackground** can be set to `true` to keep the + collection/shards available for write operations by not using an exclusive + write lock for the duration of the index creation. parameters: - name: database-name in: path diff --git a/site/content/3.13/develop/http-api/indexes/fulltext.md b/site/content/3.13/develop/http-api/indexes/fulltext.md index cdcb37c915..687ca29230 100644 --- a/site/content/3.13/develop/http-api/indexes/fulltext.md +++ b/site/content/3.13/develop/http-api/indexes/fulltext.md @@ -73,10 +73,11 @@ paths: type: integer inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.13/develop/http-api/indexes/geo-spatial.md b/site/content/3.13/develop/http-api/indexes/geo-spatial.md index 960ae7de51..ad51599a94 100644 --- a/site/content/3.13/develop/http-api/indexes/geo-spatial.md +++ b/site/content/3.13/develop/http-api/indexes/geo-spatial.md @@ -101,10 +101,11 @@ paths: type: boolean inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.13/develop/http-api/indexes/inverted.md b/site/content/3.13/develop/http-api/indexes/inverted.md index cf4f2dc956..4826aaae55 100644 --- a/site/content/3.13/develop/http-api/indexes/inverted.md +++ b/site/content/3.13/develop/http-api/indexes/inverted.md @@ -489,10 +489,11 @@ paths: type: integer inBackground: description: | - This attribute can be set to `true` to create the index - in the background, not write-locking the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false cleanupIntervalStep: description: | Wait at least this many commits between removing unused files in the diff --git a/site/content/3.13/develop/http-api/indexes/multi-dimensional.md b/site/content/3.13/develop/http-api/indexes/multi-dimensional.md index d0ddac4fce..53aba9abe6 100644 --- a/site/content/3.13/develop/http-api/indexes/multi-dimensional.md +++ b/site/content/3.13/develop/http-api/indexes/multi-dimensional.md @@ -130,10 +130,11 @@ paths: default: true inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.13/develop/http-api/indexes/persistent.md b/site/content/3.13/develop/http-api/indexes/persistent.md index 690ef723f5..99be09a69f 100644 --- a/site/content/3.13/develop/http-api/indexes/persistent.md +++ b/site/content/3.13/develop/http-api/indexes/persistent.md @@ -159,10 +159,11 @@ paths: type: boolean inBackground: description: | - This attribute can be set to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.13/develop/http-api/indexes/ttl.md b/site/content/3.13/develop/http-api/indexes/ttl.md index b42d24426b..11fb88d96a 100644 --- a/site/content/3.13/develop/http-api/indexes/ttl.md +++ b/site/content/3.13/develop/http-api/indexes/ttl.md @@ -68,10 +68,11 @@ paths: type: number inBackground: description: | - You can set this option to `true` to create the index - in the background, which will not write-lock the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false responses: '200': description: | diff --git a/site/content/3.13/develop/http-api/indexes/vector.md b/site/content/3.13/develop/http-api/indexes/vector.md index 905a8fb06b..a98bd79166 100644 --- a/site/content/3.13/develop/http-api/indexes/vector.md +++ b/site/content/3.13/develop/http-api/indexes/vector.md @@ -67,14 +67,15 @@ paths: type: string parallelism: description: | - The number of threads to use for indexing the fields. Default: `2` + The number of threads to use for indexing. Default: `2` type: integer inBackground: description: | - This attribute can be set to `true` to create the index - in the background, not write-locking the underlying collection for - as long as if the index is built in the foreground. The default value is `false`. + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. type: boolean + default: false params: description: | The parameters as used by the Faiss library. @@ -96,10 +97,11 @@ paths: type: integer nLists: description: | - The number of centroids in the index. What value to choose + The number of Voronoi cells to partition the vector space + into, respectively the number of centroids in the index. What value to choose depends on the data distribution and chosen metric. According to [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be - around `N / 15` where `N` is the number of documents in the collection, + around `15 * sqrt(N)` where `N` is the number of documents in the collection, respectively the number of documents in the shard for cluster deployments. A bigger value produces more correct results but increases the training time and thus how long it takes to build the index. It cannot be bigger than the @@ -123,16 +125,17 @@ paths: default: 25 factory: description: | - You can specify a factory string to pass - through to the underlying Faiss library, allowing you to combine different - options, for example: + You can specify an index factory string that is + forwarded to the underlying Faiss library, allowing you to combine different + advanced options. Examples: - `"IVF100_HNSW10,Flat"` - `"IVF100,SQ4"` - `"IVF10_HNSW5,Flat"` - `"IVF100_HNSW5,PQ256x16"` - The base index must be an IVF to work with ArangoDB. For more information on - how to create these custom indexes, see the - [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + The base index must be an inverted file (IVF) to work with ArangoDB. + If you don't specify an index factory, the value is equivalent to + `IVF,Flat`. For more information on how to create these custom + indexes, see the [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). type: string responses: '200': diff --git a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md index 589186e9a0..90b82edbf9 100644 --- a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -68,11 +68,18 @@ centroids and the quality of vector search thus degrades. If you want to index another vector embedding attribute, you need to create a separate vector index. +- **parallelism** (number): + The number of threads to use for indexing. The default is `2`. +- **inBackground** (boolean): + Set this option to `true` to keep the collection/shards available for + write operations by not using an exclusive write lock for the duration + of the index creation. The default is `false`. - **params**: The parameters as used by the Faiss library. - **metric** (string): Whether to use `cosine` or `l2` (Euclidean) distance calculation. - **dimension** (number): The vector dimension. The attribute to index needs to have this many elements in the array that stores the vector embedding. - - **nLists** (number): The number of centroids in the index. What value to choose + - **nLists** (number): The number of Voronoi cells to partition the vector space + into, respectively the number of centroids in the index. What value to choose depends on the data distribution and chosen metric. According to [The Faiss library paper](https://arxiv.org/abs/2401.08281), it should be around `N / 15` where `N` is the number of documents in the collection, @@ -88,16 +95,17 @@ centroids and the quality of vector search thus degrades. - **trainingIterations** (number, _optional_): The number of iterations in the training process. The default is `25`. Smaller values lead to a faster index creation but may yield worse search results. - - **factory** (string, _optional_): You can specify a factory string to pass - through to the underlying Faiss library, allowing you to combine different - options, for example: + - **factory** (string, _optional_): You can specify an index factory string that is + forwarded to the underlying Faiss library, allowing you to combine different + advanced options. Examples: - `"IVF100_HNSW10,Flat"` - `"IVF100,SQ4"` - `"IVF10_HNSW5,Flat"` - `"IVF100_HNSW5,PQ256x16"` - The base index must be an IVF to work with ArangoDB. For more information on - how to create these custom indexes, see the - [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). + The base index must be an inverted file (IVF) to work with ArangoDB. + If you don't specify an index factory, the value is equivalent to + `IVF,Flat`. For more information on how to create these custom + indexes, see the [Faiss Wiki](https://github.com/facebookresearch/faiss/wiki/The-index-factory). ## Interfaces From 1aa411baf1aeed43c989871e75c75a324a1321f9 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Tue, 10 Jun 2025 13:25:36 +0200 Subject: [PATCH 6/9] Remove leftover line --- site/content/3.12/aql/functions/vector.md | 2 -- site/content/3.13/aql/functions/vector.md | 2 -- 2 files changed, 4 deletions(-) diff --git a/site/content/3.12/aql/functions/vector.md b/site/content/3.12/aql/functions/vector.md index ddbcd3c906..e6a6de0134 100644 --- a/site/content/3.12/aql/functions/vector.md +++ b/site/content/3.12/aql/functions/vector.md @@ -76,7 +76,6 @@ can provide. - **vector1** (array of numbers): The first vector. Either this parameter or `vector2` needs to reference a stored attribute holding the vector embedding. - attribute of a stored document that stores a vector, like `doc.vector` - **vector2** (array of numbers): The second vector. Either this parameter or `vector1` needs to reference a stored attribute holding the vector embedding. - **options** (object, _optional_): @@ -141,7 +140,6 @@ the `l2` metric can provide. - **vector1** (array of numbers): The first vector. Either this parameter or `vector2` needs to reference a stored attribute holding the vector embedding. - attribute of a stored document that stores a vector, like `doc.vector` - **vector2** (array of numbers): The second vector. Either this parameter or `vector1` needs to reference a stored attribute holding the vector embedding. - **options** (object, _optional_): diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md index ddbcd3c906..e6a6de0134 100644 --- a/site/content/3.13/aql/functions/vector.md +++ b/site/content/3.13/aql/functions/vector.md @@ -76,7 +76,6 @@ can provide. - **vector1** (array of numbers): The first vector. Either this parameter or `vector2` needs to reference a stored attribute holding the vector embedding. - attribute of a stored document that stores a vector, like `doc.vector` - **vector2** (array of numbers): The second vector. Either this parameter or `vector1` needs to reference a stored attribute holding the vector embedding. - **options** (object, _optional_): @@ -141,7 +140,6 @@ the `l2` metric can provide. - **vector1** (array of numbers): The first vector. Either this parameter or `vector2` needs to reference a stored attribute holding the vector embedding. - attribute of a stored document that stores a vector, like `doc.vector` - **vector2** (array of numbers): The second vector. Either this parameter or `vector1` needs to reference a stored attribute holding the vector embedding. - **options** (object, _optional_): From 5a017d70622defbbdb4af43adf58192e31c2f1e2 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Wed, 11 Jun 2025 11:58:21 +0200 Subject: [PATCH 7/9] Add internal links to release notes --- .../version-3.12/whats-new-in-3-12.md | 15 ++++++++++++--- .../version-3.12/whats-new-in-3-12.md | 15 ++++++++++++--- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 39ffd8435e..636d14377b 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -1325,15 +1325,24 @@ you to find items with similar properties by comparing vector embeddings, which are numerical representations generated by machine learning models. To try out this feature, start an ArangoDB server (`arangod`) with the -`--experimental-vector-index` startup option and follow the guide in this -blog post: +`--experimental-vector-index` startup option. You need to generate +vector embeddings before creating a vector index. For more information about +the vector index type including the available settings, see the +[Vector indexes](../../index-and-search/indexing/working-with-indexes/vector-indexes.md) +documentation. + +You can also follow the guide in this blog post: [Vector Search in ArangoDB: Practical Insights and Hands-On Examples](https://arangodb.com/2024/11/vector-search-in-arangodb-practical-insights-and-hands-on-examples/) If the vector index type is enabled, the following new AQL functions are -available: +available to retrieve similar documents: + - `APPROX_NEAR_COSINE()` - `APPROX_NEAR_L2()` +For how to use these functions as well as query examples, see +[Vector search functions in AQL](../../aql/functions/vector.md). + Two startup options for the storage engine related to vector indexes have been added: - `--rocksdb.max-write-buffer-number-vector` diff --git a/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md index 39ffd8435e..636d14377b 100644 --- a/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md @@ -1325,15 +1325,24 @@ you to find items with similar properties by comparing vector embeddings, which are numerical representations generated by machine learning models. To try out this feature, start an ArangoDB server (`arangod`) with the -`--experimental-vector-index` startup option and follow the guide in this -blog post: +`--experimental-vector-index` startup option. You need to generate +vector embeddings before creating a vector index. For more information about +the vector index type including the available settings, see the +[Vector indexes](../../index-and-search/indexing/working-with-indexes/vector-indexes.md) +documentation. + +You can also follow the guide in this blog post: [Vector Search in ArangoDB: Practical Insights and Hands-On Examples](https://arangodb.com/2024/11/vector-search-in-arangodb-practical-insights-and-hands-on-examples/) If the vector index type is enabled, the following new AQL functions are -available: +available to retrieve similar documents: + - `APPROX_NEAR_COSINE()` - `APPROX_NEAR_L2()` +For how to use these functions as well as query examples, see +[Vector search functions in AQL](../../aql/functions/vector.md). + Two startup options for the storage engine related to vector indexes have been added: - `--rocksdb.max-write-buffer-number-vector` From 13f4730993e3d0933387065898d6270440ab7071 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Tue, 24 Jun 2025 08:48:02 +0200 Subject: [PATCH 8/9] Cosine similarity value out of range has been fixed --- site/content/3.12/aql/functions/vector.md | 6 +++--- .../version-3.12/incompatible-changes-in-3-12.md | 11 +++++++++++ site/content/3.13/aql/functions/vector.md | 6 +++--- .../version-3.12/incompatible-changes-in-3-12.md | 11 +++++++++++ 4 files changed, 28 insertions(+), 6 deletions(-) diff --git a/site/content/3.12/aql/functions/vector.md b/site/content/3.12/aql/functions/vector.md index e6a6de0134..9a43bb6297 100644 --- a/site/content/3.12/aql/functions/vector.md +++ b/site/content/3.12/aql/functions/vector.md @@ -67,9 +67,9 @@ be found depends on the data as well as the search effort (see the `nProbe` opti Retrieve the approximate angular similarity using the cosine metric, accelerated by a matching vector index. -The higher the cosine similarity value is, the more similar the two vectors -are. The closer it is to 0, the more different they are. The value can also -be negative, indicating that the vectors are not similar and point in opposite +The closer the cosine similarity value is to 1, the more similar the two vectors +are. The closer it is to 0, the more different they are. The value can also be +negative up to -1, indicating that the vectors are not similar and point in opposite directions. You need to sort in descending order so that the most similar documents come first, which is what a vector index using the `cosine` metric can provide. diff --git a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md index 30284d66ee..4e5099a80e 100644 --- a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md @@ -897,6 +897,17 @@ the following steps. 4. Restore the dump to the new deployment. You can directly move from any 3.11 or 3.12 version to 3.12.4 (or later) this way. +## Cosine similarity fix for vector indexes + +Introduced in: v3.12.6 + +A normalization issue has been addressed for the experimental vector index type. +It was possible for the cosine similarity value returned by `APPROX_NEAR_COSINE()` +to be outside the expected range of `[-1, 1]`. + +It is recommended to recreate all vector indexes that use the `cosine` metric +after upgrading to v3.12.6 or later. + ## HTTP RESTful API ### JavaScript-based traversal using `/_api/traversal` removed diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md index e6a6de0134..9a43bb6297 100644 --- a/site/content/3.13/aql/functions/vector.md +++ b/site/content/3.13/aql/functions/vector.md @@ -67,9 +67,9 @@ be found depends on the data as well as the search effort (see the `nProbe` opti Retrieve the approximate angular similarity using the cosine metric, accelerated by a matching vector index. -The higher the cosine similarity value is, the more similar the two vectors -are. The closer it is to 0, the more different they are. The value can also -be negative, indicating that the vectors are not similar and point in opposite +The closer the cosine similarity value is to 1, the more similar the two vectors +are. The closer it is to 0, the more different they are. The value can also be +negative up to -1, indicating that the vectors are not similar and point in opposite directions. You need to sort in descending order so that the most similar documents come first, which is what a vector index using the `cosine` metric can provide. diff --git a/site/content/3.13/release-notes/version-3.12/incompatible-changes-in-3-12.md b/site/content/3.13/release-notes/version-3.12/incompatible-changes-in-3-12.md index 30284d66ee..4e5099a80e 100644 --- a/site/content/3.13/release-notes/version-3.12/incompatible-changes-in-3-12.md +++ b/site/content/3.13/release-notes/version-3.12/incompatible-changes-in-3-12.md @@ -897,6 +897,17 @@ the following steps. 4. Restore the dump to the new deployment. You can directly move from any 3.11 or 3.12 version to 3.12.4 (or later) this way. +## Cosine similarity fix for vector indexes + +Introduced in: v3.12.6 + +A normalization issue has been addressed for the experimental vector index type. +It was possible for the cosine similarity value returned by `APPROX_NEAR_COSINE()` +to be outside the expected range of `[-1, 1]`. + +It is recommended to recreate all vector indexes that use the `cosine` metric +after upgrading to v3.12.6 or later. + ## HTTP RESTful API ### JavaScript-based traversal using `/_api/traversal` removed From 2262a4a4eb1c406cddcffe435aa536fc6eb7270d Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 30 Jun 2025 17:38:03 +0200 Subject: [PATCH 9/9] Add innerProduct metric --- site/content/3.12/aql/functions/vector.md | 89 ++++++++++++++++--- .../3.12/develop/http-api/indexes/vector.md | 11 ++- .../working-with-indexes/vector-indexes.md | 13 +-- .../version-3.12/whats-new-in-3-12.md | 12 +++ site/content/3.13/aql/functions/vector.md | 89 ++++++++++++++++--- .../3.13/develop/http-api/indexes/vector.md | 11 ++- .../working-with-indexes/vector-indexes.md | 13 +-- .../version-3.12/whats-new-in-3-12.md | 12 +++ 8 files changed, 214 insertions(+), 36 deletions(-) diff --git a/site/content/3.12/aql/functions/vector.md b/site/content/3.12/aql/functions/vector.md index 9a43bb6297..6d4c241dab 100644 --- a/site/content/3.12/aql/functions/vector.md +++ b/site/content/3.12/aql/functions/vector.md @@ -64,13 +64,14 @@ be found depends on the data as well as the search effort (see the `nProbe` opti `APPROX_NEAR_COSINE(vector1, vector2, options) → similarity` -Retrieve the approximate angular similarity using the cosine metric, accelerated -by a matching vector index. -The closer the cosine similarity value is to 1, the more similar the two vectors +Retrieve the approximate cosine of the angle between two vectors, accelerated +by a matching vector index with the `cosine` metric. + +The closer the similarity value is to 1, the more similar the two vectors are. The closer it is to 0, the more different they are. The value can also be negative up to -1, indicating that the vectors are not similar and point in opposite -directions. You need to sort in descending order so that the most similar +directions. You need to **sort in descending order** so that the most similar documents come first, which is what a vector index using the `cosine` metric can provide. @@ -83,8 +84,8 @@ can provide. closest Voronoi cells to consider for the search results. The larger the number, the slower the search but the better the search results. If not specified, the `defaultNProbe` value of the vector index is used. -- returns **similarity** (number): The approximate angular similarity between - both vectors. +- returns **similarity** (number): The approximate cosine similarity of + both normalized vectors. The value range is `[-1, 1]`. **Examples** @@ -126,15 +127,83 @@ FOR docOuter IN coll RETURN { key: docOuter._key, neighbors } ``` +### APPROX_NEAR_INNER_PRODUCT() + +Introduced in: v3.12.6 + +`APPROX_NEAR_INNER_PRODUCT(vector1, vector2, options) → similarity` + +Retrieve the approximate dot product of two vectors, accelerated by a matching +vector index with the `innerProduct` metric. + +The higher the similarity value is, the more similar the two vectors +are. The closer it is to 0, the more different they are. The value can also +be negative, indicating that the vectors are not similar and point in opposite +directions. You need to **sort in descending order** so that the most similar +documents come first, which is what a vector index using the `innerProduct` +metric can provide. + +- **vector1** (array of numbers): The first vector. Either this parameter or + `vector2` needs to reference a stored attribute holding the vector embedding. +- **vector2** (array of numbers): The second vector. Either this parameter or + `vector1` needs to reference a stored attribute holding the vector embedding. +- **options** (object, _optional_): + - **nProbe** (number, _optional_): How many neighboring centroids respectively + closest Voronoi cells to consider for the search results. The larger the number, + the slower the search but the better the search results. If not specified, the + `defaultNProbe` value of the vector index is used. +- returns **similarity** (number): The approximate dot product + of both vectors without normalization. The value range is unbounded. + +**Examples** + +Return up to `10` similar documents based on their closeness to the vector +`@q` according to the inner product metric: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_INNER_PRODUCT(doc.vector, @q) DESC + LIMIT 10 + RETURN doc +``` + +Return up to `5` similar documents as well as the similarity value, +considering `20` neighboring centroids respectively closest Voronoi cells: + +```aql +FOR doc IN coll + LET similarity = APPROX_NEAR_INNER_PRODUCT(doc.vector, @q, { nProbe: 20 }) + SORT similarity DESC + LIMIT 5 + RETURN MERGE( { similarity }, doc) +``` + +Return the similarity value and the document keys of up to `3` similar documents +for multiple input vectors using a subquery. In this example, the input vectors +are taken from ten random documents of the same collection: + +```aql +FOR docOuter IN coll + LIMIT 10 + LET neighbors = ( + FOR docInner IN coll + LET similarity = APPROX_NEAR_INNER_PRODUCT(docInner.vector, docOuter.vector) + SORT similarity DESC + LIMIT 3 + RETURN { key: docInner._key, similarity } + ) + RETURN { key: docOuter._key, neighbors } +``` + ### APPROX_NEAR_L2() -`APPROX_NEAR_L2(vector1, vector2, options) → similarity` +`APPROX_NEAR_L2(vector1, vector2, options) → distance` Retrieve the approximate distance using the L2 (Euclidean) metric, accelerated -by a matching vector index. +by a matching vector index with the `l2` metric. The closer the distance is to 0, the more similar the two vectors are. The higher -the value, the more different the they are. You need to sort in ascending order +the value, the more different the they are. You need to **sort in ascending order** so that the most similar documents come first, which is what a vector index using the `l2` metric can provide. @@ -147,7 +216,7 @@ the `l2` metric can provide. for the search results. The larger the number, the slower the search but the better the search results. If not specified, the `defaultNProbe` value of the vector index is used. -- returns **similarity** (number): The approximate L2 (Euclidean) distance between +- returns **distance** (number): The approximate L2 (Euclidean) distance between both vectors. **Examples** diff --git a/site/content/3.12/develop/http-api/indexes/vector.md b/site/content/3.12/develop/http-api/indexes/vector.md index a98bd79166..0da7d60206 100644 --- a/site/content/3.12/develop/http-api/indexes/vector.md +++ b/site/content/3.12/develop/http-api/indexes/vector.md @@ -87,9 +87,14 @@ paths: properties: metric: description: | - Whether to use `cosine` or `l2` (Euclidean) distance calculation. - type: string - enum: ["cosine", "l2"] + The measure for calculating the vector similarity: + - `"cosine"`: Angular similarity. Vectors are automatically + normalized before insertion and search. + - `"innerProduct"` (introduced in v3.12.6): + Similarity in terms of angle and magnitude. + Vectors are not normalized, making it faster than `cosine`. + - `"l2":` Euclidean distance. + enum: ["cosine", "innerProduct", "l2"] dimension: description: | The vector dimension. The attribute to index needs to diff --git a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md index 90b82edbf9..3e482df4d3 100644 --- a/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.12/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -14,10 +14,7 @@ data numerically and can be generated with machine learning models. You can then quickly find a given number of semantically similar documents by searching for close neighbors in a high-dimensional vector space. -The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/) -to support L2 and cosine metrics. The index used is IndexIVFFlat, the quantizer -for L2 is IndexFlatL2, and the cosine uses IndexFlatIP, where vectors are -normalized before insertion and search. +The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/). ## How to use vector indexes @@ -75,7 +72,13 @@ centroids and the quality of vector search thus degrades. write operations by not using an exclusive write lock for the duration of the index creation. The default is `false`. - **params**: The parameters as used by the Faiss library. - - **metric** (string): Whether to use `cosine` or `l2` (Euclidean) distance calculation. + - **metric** (string): The measure for calculating the vector similarity: + - `"cosine"`: Angular similarity. Vectors are automatically + normalized before insertion and search. + - `"innerProduct"` (introduced in v3.12.6): + Similarity in terms of angle and magnitude. + Vectors are not normalized, making it faster than `cosine`. + - `"l2":` Euclidean distance. - **dimension** (number): The vector dimension. The attribute to index needs to have this many elements in the array that stores the vector embedding. - **nLists** (number): The number of Voronoi cells to partition the vector space diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 77b8c26bb2..4400d9da87 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -1407,6 +1407,18 @@ utilizing vector indexes in queries. Furthermore, a new error code `ERROR_QUERY_VECTOR_SEARCH_NOT_APPLIED` (1554) has been added. +--- + +Introduced in: v3.12.6 + +Another metric has been added. The `innerProduct` is a vector similarity measure +calculated using the dot product of two vectors without normalizing them. +Therefore, it compares not only the angle but also the magnitudes. + +The accompanying AQL function is the following: + +- `APPROX_NEAR_INNER_PRODUCT()` + ## Server options ### Effective and available startup options diff --git a/site/content/3.13/aql/functions/vector.md b/site/content/3.13/aql/functions/vector.md index 9a43bb6297..6d4c241dab 100644 --- a/site/content/3.13/aql/functions/vector.md +++ b/site/content/3.13/aql/functions/vector.md @@ -64,13 +64,14 @@ be found depends on the data as well as the search effort (see the `nProbe` opti `APPROX_NEAR_COSINE(vector1, vector2, options) → similarity` -Retrieve the approximate angular similarity using the cosine metric, accelerated -by a matching vector index. -The closer the cosine similarity value is to 1, the more similar the two vectors +Retrieve the approximate cosine of the angle between two vectors, accelerated +by a matching vector index with the `cosine` metric. + +The closer the similarity value is to 1, the more similar the two vectors are. The closer it is to 0, the more different they are. The value can also be negative up to -1, indicating that the vectors are not similar and point in opposite -directions. You need to sort in descending order so that the most similar +directions. You need to **sort in descending order** so that the most similar documents come first, which is what a vector index using the `cosine` metric can provide. @@ -83,8 +84,8 @@ can provide. closest Voronoi cells to consider for the search results. The larger the number, the slower the search but the better the search results. If not specified, the `defaultNProbe` value of the vector index is used. -- returns **similarity** (number): The approximate angular similarity between - both vectors. +- returns **similarity** (number): The approximate cosine similarity of + both normalized vectors. The value range is `[-1, 1]`. **Examples** @@ -126,15 +127,83 @@ FOR docOuter IN coll RETURN { key: docOuter._key, neighbors } ``` +### APPROX_NEAR_INNER_PRODUCT() + +Introduced in: v3.12.6 + +`APPROX_NEAR_INNER_PRODUCT(vector1, vector2, options) → similarity` + +Retrieve the approximate dot product of two vectors, accelerated by a matching +vector index with the `innerProduct` metric. + +The higher the similarity value is, the more similar the two vectors +are. The closer it is to 0, the more different they are. The value can also +be negative, indicating that the vectors are not similar and point in opposite +directions. You need to **sort in descending order** so that the most similar +documents come first, which is what a vector index using the `innerProduct` +metric can provide. + +- **vector1** (array of numbers): The first vector. Either this parameter or + `vector2` needs to reference a stored attribute holding the vector embedding. +- **vector2** (array of numbers): The second vector. Either this parameter or + `vector1` needs to reference a stored attribute holding the vector embedding. +- **options** (object, _optional_): + - **nProbe** (number, _optional_): How many neighboring centroids respectively + closest Voronoi cells to consider for the search results. The larger the number, + the slower the search but the better the search results. If not specified, the + `defaultNProbe` value of the vector index is used. +- returns **similarity** (number): The approximate dot product + of both vectors without normalization. The value range is unbounded. + +**Examples** + +Return up to `10` similar documents based on their closeness to the vector +`@q` according to the inner product metric: + +```aql +FOR doc IN coll + SORT APPROX_NEAR_INNER_PRODUCT(doc.vector, @q) DESC + LIMIT 10 + RETURN doc +``` + +Return up to `5` similar documents as well as the similarity value, +considering `20` neighboring centroids respectively closest Voronoi cells: + +```aql +FOR doc IN coll + LET similarity = APPROX_NEAR_INNER_PRODUCT(doc.vector, @q, { nProbe: 20 }) + SORT similarity DESC + LIMIT 5 + RETURN MERGE( { similarity }, doc) +``` + +Return the similarity value and the document keys of up to `3` similar documents +for multiple input vectors using a subquery. In this example, the input vectors +are taken from ten random documents of the same collection: + +```aql +FOR docOuter IN coll + LIMIT 10 + LET neighbors = ( + FOR docInner IN coll + LET similarity = APPROX_NEAR_INNER_PRODUCT(docInner.vector, docOuter.vector) + SORT similarity DESC + LIMIT 3 + RETURN { key: docInner._key, similarity } + ) + RETURN { key: docOuter._key, neighbors } +``` + ### APPROX_NEAR_L2() -`APPROX_NEAR_L2(vector1, vector2, options) → similarity` +`APPROX_NEAR_L2(vector1, vector2, options) → distance` Retrieve the approximate distance using the L2 (Euclidean) metric, accelerated -by a matching vector index. +by a matching vector index with the `l2` metric. The closer the distance is to 0, the more similar the two vectors are. The higher -the value, the more different the they are. You need to sort in ascending order +the value, the more different the they are. You need to **sort in ascending order** so that the most similar documents come first, which is what a vector index using the `l2` metric can provide. @@ -147,7 +216,7 @@ the `l2` metric can provide. for the search results. The larger the number, the slower the search but the better the search results. If not specified, the `defaultNProbe` value of the vector index is used. -- returns **similarity** (number): The approximate L2 (Euclidean) distance between +- returns **distance** (number): The approximate L2 (Euclidean) distance between both vectors. **Examples** diff --git a/site/content/3.13/develop/http-api/indexes/vector.md b/site/content/3.13/develop/http-api/indexes/vector.md index a98bd79166..0da7d60206 100644 --- a/site/content/3.13/develop/http-api/indexes/vector.md +++ b/site/content/3.13/develop/http-api/indexes/vector.md @@ -87,9 +87,14 @@ paths: properties: metric: description: | - Whether to use `cosine` or `l2` (Euclidean) distance calculation. - type: string - enum: ["cosine", "l2"] + The measure for calculating the vector similarity: + - `"cosine"`: Angular similarity. Vectors are automatically + normalized before insertion and search. + - `"innerProduct"` (introduced in v3.12.6): + Similarity in terms of angle and magnitude. + Vectors are not normalized, making it faster than `cosine`. + - `"l2":` Euclidean distance. + enum: ["cosine", "innerProduct", "l2"] dimension: description: | The vector dimension. The attribute to index needs to diff --git a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md index 90b82edbf9..3e482df4d3 100644 --- a/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md +++ b/site/content/3.13/index-and-search/indexing/working-with-indexes/vector-indexes.md @@ -14,10 +14,7 @@ data numerically and can be generated with machine learning models. You can then quickly find a given number of semantically similar documents by searching for close neighbors in a high-dimensional vector space. -The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/) -to support L2 and cosine metrics. The index used is IndexIVFFlat, the quantizer -for L2 is IndexFlatL2, and the cosine uses IndexFlatIP, where vectors are -normalized before insertion and search. +The vector index implementation uses the [Faiss library](https://github.com/facebookresearch/faiss/). ## How to use vector indexes @@ -75,7 +72,13 @@ centroids and the quality of vector search thus degrades. write operations by not using an exclusive write lock for the duration of the index creation. The default is `false`. - **params**: The parameters as used by the Faiss library. - - **metric** (string): Whether to use `cosine` or `l2` (Euclidean) distance calculation. + - **metric** (string): The measure for calculating the vector similarity: + - `"cosine"`: Angular similarity. Vectors are automatically + normalized before insertion and search. + - `"innerProduct"` (introduced in v3.12.6): + Similarity in terms of angle and magnitude. + Vectors are not normalized, making it faster than `cosine`. + - `"l2":` Euclidean distance. - **dimension** (number): The vector dimension. The attribute to index needs to have this many elements in the array that stores the vector embedding. - **nLists** (number): The number of Voronoi cells to partition the vector space diff --git a/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md index 98669fffb7..020f5fcb84 100644 --- a/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md @@ -1407,6 +1407,18 @@ utilizing vector indexes in queries. Furthermore, a new error code `ERROR_QUERY_VECTOR_SEARCH_NOT_APPLIED` (1554) has been added. +--- + +Introduced in: v3.12.6 + +Another metric has been added. The `innerProduct` is a vector similarity measure +calculated using the dot product of two vectors without normalizing them. +Therefore, it compares not only the angle but also the magnitudes. + +The accompanying AQL function is the following: + +- `APPROX_NEAR_INNER_PRODUCT()` + ## Server options ### Effective and available startup options