Vector Store: Support for Cosine similarity and Dot Product when creating a FLOAT_VECTOR

[ckurze]

Problem Statement

Today, when creating a FLOAT_VECTOR, it uses the default EUCLIDEAN_HNSW (L2) similarity.

During creation time, it should be possible to provide Cosine Similarity or Dot Product as similarity functions as available in lucene (https://lucene.apache.org/core/9_7_0/core/org/apache/lucene/document/KnnFloatVectorField.html). Default can remain Euclidian similarity.

Possible Solutions

Exposing options to choose from "cosine" or "dot" similarity function when not using the default "euclidian" function (https://lucene.apache.org/core/9_7_0/core/org/apache/lucene/index/VectorSimilarityFunction.html)

Considered Alternatives

No response

[mfussenegger]

Today, when creating a FLOAT_VECTOR, it uses the default EUCLIDEAN_HNSW (L2) similarity.
During creation time, it should be possible to provide Cosine Similarity or Dot Product as similarity functions as available in lucene

"Feature I want is missing" is not a very good problem statement. Could you elaborate on the concrete use-case a bit more, to get a better idea about why cosine or dot product similarity would be beneficial there?

[ckurze]

While euclidean distance focuses on the absolute magnitude difference between vectors, cosine similarity focuses on the angular difference, and dot product considers both the magnitude and angular difference. The choice of metric depends on the specific use case and the properties of the vector embeddings. The developer should be able to differentiate between these different options to create an optimal implementation for their use case - which might require to use a different similarity function.

Euclidean Distance

• Measures the straight-line distance between two vectors in a multi-dimensional space.
• Calculated as the square root of the sum of the squared differences between the corresponding vector components.
• Sensitive to the magnitude of the vectors, so vectors with large values will have larger Euclidean distances even if they are otherwise similar.
• Useful when the vector embeddings contain information about counts or measures, like in recommendation systems.

Cosine Similarity

• Measures the cosine of the angle between two vectors.
• Calculated as the dot product of the vectors divided by the product of their magnitudes.
• Only considers the direction of the vectors, not their magnitude.
• Commonly used in natural language processing to measure the similarity between documents.
• Inverse relationship with cosine distance - as distance increases, similarity decreases.

Dot Product

• Measures the projection of one vector onto another.
• Calculated by summing the products of the corresponding vector components.
• Proportional to both the vectors' magnitudes and the cosine of the angle between them.
• Can be affected by the length of the vectors, so popular examples may skew the similarity metric.

[mfussenegger]

The cosine variant is likely going to be removed from Lucene: apache/lucene#13281