From ffd718c8b623d27f8edf7ea5745762ace2fd1518 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 01:20:14 +0000 Subject: [PATCH 01/11] Initial plan From 424793f2a46a2f4f70daf6a9494fabb33cfb0792 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 01:31:12 +0000 Subject: [PATCH 02/11] Beef up MEVD docs: expand vector-databases.md, add use-vector-stores how-to, new TOC section Co-authored-by: gewarren <24882762+gewarren@users.noreply.github.com> --- docs/ai/conceptual/vector-databases.md | 158 ++++++++++++++++-- .../csharp/VectorStoresExamples/.gitignore | 2 + .../csharp/VectorStoresExamples/Hotel.cs | 21 +++ .../csharp/VectorStoresExamples/Program.cs | 129 ++++++++++++++ .../VectorStoresExamples.csproj | 15 ++ docs/ai/how-to/use-vector-stores.md | 123 ++++++++++++++ docs/ai/toc.yml | 6 + 7 files changed, 444 insertions(+), 10 deletions(-) create mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore create mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs create mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs create mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/VectorStoresExamples.csproj create mode 100644 docs/ai/how-to/use-vector-stores.md diff --git a/docs/ai/conceptual/vector-databases.md b/docs/ai/conceptual/vector-databases.md index 2373efce66156..56d168df78776 100644 --- a/docs/ai/conceptual/vector-databases.md +++ b/docs/ai/conceptual/vector-databases.md @@ -1,13 +1,14 @@ --- -title: "Using Vector Databases to Extend LLM Capabilities" -description: "Learn how vector databases extend LLM capabilities by storing and processing embeddings in .NET." +title: "Vector databases for .NET AI apps" +description: "Learn how vector databases extend LLM capabilities by storing and processing embeddings in .NET, and how to use Microsoft.Extensions.VectorData to build semantic search features." ms.topic: concept-article -ms.date: 05/29/2025 +ms.date: 02/24/2026 +ai-usage: ai-assisted --- -# Vector databases for .NET + AI +# Vector databases for .NET AI apps -Vector databases are designed to store and manage vector [embeddings](embeddings.md). Embeddings are numeric representations of non-numeric data that preserve semantic meaning. Words, documents, images, audio, and other types of data can all be vectorized. You can use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions. +Vector databases store and manage vector [embeddings](embeddings.md)—numeric representations of data that preserve semantic meaning. Words, documents, images, audio, and other types of data can all be vectorized. You can use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions. For example, you can use a vector database to: @@ -19,28 +20,165 @@ For example, you can use a vector database to: ## Understand vector search -Vector databases provide vector search capabilities to find similar items based on their data characteristics rather than by exact matches on a property field. Vector search works by analyzing the vector representations of your data that you created using an AI embedding model such the [Azure OpenAI embedding models](/azure/ai-services/openai/concepts/models#embeddings-models). The search process measures the distance between the data vectors and your query vector. The data vectors that are closest to your query vector are the ones that are found to be most similar semantically. +Vector databases provide vector search capabilities to find similar items based on their data characteristics rather than by exact matches on a property field. Vector search works by analyzing the vector representations of your data that you created using an AI embedding model such as the [Azure OpenAI embedding models](/azure/ai-services/openai/concepts/models#embeddings-models). The search process measures the distance between the data vectors and your query vector. The data vectors that are closest to your query vector are the ones that are most similar semantically. Some services such as [Azure Cosmos DB for MongoDB vCore](/azure/cosmos-db/mongodb/vcore/vector-search) provide native vector search capabilities for your data. Other databases can be enhanced with vector search by indexing the stored data using a service such as Azure AI Search, which can scan and index your data to provide vector search capabilities. ## Vector search workflows with .NET and OpenAI -Vector databases and their search features are especially useful in [RAG pattern](rag.md) workflows with Azure OpenAI. This pattern allows you to augment or enhance your AI model with additional semantically rich knowledge of your data. A common AI workflow using vector databases might include the following steps: +Vector databases and their search features are especially useful in [RAG pattern](rag.md) workflows with Azure OpenAI. This pattern allows you to augment or enhance your AI model with additional semantically rich knowledge of your data. A common AI workflow using vector databases includes the following steps: 1. Create embeddings for your data using an OpenAI embedding model. 1. Store and index the embeddings in a vector database or search service. 1. Convert user prompts from your application to embeddings. -1. Run a vector search across your data, comparing the user prompt embedding to the embeddings your database. -1. Use a language model such as GPT-35 or GPT-4 to assemble a user friendly completion from the vector search results. +1. Run a vector search across your data, comparing the user prompt embedding to the embeddings in your database. +1. Use a language model such as GPT-4 to assemble a user-friendly completion from the vector search results. Visit the [Implement Azure OpenAI with RAG using vector search in a .NET app](../tutorials/tutorial-ai-vector-search.md) tutorial for a hands-on example of this flow. Other benefits of the RAG pattern include: - Generate contextually relevant and accurate responses to user prompts from AI models. -- Overcome LLM tokens limits - the heavy lifting is done through the database vector search. +- Overcome LLM token limits—the heavy lifting is done through the database vector search. - Reduce the costs from frequent fine-tuning on updated data. +## The Microsoft.Extensions.VectorData library + +The [📦 Microsoft.Extensions.VectorData.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions) package provides a unified layer of abstractions for interacting with vector stores in .NET. These abstractions let you write code against a single API and swap out the underlying vector store with minimal changes to your application. + +The library provides the following key capabilities: + +- **Unified data model**: Define your data model once using .NET attributes and use it across any supported vector store. +- **CRUD operations**: Create, read, update, and delete records in a vector store. +- **Vector and text search**: Query records by semantic similarity using vector search, or by keyword using text search. +- **Collection management**: Create, list, and delete collections (tables or indices) in a vector store. + +### Define a data model + +To store records in a vector store, define a .NET class and annotate its properties with the following attributes from the namespace: + +- : Marks the property that uniquely identifies each record (the primary key). +- : Marks properties that contain regular data (strings, numbers, and so on) to store and optionally filter on. +- : Marks properties that store embedding vectors. You specify the number of dimensions and the distance function to use for similarity comparisons. + +The following example defines a data model for a hotel: + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public int HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string? HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 1536, DistanceFunction = DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string[]? Tags { get; set; } +} +``` + +The `[VectorStoreVector]` attribute requires you to specify the number of dimensions, which must match the output size of the embedding model you use. For example, the `text-embedding-3-small` model produces 1536-dimensional vectors. + +### Key abstractions + +The `Microsoft.Extensions.VectorData.Abstractions` library exposes two main abstract classes: + +- : The top-level class for a vector database. Use it to retrieve and manage collections. +- : Represents a named collection of records within a vector store. Use it to perform CRUD and search operations. + +The following example shows how to get a collection from a vector store and upsert (insert or update) records: + +```csharp +// Get or create a collection named "hotels". +VectorStoreCollection collection = + vectorStore.GetCollection("hotels"); + +// Ensure the collection exists in the database. +await collection.EnsureCollectionExistsAsync(); + +// Upsert a record. +await collection.UpsertAsync(new Hotel +{ + HotelId = 1, + HotelName = "Seaside Retreat", + Description = "A peaceful hotel on the coast with stunning ocean views.", + DescriptionEmbedding = await embeddingGenerator.GenerateVectorAsync( + "A peaceful hotel on the coast with stunning ocean views."), + Tags = ["beach", "ocean", "relaxation"] +}); +``` + +### Perform vector search + +Use the `SearchAsync` method to search for semantically similar records. Pass in an embedding vector for your query and specify the number of results to return: + +```csharp +// Generate an embedding for the search query. +ReadOnlyMemory queryEmbedding = + await embeddingGenerator.GenerateVectorAsync("beachfront hotel"); + +// Search for the top 3 most similar hotels. +IAsyncEnumerable> results = + collection.SearchAsync(queryEmbedding, top: 3); + +await foreach (VectorSearchResult result in results) +{ + Console.WriteLine($"Hotel: {result.Record.HotelName}"); + Console.WriteLine($"Score: {result.Score}"); +} +``` + +### Filter search results + +Use the class to filter vector search results by field values. Only properties marked with `IsIndexed = true` in `[VectorStoreData]` can be used in filters: + +```csharp +var searchOptions = new VectorSearchOptions +{ + Filter = hotel => hotel.HotelName == "Seaside Retreat" +}; + +IAsyncEnumerable> results = + collection.SearchAsync(queryEmbedding, top: 3, searchOptions); +``` + +## Available vector store connectors + +The `Microsoft.Extensions.VectorData.Abstractions` package defines the abstractions, and separate connector packages implement them for specific vector databases. Choose the connector that matches your vector database: + +| Connector | NuGet package | +|---|---| +| In-memory (for testing/development) | [Microsoft.SemanticKernel.Connectors.InMemory](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.InMemory) | +| Azure AI Search | [Microsoft.SemanticKernel.Connectors.AzureAISearch](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.AzureAISearch) | +| Azure Cosmos DB (NoSQL) | [Microsoft.SemanticKernel.Connectors.CosmosNoSQL](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosNoSQL) | +| Azure Cosmos DB (MongoDB) | [Microsoft.SemanticKernel.Connectors.CosmosMongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosMongoDB) | +| Azure SQL / SQL Server | [Microsoft.SemanticKernel.Connectors.SqlServer](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqlServer) | +| MongoDB | [Microsoft.SemanticKernel.Connectors.MongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.MongoDB) | +| PostgreSQL (pgvector) | [Microsoft.SemanticKernel.Connectors.PgVector](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.PgVector) | +| Qdrant | [Microsoft.SemanticKernel.Connectors.Qdrant](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Qdrant) | +| Redis | [Microsoft.SemanticKernel.Connectors.Redis](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Redis) | +| SQLite | [Microsoft.SemanticKernel.Connectors.SqliteVec](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqliteVec) | +| Pinecone | [Microsoft.SemanticKernel.Connectors.Pinecone](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Pinecone) | +| Weaviate | [Microsoft.SemanticKernel.Connectors.Weaviate](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Weaviate) | +| Elasticsearch | [Elastic.SemanticKernel.Connectors.Elasticsearch](https://www.nuget.org/packages/Elastic.SemanticKernel.Connectors.Elasticsearch) | + +All connectors implement the same `VectorStore` and `VectorStoreCollection` abstract classes, so you can switch between them without changing your application logic. + +> [!TIP] +> Use the in-memory connector (`Microsoft.SemanticKernel.Connectors.InMemory`) during development and testing. It doesn't require any external service or configuration, and you can swap it out for a production connector later. + ## Related content +- [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md) - [Implement Azure OpenAI with RAG using vector search in a .NET app](../tutorials/tutorial-ai-vector-search.md) +- [Use vector stores in .NET AI apps](../how-to/use-vector-stores.md) +- [Data ingestion](data-ingestion.md) +- [Embeddings in .NET](embeddings.md) diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore new file mode 100644 index 0000000000000..cd42ee34e873b --- /dev/null +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore @@ -0,0 +1,2 @@ +bin/ +obj/ diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs new file mode 100644 index 0000000000000..05629de6cacc9 --- /dev/null +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs @@ -0,0 +1,21 @@ +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public int HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string? HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 1536, DistanceFunction = DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string[]? Tags { get; set; } +} +// diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs new file mode 100644 index 0000000000000..a7bfd460ecf98 --- /dev/null +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs @@ -0,0 +1,129 @@ +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; + +// +// Create an in-memory vector store (no external service required). +// For production, replace this with a connector for your preferred database, +// such as Azure AI Search or Qdrant. +var vectorStore = new InMemoryVectorStore(); +// + +// +// Get a reference to a collection named "hotels". +// The collection stores records of type Hotel with an integer key. +VectorStoreCollection collection = + vectorStore.GetCollection("hotels"); + +// Ensure the collection exists in the database. +await collection.EnsureCollectionExistsAsync(); +// + +// +// Create some sample records. +var hotels = new List +{ + new() + { + HotelId = 1, + HotelName = "Seaside Retreat", + Description = "A peaceful hotel on the coast with stunning ocean views.", + DescriptionEmbedding = CreateFakeEmbedding(1), + Tags = ["beach", "ocean", "relaxation"] + }, + new() + { + HotelId = 2, + HotelName = "Mountain Lodge", + Description = "A cozy lodge in the mountains with hiking trails nearby.", + DescriptionEmbedding = CreateFakeEmbedding(2), + Tags = ["mountain", "hiking", "nature"] + }, + new() + { + HotelId = 3, + HotelName = "City Centre Hotel", + Description = "A modern hotel in the heart of the city, close to attractions.", + DescriptionEmbedding = CreateFakeEmbedding(3), + Tags = ["city", "business", "urban"] + } +}; + +// Upsert records into the collection. +// Upsert inserts a record if the key doesn't exist, or updates it if it does. +await collection.UpsertAsync(hotels[0]); +await collection.UpsertAsync(hotels[1]); +await collection.UpsertAsync(hotels[2]); +// + +// +// Get a specific record by its key. +Hotel? hotel = await collection.GetAsync(1); +if (hotel is not null) +{ + Console.WriteLine($"Hotel: {hotel.HotelName}"); + Console.WriteLine($"Description: {hotel.Description}"); +} +// + +// +// Get multiple records by their keys. +IAsyncEnumerable hotelBatch = collection.GetAsync([1, 2, 3]); +await foreach (Hotel h in hotelBatch) +{ + Console.WriteLine($"Batch hotel: {h.HotelName}"); +} +// + +// +// Perform a vector search using a query embedding. +// In a real app, generate this embedding from the user's search query using +// an IEmbeddingGenerator. Here we use a placeholder. +ReadOnlyMemory queryEmbedding = CreateFakeEmbedding(1); + +IAsyncEnumerable> searchResults = + collection.SearchAsync(queryEmbedding, top: 2); + +await foreach (VectorSearchResult result in searchResults) +{ + Console.WriteLine($"Found: {result.Record.HotelName} (score: {result.Score:F4})"); +} +// + +// +// Filter search results to only include a specific hotel. +var searchOptions = new VectorSearchOptions +{ + Filter = h => h.HotelName == "Seaside Retreat" +}; + +IAsyncEnumerable> filteredResults = + collection.SearchAsync(queryEmbedding, top: 2, searchOptions); + +await foreach (VectorSearchResult result in filteredResults) +{ + Console.WriteLine($"Filtered: {result.Record.HotelName} (score: {result.Score:F4})"); +} +// + +// +// Delete a record by its key. +await collection.DeleteAsync(3); +// + +// +// Delete the entire collection from the vector store. +await collection.EnsureCollectionDeletedAsync(); +// + +// Helper to create a predictable fake embedding for demonstration. +// In a real app, use IEmbeddingGenerator to generate real embeddings. +static ReadOnlyMemory CreateFakeEmbedding(int seed) +{ + var random = new Random(seed); + float[] vector = new float[1536]; + for (int i = 0; i < vector.Length; i++) + { + vector[i] = (float)random.NextDouble(); + } + return new ReadOnlyMemory(vector); +} diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/VectorStoresExamples.csproj b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/VectorStoresExamples.csproj new file mode 100644 index 0000000000000..418c89330cd13 --- /dev/null +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/VectorStoresExamples.csproj @@ -0,0 +1,15 @@ + + + + Exe + net10.0 + enable + enable + + + + + + + + diff --git a/docs/ai/how-to/use-vector-stores.md b/docs/ai/how-to/use-vector-stores.md new file mode 100644 index 0000000000000..1e5ff0011dbde --- /dev/null +++ b/docs/ai/how-to/use-vector-stores.md @@ -0,0 +1,123 @@ +--- +title: Use vector stores in .NET AI apps +description: Learn how to use Microsoft.Extensions.VectorData to store, search, and manage embeddings in vector databases for .NET AI applications. +ms.topic: how-to +ms.date: 02/24/2026 +ai-usage: ai-assisted +--- + +# Use vector stores in .NET AI apps + +The [📦 Microsoft.Extensions.VectorData.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions) package provides a unified API for working with vector stores in .NET. You can use the same code to store and search embeddings across different vector database providers. + +This article shows you how to use the key features of the library. + +## Prerequisites + +- [.NET 9 SDK](https://dotnet.microsoft.com/download/dotnet/9.0) or later +- An understanding of [embeddings](../conceptual/embeddings.md) and [vector databases](../conceptual/vector-databases.md) + +## Install the packages + +Install the `Microsoft.Extensions.VectorData.Abstractions` package and a connector for your vector database. The following example uses the in-memory connector for development and testing: + +```dotnetcli +dotnet add package Microsoft.Extensions.VectorData.Abstractions +dotnet add package Microsoft.SemanticKernel.Connectors.InMemory --prerelease +``` + +For production scenarios, replace `Microsoft.SemanticKernel.Connectors.InMemory` with the connector for your database. For a full list of available connectors, see [Available vector store connectors](../conceptual/vector-databases.md#available-vector-store-connectors). + +## Define a data model + +Define a .NET class to represent the records you want to store in the vector store. Use the following attributes from the namespace to describe the properties: + +- : The primary key of each record. +- : A data property that gets stored. Set `IsIndexed = true` to enable filtering on the property. +- : An embedding vector property. Set the `Dimensions` parameter to match your embedding model's output size. + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs" id="DataModel"::: + +The `Dimensions` parameter in `[VectorStoreVector]` must match the output size of the embedding model you use. Common values include 1536 for `text-embedding-3-small` and 3072 for `text-embedding-3-large`. + +## Create a vector store + +Create an instance of the `VectorStore` implementation for your chosen database. The following example creates an in-memory vector store: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="CreateVectorStore"::: + +## Get a collection + +Call `GetCollection` on the vector store to get a typed reference to a collection. Then call `EnsureCollectionExistsAsync` to create the collection if it doesn't already exist: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetCollection"::: + +The collection name maps to the underlying storage concept for your database (for example, a table in SQL Server, an index in Azure AI Search, or a container in Cosmos DB). + +## Upsert records + +Use `UpsertAsync` to insert or update records in the collection. If a record with the same key already exists, it gets updated: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="UpsertRecords"::: + +> [!IMPORTANT] +> In a real app, you should generate the embedding vectors using an `IEmbeddingGenerator` before storing the records. In the following example, the placeholder `CreateFakeEmbedding` method generates dummy vectors. For a working example with real embeddings, see [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md). + +## Get records + +Use `GetAsync` to retrieve a single record by its key. To retrieve multiple records, pass an `IEnumerable` to `GetAsync`: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetRecord"::: + +To retrieve multiple records at once: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetBatch"::: + +## Perform vector search + +Use `SearchAsync` to find records that are semantically similar to a query. Pass the embedding vector for your query and the number of results to return: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchBasic"::: + +Each `VectorSearchResult` contains the matching record and a similarity score. Higher scores indicate a closer semantic match. + +## Filter search results + +Use to filter search results before the vector comparison. You can filter on any property marked with `IsIndexed = true`: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchWithFilter"::: + +## Delete records + +To delete a single record by key, use `DeleteAsync`: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="DeleteRecord"::: + +## Delete a collection + +To remove an entire collection from the vector store, use `EnsureCollectionDeletedAsync`: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="DeleteCollection"::: + +## Switch vector store connectors + +Because all connectors implement the same `VectorStore` abstract class, you can switch between them by changing the concrete type at startup. Your collection and search code remains the same. + +For example, to switch from in-memory to Azure AI Search: + +```csharp +// Development: in-memory +var vectorStore = new InMemoryVectorStore(); + +// Production: Azure AI Search (swap in at startup) +// var vectorStore = new AzureAISearchVectorStore(new SearchIndexClient(...)); +``` + +This approach lets you develop and test locally without any external services, then deploy to a production database with minimal changes. + +## Related content + +- [Vector databases for .NET AI apps](../conceptual/vector-databases.md) +- [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md) +- [Data ingestion](../conceptual/data-ingestion.md) +- [Embeddings in .NET](../conceptual/embeddings.md) diff --git a/docs/ai/toc.yml b/docs/ai/toc.yml index 13e86bcf0a29e..466dc8c6c5500 100644 --- a/docs/ai/toc.yml +++ b/docs/ai/toc.yml @@ -94,6 +94,12 @@ items: items: - name: Use Microsoft.ML.Tokenizers href: how-to/use-tokenizers.md +- name: Vector stores + items: + - name: Vector databases overview + href: conceptual/vector-databases.md + - name: Use vector stores + href: how-to/use-vector-stores.md - name: Security and content safety items: - name: Authentication for Azure-hosted apps and services From fe8c80cc866854ab16e0bf5f2fba39c5a9081882 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 01:58:51 +0000 Subject: [PATCH 03/11] Port SK vector store docs: attribute params tables, auto-embedding, hybrid search, record definitions, more connectors Co-authored-by: gewarren <24882762+gewarren@users.noreply.github.com> --- docs/ai/conceptual/vector-databases.md | 62 +++++++- .../VectorStoresExamples/AutoEmbedding.cs | 56 +++++++ .../csharp/VectorStoresExamples/Program.cs | 38 +++-- .../VectorStoresExamples/RecordDefinition.cs | 35 +++++ docs/ai/how-to/use-vector-stores.md | 142 +++++++++++++++++- 5 files changed, 314 insertions(+), 19 deletions(-) create mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs create mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs diff --git a/docs/ai/conceptual/vector-databases.md b/docs/ai/conceptual/vector-databases.md index 56d168df78776..628ac715aa1d7 100644 --- a/docs/ai/conceptual/vector-databases.md +++ b/docs/ai/conceptual/vector-databases.md @@ -77,7 +77,7 @@ public class Hotel [VectorStoreData(IsFullTextIndexed = true)] public string? Description { get; set; } - [VectorStoreVector(Dimensions: 1536, DistanceFunction = DistanceFunction.CosineSimilarity)] + [VectorStoreVector(Dimensions: 1536, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] public ReadOnlyMemory? DescriptionEmbedding { get; set; } [VectorStoreData(IsIndexed = true)] @@ -85,14 +85,36 @@ public class Hotel } ``` -The `[VectorStoreVector]` attribute requires you to specify the number of dimensions, which must match the output size of the embedding model you use. For example, the `text-embedding-3-small` model produces 1536-dimensional vectors. +The `Dimensions` parameter must match the output size of the embedding model you use. For example, `text-embedding-3-small` produces 1536-dimensional vectors, while `text-embedding-3-large` produces 3072-dimensional vectors. + +### Automatic embedding generation + +Instead of generating embeddings yourself before each upsert, you can declare your vector property as a `string` type and configure an `IEmbeddingGenerator` on the vector store. The store then generates the embedding automatically when you upsert a record, using the string value you provide: + +```csharp +public class FinanceInfo +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public string Text { get; set; } = ""; + + // Declare as string to enable automatic embedding generation on upsert. + [VectorStoreVector(1536)] + public string EmbeddingSource { get; set; } = ""; +} +``` + +You can configure the `IEmbeddingGenerator` at the vector store level, collection level, or on individual vector properties. With auto-embedding, you can also pass a `string` directly to `SearchAsync` instead of a precomputed vector—the store generates the search embedding for you. For more information, see [Use built-in embedding generation](../how-to/use-vector-stores.md#use-built-in-embedding-generation). ### Key abstractions -The `Microsoft.Extensions.VectorData.Abstractions` library exposes two main abstract classes: +The `Microsoft.Extensions.VectorData.Abstractions` library exposes the following main abstract classes: - : The top-level class for a vector database. Use it to retrieve and manage collections. -- : Represents a named collection of records within a vector store. Use it to perform CRUD and search operations. +- : Represents a named collection of records within a vector store. Use it to perform CRUD and search operations. Also inherits from `IVectorSearchable`. +- `IKeywordHybridSearchable`: Implemented by collections that support hybrid search combining vector similarity with keyword matching. The following example shows how to get a collection from a vector store and upsert (insert or update) records: @@ -150,6 +172,29 @@ IAsyncEnumerable> results = collection.SearchAsync(queryEmbedding, top: 3, searchOptions); ``` +Filters are expressed as LINQ expressions and compiled into the query syntax of the underlying database. The supported operations vary by connector. + +### Hybrid search + +Some connectors support *hybrid search*, which combines vector similarity with full-text keyword matching. To use hybrid search, check whether your collection implements `IKeywordHybridSearchable` and use the `HybridSearchAsync` method: + +```csharp +if (collection is IKeywordHybridSearchable hybridSearch) +{ + var results = hybridSearch.HybridSearchAsync( + queryEmbedding, + keywords: ["ocean", "beach"], + top: 3); + + await foreach (var result in results) + { + Console.WriteLine($"Hotel: {result.Record.HotelName}, Score: {result.Score}"); + } +} +``` + +For hybrid search to work, the data model must have at least one vector property and one string property with `IsFullTextIndexed = true`. + ## Available vector store connectors The `Microsoft.Extensions.VectorData.Abstractions` package defines the abstractions, and separate connector packages implement them for specific vector databases. Choose the connector that matches your vector database: @@ -161,20 +206,25 @@ The `Microsoft.Extensions.VectorData.Abstractions` package defines the abstracti | Azure Cosmos DB (NoSQL) | [Microsoft.SemanticKernel.Connectors.CosmosNoSQL](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosNoSQL) | | Azure Cosmos DB (MongoDB) | [Microsoft.SemanticKernel.Connectors.CosmosMongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosMongoDB) | | Azure SQL / SQL Server | [Microsoft.SemanticKernel.Connectors.SqlServer](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqlServer) | +| Couchbase | [CouchbaseVectorStore.SemanticKernel](https://www.nuget.org/packages/CouchbaseVectorStore.SemanticKernel) | +| Elasticsearch | [Elastic.SemanticKernel.Connectors.Elasticsearch](https://www.nuget.org/packages/Elastic.SemanticKernel.Connectors.Elasticsearch) | | MongoDB | [Microsoft.SemanticKernel.Connectors.MongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.MongoDB) | +| Oracle | [OracleVectorStore.SemanticKernel](https://www.nuget.org/packages/OracleVectorStore.SemanticKernel) | +| Pinecone | [Microsoft.SemanticKernel.Connectors.Pinecone](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Pinecone) | | PostgreSQL (pgvector) | [Microsoft.SemanticKernel.Connectors.PgVector](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.PgVector) | | Qdrant | [Microsoft.SemanticKernel.Connectors.Qdrant](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Qdrant) | | Redis | [Microsoft.SemanticKernel.Connectors.Redis](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Redis) | | SQLite | [Microsoft.SemanticKernel.Connectors.SqliteVec](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqliteVec) | -| Pinecone | [Microsoft.SemanticKernel.Connectors.Pinecone](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Pinecone) | | Weaviate | [Microsoft.SemanticKernel.Connectors.Weaviate](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Weaviate) | -| Elasticsearch | [Elastic.SemanticKernel.Connectors.Elasticsearch](https://www.nuget.org/packages/Elastic.SemanticKernel.Connectors.Elasticsearch) | All connectors implement the same `VectorStore` and `VectorStoreCollection` abstract classes, so you can switch between them without changing your application logic. > [!TIP] > Use the in-memory connector (`Microsoft.SemanticKernel.Connectors.InMemory`) during development and testing. It doesn't require any external service or configuration, and you can swap it out for a production connector later. +> [!IMPORTANT] +> Not all connectors are maintained by the Microsoft Semantic Kernel project. When evaluating a connector, review its quality, licensing, support, and compatibility to ensure it meets your requirements. + ## Related content - [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md) diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs new file mode 100644 index 0000000000000..7a9813b7766b0 --- /dev/null +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs @@ -0,0 +1,56 @@ +using Microsoft.Extensions.AI; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; + +// +// When the vector property type is string (not ReadOnlyMemory), +// the vector store automatically generates embeddings using the configured IEmbeddingGenerator. +public class FinanceInfo +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public string Text { get; set; } = ""; + + // The string value placed here before upsert is automatically converted to a vector. + [VectorStoreVector(1536)] + public string EmbeddingSource { get; set; } = ""; +} +// + +public static class AutoEmbeddingExample +{ + public static async Task RunAsync(IEmbeddingGenerator> embeddingGenerator) + { + // + // Configure the embedding generator at the vector store level. + // All collections in this store will use it unless overridden. + var vectorStore = new InMemoryVectorStore(new InMemoryVectorStoreOptions + { + EmbeddingGenerator = embeddingGenerator + }); + + var collection = vectorStore.GetCollection("finance"); + await collection.EnsureCollectionExistsAsync(); + + // Embeddings are generated automatically on upsert. + var records = new[] + { + new FinanceInfo { Key = 1, Text = "2024 Budget", EmbeddingSource = "The budget for 2024 is $100,000" }, + new FinanceInfo { Key = 2, Text = "2023 Budget", EmbeddingSource = "The budget for 2023 is $80,000" } + }; + + await collection.UpsertAsync(records[0]); + await collection.UpsertAsync(records[1]); + + // Embeddings for search are also generated automatically. + var results = collection.SearchAsync("What is my 2024 budget?", top: 1); + + await foreach (var result in results) + { + Console.WriteLine($"Found: Key={result.Record.Key}, Text={result.Record.Text}"); + } + // + } +} diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs index a7bfd460ecf98..6577317079d05 100644 --- a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs @@ -1,16 +1,15 @@ +using Microsoft.Extensions.AI; using Microsoft.Extensions.VectorData; using Microsoft.SemanticKernel.Connectors.InMemory; // // Create an in-memory vector store (no external service required). -// For production, replace this with a connector for your preferred database, -// such as Azure AI Search or Qdrant. +// For production, replace this with a connector for your preferred database. var vectorStore = new InMemoryVectorStore(); // // // Get a reference to a collection named "hotels". -// The collection stores records of type Hotel with an integer key. VectorStoreCollection collection = vectorStore.GetCollection("hotels"); @@ -19,7 +18,10 @@ // // -// Create some sample records. +// Upsert records into the collection. +// In a real app, generate embeddings using an IEmbeddingGenerator. +// The CreateFakeEmbedding helper at the bottom of this file generates +// placeholder vectors for demonstration purposes only. var hotels = new List { new() @@ -48,8 +50,6 @@ } }; -// Upsert records into the collection. -// Upsert inserts a record if the key doesn't exist, or updates it if it does. await collection.UpsertAsync(hotels[0]); await collection.UpsertAsync(hotels[1]); await collection.UpsertAsync(hotels[2]); @@ -74,12 +74,10 @@ } // -// -// Perform a vector search using a query embedding. -// In a real app, generate this embedding from the user's search query using -// an IEmbeddingGenerator. Here we use a placeholder. ReadOnlyMemory queryEmbedding = CreateFakeEmbedding(1); +// +// Search for the top 2 hotels most similar to the query embedding. IAsyncEnumerable> searchResults = collection.SearchAsync(queryEmbedding, top: 2); @@ -90,7 +88,8 @@ // // -// Filter search results to only include a specific hotel. +// Filter results before the vector comparison. +// Only properties marked with IsIndexed = true can be used in filters. var searchOptions = new VectorSearchOptions { Filter = h => h.HotelName == "Seaside Retreat" @@ -105,6 +104,23 @@ } // +// +// Use VectorSearchOptions to control paging and vector inclusion. +var pagedOptions = new VectorSearchOptions +{ + Skip = 1, // Skip the first result (useful for paging). + IncludeVectors = false // Don't include vector data in results (default). +}; + +IAsyncEnumerable> pagedResults = + collection.SearchAsync(queryEmbedding, top: 2, pagedOptions); + +await foreach (VectorSearchResult result in pagedResults) +{ + Console.WriteLine($"Paged: {result.Record.HotelName}"); +} +// + // // Delete a record by its key. await collection.DeleteAsync(3); diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs new file mode 100644 index 0000000000000..64da33ae4b470 --- /dev/null +++ b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs @@ -0,0 +1,35 @@ +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; + +public static class RecordDefinitionExample +{ + public static async Task RunAsync() + { + // + // An alternative to using attributes is to define the schema programmatically. + // This is useful when you want to use the same data model with different configurations, + // or when you can't modify the data model class to add attributes. + var hotelDefinition = new VectorStoreCollectionDefinition + { + Properties = new List + { + new VectorStoreKeyProperty("HotelId", typeof(int)), + new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, + new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(ReadOnlyMemory?), dimensions: 1536) + { + DistanceFunction = DistanceFunction.CosineSimilarity, + IndexKind = IndexKind.Flat + } + } + }; + + // Pass the definition to GetCollection to override the attribute-based schema. + var vectorStore = new InMemoryVectorStore(); + VectorStoreCollection collection = + vectorStore.GetCollection("hotels", hotelDefinition); + + await collection.EnsureCollectionExistsAsync(); + // + } +} diff --git a/docs/ai/how-to/use-vector-stores.md b/docs/ai/how-to/use-vector-stores.md index 1e5ff0011dbde..de96757d2c859 100644 --- a/docs/ai/how-to/use-vector-stores.md +++ b/docs/ai/how-to/use-vector-stores.md @@ -33,13 +33,46 @@ For production scenarios, replace `Microsoft.SemanticKernel.Connectors.InMemory` Define a .NET class to represent the records you want to store in the vector store. Use the following attributes from the namespace to describe the properties: - : The primary key of each record. -- : A data property that gets stored. Set `IsIndexed = true` to enable filtering on the property. +- : A data property that gets stored. Set `IsIndexed = true` to enable filtering on the property, or `IsFullTextIndexed = true` to enable full-text search on it. - : An embedding vector property. Set the `Dimensions` parameter to match your embedding model's output size. :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs" id="DataModel"::: The `Dimensions` parameter in `[VectorStoreVector]` must match the output size of the embedding model you use. Common values include 1536 for `text-embedding-3-small` and 3072 for `text-embedding-3-large`. +### Attribute parameters + +The following tables describe all available parameters for each attribute. + +#### VectorStoreKey parameters + +| Parameter | Required | Description | +|---|---|---| +| `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | + +#### VectorStoreData parameters + +| Parameter | Required | Description | +|---|---|---| +| `IsIndexed` | No | Whether to index this property for filtering. Default is `false`. | +| `IsFullTextIndexed` | No | Whether to index this property for full-text search. Default is `false`. | +| `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | + +#### VectorStoreVector parameters + +| Parameter | Required | Description | +|---|---|---| +| `Dimensions` | Yes (for collection creation) | The number of dimensions in the vector. Must match your embedding model. | +| `IndexKind` | No | The type of index to use. Defaults vary by connector. Common values: `Hnsw`, `Flat`. | +| `DistanceFunction` | No | The distance function for similarity comparisons. Defaults vary by connector. Common values: `CosineSimilarity`, `DotProductSimilarity`, `EuclideanDistance`. | +| `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | + +## Define a schema programmatically + +As an alternative to using attributes, you can define your schema programmatically using a . This approach is useful when you want to use the same data model with different configurations, or when you can't add attributes to the data model class. + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs" id="RecordDefinition"::: + ## Create a vector store Create an instance of the `VectorStore` implementation for your chosen database. The following example creates an in-memory vector store: @@ -61,7 +94,7 @@ Use `UpsertAsync` to insert or update records in the collection. If a record wit :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="UpsertRecords"::: > [!IMPORTANT] -> In a real app, you should generate the embedding vectors using an `IEmbeddingGenerator` before storing the records. In the following example, the placeholder `CreateFakeEmbedding` method generates dummy vectors. For a working example with real embeddings, see [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md). +> In a real app, generate the embedding vectors using an `IEmbeddingGenerator` before storing the records. For a working example with real embeddings, see [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md). ## Get records @@ -87,6 +120,111 @@ Use to filter searc :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchWithFilter"::: +Filters are expressed as LINQ expressions. The supported operations vary by connector, but all connectors support common comparisons like equality, inequality, and logical `&&` and `||`. + +## Control search behavior with VectorSearchOptions + +Use `VectorSearchOptions` to control various aspects of vector search behavior: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchOptions"::: + +The following table describes the available options: + +| Option | Description | +|---|---| +| `Filter` | A LINQ expression to filter records before vector comparison. | +| `VectorProperty` | The vector property to search on. Required when the data model has multiple vector properties. | +| `Skip` | Number of results to skip before returning. Useful for paging. Default is `0`. | +| `IncludeVectors` | Whether to include vector data in the returned records. Omitting vectors reduces data transfer. Default is `false`. | + +### Target a specific vector property + +When your data model has multiple vector properties, use `VectorProperty` to specify which one to search: + +```csharp +// A data model with two vector properties. +public class Hotel +{ + [VectorStoreKey] + public int HotelId { get; set; } + + [VectorStoreData] + public string? HotelName { get; set; } + + // Two separate embeddings for different aspects of the hotel. + [VectorStoreVector(1536)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory? AmenitiesEmbedding { get; set; } +} + +// Target the amenities embedding specifically. +var options = new VectorSearchOptions +{ + VectorProperty = r => r.AmenitiesEmbedding +}; + +IAsyncEnumerable> results = + collection.SearchAsync(queryEmbedding, top: 3, options); +``` + +## Use built-in embedding generation + +Instead of generating embeddings manually before each upsert, you can configure an `IEmbeddingGenerator` on the vector store or collection. When you do, declare your vector property as a `string` type (the source text) and the store generates the embedding automatically. + +```csharp +public class FinanceInfo +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public string Text { get; set; } = ""; + + // Use a string type to trigger automatic embedding generation on upsert. + [VectorStoreVector(1536)] + public string EmbeddingSource { get; set; } = ""; +} +``` + +Then configure the embedding generator when creating the vector store: + +:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs" id="AutoEmbeddingVectorStore"::: + +With this approach, `SearchAsync` also accepts a `string` query directly—you don't need to generate a query embedding manually. + +> [!IMPORTANT] +> Automatic embedding generation doesn't retrieve or store the original source text. If you need to access the original text later, store it in a separate `[VectorStoreData]` property on your model. + +## Hybrid search + +Some vector stores support *hybrid search*, which combines vector similarity with keyword matching. This approach can improve result relevance compared to vector-only search. + +To use hybrid search, check whether your collection implements `IKeywordHybridSearchable`. Only connectors for databases that support this feature implement this interface. + +```csharp +// Check whether the collection supports hybrid search. +if (collection is IKeywordHybridSearchable hybridCollection) +{ + ReadOnlyMemory queryEmbedding = + await embeddingGenerator.GenerateVectorAsync("peaceful beachfront hotel"); + + // Provide both a vector and keywords for hybrid search. + var results = hybridCollection.HybridSearchAsync( + queryEmbedding, + keywords: ["ocean", "beach"], + top: 3); + + await foreach (var result in results) + { + Console.WriteLine($"Hotel: {result.Record.HotelName}, Score: {result.Score}"); + } +} +``` + +For hybrid search to work, the data model must have at least one vector property and one text property with `IsFullTextIndexed = true`. For a list of connectors that support hybrid search, see the documentation for each connector. + ## Delete records To delete a single record by key, use `DeleteAsync`: From 1ddb6a138fc698c9950b74cd7af0c333c67c68f6 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 26 Feb 2026 00:03:21 +0000 Subject: [PATCH 04/11] Address review feedback: xref links, ai-usage frontmatter, remove .gitignore, fix TOC duplicate, update intro sentence Co-authored-by: gewarren <24882762+gewarren@users.noreply.github.com> --- docs/ai/conceptual/vector-databases.md | 4 ++-- .../csharp/VectorStoresExamples/.gitignore | 2 -- docs/ai/how-to/use-vector-stores.md | 22 +++++++++---------- docs/ai/toc.yml | 2 -- 4 files changed, 13 insertions(+), 17 deletions(-) delete mode 100644 docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore diff --git a/docs/ai/conceptual/vector-databases.md b/docs/ai/conceptual/vector-databases.md index 628ac715aa1d7..d8b5728f2ebb4 100644 --- a/docs/ai/conceptual/vector-databases.md +++ b/docs/ai/conceptual/vector-databases.md @@ -8,7 +8,7 @@ ai-usage: ai-assisted # Vector databases for .NET AI apps -Vector databases store and manage vector [embeddings](embeddings.md)—numeric representations of data that preserve semantic meaning. Words, documents, images, audio, and other types of data can all be vectorized. You can use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions. +Vector databases store and manage vector [*embeddings*](embeddings.md). Embeddings are numeric representations of data that preserve semantic meaning. Words, documents, images, audio, and other types of data can all be vectorized. You can use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions. For example, you can use a vector database to: @@ -106,7 +106,7 @@ public class FinanceInfo } ``` -You can configure the `IEmbeddingGenerator` at the vector store level, collection level, or on individual vector properties. With auto-embedding, you can also pass a `string` directly to `SearchAsync` instead of a precomputed vector—the store generates the search embedding for you. For more information, see [Use built-in embedding generation](../how-to/use-vector-stores.md#use-built-in-embedding-generation). +You can configure the `IEmbeddingGenerator` at the vector store or collection level, or on individual vector properties. With auto-embedding, you can also pass a `string` directly to `SearchAsync` instead of a precomputed vector—the store generates the search embedding for you. For more information, see [Use built-in embedding generation](../how-to/use-vector-stores.md#use-built-in-embedding-generation). ### Key abstractions diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore b/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore deleted file mode 100644 index cd42ee34e873b..0000000000000 --- a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -bin/ -obj/ diff --git a/docs/ai/how-to/use-vector-stores.md b/docs/ai/how-to/use-vector-stores.md index de96757d2c859..a18f18ab43fed 100644 --- a/docs/ai/how-to/use-vector-stores.md +++ b/docs/ai/how-to/use-vector-stores.md @@ -3,7 +3,7 @@ title: Use vector stores in .NET AI apps description: Learn how to use Microsoft.Extensions.VectorData to store, search, and manage embeddings in vector databases for .NET AI applications. ms.topic: how-to ms.date: 02/24/2026 -ai-usage: ai-assisted +ai-usage: ai-generated --- # Use vector stores in .NET AI apps @@ -44,13 +44,13 @@ The `Dimensions` parameter in `[VectorStoreVector]` must match the output size o The following tables describe all available parameters for each attribute. -#### VectorStoreKey parameters +#### parameters | Parameter | Required | Description | |---|---|---| | `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | -#### VectorStoreData parameters +#### parameters | Parameter | Required | Description | |---|---|---| @@ -58,7 +58,7 @@ The following tables describe all available parameters for each attribute. | `IsFullTextIndexed` | No | Whether to index this property for full-text search. Default is `false`. | | `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | -#### VectorStoreVector parameters +#### parameters | Parameter | Required | Description | |---|---|---| @@ -81,7 +81,7 @@ Create an instance of the `VectorStore` implementation for your chosen database. ## Get a collection -Call `GetCollection` on the vector store to get a typed reference to a collection. Then call `EnsureCollectionExistsAsync` to create the collection if it doesn't already exist: +Call `GetCollection` on the to get a typed reference. Then call to create the collection if it doesn't already exist: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetCollection"::: @@ -108,11 +108,11 @@ To retrieve multiple records at once: ## Perform vector search -Use `SearchAsync` to find records that are semantically similar to a query. Pass the embedding vector for your query and the number of results to return: +Use to find records that are semantically similar to a query. Pass the embedding vector for your query and the number of results to return: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchBasic"::: -Each `VectorSearchResult` contains the matching record and a similarity score. Higher scores indicate a closer semantic match. +Each contains the matching record and a similarity score. Higher scores indicate a closer semantic match. ## Filter search results @@ -201,7 +201,7 @@ With this approach, `SearchAsync` also accepts a `string` query directly—you d Some vector stores support *hybrid search*, which combines vector similarity with keyword matching. This approach can improve result relevance compared to vector-only search. -To use hybrid search, check whether your collection implements `IKeywordHybridSearchable`. Only connectors for databases that support this feature implement this interface. +To use hybrid search, check whether your collection implements . Only connectors for databases that support this feature implement this interface. ```csharp // Check whether the collection supports hybrid search. @@ -227,19 +227,19 @@ For hybrid search to work, the data model must have at least one vector property ## Delete records -To delete a single record by key, use `DeleteAsync`: +To delete a single record by key, use : :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="DeleteRecord"::: ## Delete a collection -To remove an entire collection from the vector store, use `EnsureCollectionDeletedAsync`: +To remove an entire collection from the vector store, use : :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="DeleteCollection"::: ## Switch vector store connectors -Because all connectors implement the same `VectorStore` abstract class, you can switch between them by changing the concrete type at startup. Your collection and search code remains the same. +Because all connectors implement the same abstract class, you can switch between them by changing the concrete type at startup. Your collection and search code remains the same. For example, to switch from in-memory to Azure AI Search: diff --git a/docs/ai/toc.yml b/docs/ai/toc.yml index 466dc8c6c5500..771bca2fc6190 100644 --- a/docs/ai/toc.yml +++ b/docs/ai/toc.yml @@ -48,8 +48,6 @@ items: href: conceptual/understanding-tokens.md - name: Embeddings href: conceptual/embeddings.md - - name: Vector databases - href: conceptual/vector-databases.md - name: Data ingestion href: conceptual/data-ingestion.md - name: Prompt engineering From ab4c26a451cdeed78e291f7eb7bbf23cf725d07b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 26 Feb 2026 00:17:11 +0000 Subject: [PATCH 05/11] Add xref links to attribute parameters in tables, GetCollection, UpsertAsync, GetAsync, VectorSearchOptions Co-authored-by: gewarren <24882762+gewarren@users.noreply.github.com> --- docs/ai/how-to/use-vector-stores.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/ai/how-to/use-vector-stores.md b/docs/ai/how-to/use-vector-stores.md index a18f18ab43fed..10b9f7ec69dc6 100644 --- a/docs/ai/how-to/use-vector-stores.md +++ b/docs/ai/how-to/use-vector-stores.md @@ -48,24 +48,24 @@ The following tables describe all available parameters for each attribute. | Parameter | Required | Description | |---|---|---| -| `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | +| | No | An alternative name for the property in storage. Not supported by all connectors. | #### parameters | Parameter | Required | Description | |---|---|---| -| `IsIndexed` | No | Whether to index this property for filtering. Default is `false`. | -| `IsFullTextIndexed` | No | Whether to index this property for full-text search. Default is `false`. | -| `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | +| | No | Whether to index this property for filtering. Default is `false`. | +| | No | Whether to index this property for full-text search. Default is `false`. | +| | No | An alternative name for the property in storage. Not supported by all connectors. | #### parameters | Parameter | Required | Description | |---|---|---| -| `Dimensions` | Yes (for collection creation) | The number of dimensions in the vector. Must match your embedding model. | -| `IndexKind` | No | The type of index to use. Defaults vary by connector. Common values: `Hnsw`, `Flat`. | -| `DistanceFunction` | No | The distance function for similarity comparisons. Defaults vary by connector. Common values: `CosineSimilarity`, `DotProductSimilarity`, `EuclideanDistance`. | -| `StorageName` | No | An alternative name for the property in storage. Not supported by all connectors. | +| | Yes (for collection creation) | The number of dimensions in the vector. Must match your embedding model. | +| | No | The type of index to use. Defaults vary by connector. Common values: `Hnsw`, `Flat`. | +| | No | The distance function for similarity comparisons. Defaults vary by connector. Common values: `CosineSimilarity`, `DotProductSimilarity`, `EuclideanDistance`. | +| | No | An alternative name for the property in storage. Not supported by all connectors. | ## Define a schema programmatically @@ -81,7 +81,7 @@ Create an instance of the `VectorStore` implementation for your chosen database. ## Get a collection -Call `GetCollection` on the to get a typed reference. Then call to create the collection if it doesn't already exist: +Call on the to get a typed reference. Then call to create the collection if it doesn't already exist: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetCollection"::: @@ -89,7 +89,7 @@ The collection name maps to the underlying storage concept for your database (fo ## Upsert records -Use `UpsertAsync` to insert or update records in the collection. If a record with the same key already exists, it gets updated: +Use to insert or update records in the collection. If a record with the same key already exists, it gets updated: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="UpsertRecords"::: @@ -98,7 +98,7 @@ Use `UpsertAsync` to insert or update records in the collection. If a record wit ## Get records -Use `GetAsync` to retrieve a single record by its key. To retrieve multiple records, pass an `IEnumerable` to `GetAsync`: +Use to retrieve a single record by its key. To retrieve multiple records, pass an `IEnumerable` to : :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetRecord"::: @@ -124,7 +124,7 @@ Filters are expressed as LINQ expressions. The supported operations vary by conn ## Control search behavior with VectorSearchOptions -Use `VectorSearchOptions` to control various aspects of vector search behavior: +Use to control various aspects of vector search behavior: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchOptions"::: From 21b049ab586177e03361f834f3410d148fbe6c02 Mon Sep 17 00:00:00 2001 From: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Date: Wed, 25 Feb 2026 16:21:15 -0800 Subject: [PATCH 06/11] Apply suggestions from code review --- docs/ai/how-to/use-vector-stores.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/ai/how-to/use-vector-stores.md b/docs/ai/how-to/use-vector-stores.md index 10b9f7ec69dc6..f9c39e03adaa7 100644 --- a/docs/ai/how-to/use-vector-stores.md +++ b/docs/ai/how-to/use-vector-stores.md @@ -44,13 +44,13 @@ The `Dimensions` parameter in `[VectorStoreVector]` must match the output size o The following tables describe all available parameters for each attribute. -#### parameters +#### VectorStoreKeyAttribute parameters | Parameter | Required | Description | |---|---|---| | | No | An alternative name for the property in storage. Not supported by all connectors. | -#### parameters +#### VectorStoreDataAttribute parameters | Parameter | Required | Description | |---|---|---| @@ -58,7 +58,7 @@ The following tables describe all available parameters for each attribute. | | No | Whether to index this property for full-text search. Default is `false`. | | | No | An alternative name for the property in storage. Not supported by all connectors. | -#### parameters +#### VectorStoreVectorAttribute parameters | Parameter | Required | Description | |---|---|---| From cd8eada091cb3e30b568bad46f9d9c660681350a Mon Sep 17 00:00:00 2001 From: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Date: Fri, 27 Feb 2026 18:06:49 -0800 Subject: [PATCH 07/11] docs port in progress... --- docs/ai/toc.yml | 28 +- .../vector-stores/defining-your-data-model.md | 117 ++++++ docs/ai/vector-stores/dynamic-data-model.md | 77 ++++ docs/ai/vector-stores/embedding-generation.md | 276 ++++++++++++++ .../how-to}/build-vector-search-app.md | 0 .../how-to/build-your-own-connector.md | 276 ++++++++++++++ .../VectorStoresExamples/AutoEmbedding.cs | 0 .../snippets}/VectorStoresExamples/Hotel.cs | 0 .../snippets}/VectorStoresExamples/Program.cs | 0 .../VectorStoresExamples/RecordDefinition.cs | 0 .../VectorStoresExamples.csproj | 0 .../how-to/use-vector-stores.md | 12 +- .../how-to/vector-store-data-ingestion.md | 278 ++++++++++++++ docs/ai/vector-stores/hybrid-search.md | 243 ++++++++++++ .../azure-ai-search-connector.md | 146 ++++++++ .../azure-cosmosdb-mongodb-connector.md | 155 ++++++++ .../azure-cosmosdb-nosql-connector.md | 245 +++++++++++++ .../couchbase-connector.md | 347 ++++++++++++++++++ .../elasticsearch-connector.md | 197 ++++++++++ .../inmemory-connector.md | 76 ++++ .../mongodb-connector.md | 131 +++++++ .../oracle-connector.md | 225 ++++++++++++ .../pinecone-connector.md | 168 +++++++++ .../postgres-connector.md | 327 +++++++++++++++++ .../qdrant-connector.md | 224 +++++++++++ .../redis-connector.md | 239 ++++++++++++ .../sql-connector.md | 114 ++++++ .../sqlite-connector.md | 121 ++++++ .../volatile-connector.md | 73 ++++ .../weaviate-connector.md | 173 +++++++++ .../overview.md} | 146 +------- .../schema-with-record-definition.md | 106 ++++++ .../tutorial-vector-search.md} | 6 +- docs/ai/vector-stores/vector-search.md | 228 ++++++++++++ 34 files changed, 4602 insertions(+), 152 deletions(-) create mode 100644 docs/ai/vector-stores/defining-your-data-model.md create mode 100644 docs/ai/vector-stores/dynamic-data-model.md create mode 100644 docs/ai/vector-stores/embedding-generation.md rename docs/ai/{quickstarts => vector-stores/how-to}/build-vector-search-app.md (100%) create mode 100644 docs/ai/vector-stores/how-to/build-your-own-connector.md rename docs/ai/{how-to/snippets/use-vector-stores/csharp => vector-stores/how-to/snippets}/VectorStoresExamples/AutoEmbedding.cs (100%) rename docs/ai/{how-to/snippets/use-vector-stores/csharp => vector-stores/how-to/snippets}/VectorStoresExamples/Hotel.cs (100%) rename docs/ai/{how-to/snippets/use-vector-stores/csharp => vector-stores/how-to/snippets}/VectorStoresExamples/Program.cs (100%) rename docs/ai/{how-to/snippets/use-vector-stores/csharp => vector-stores/how-to/snippets}/VectorStoresExamples/RecordDefinition.cs (100%) rename docs/ai/{how-to/snippets/use-vector-stores/csharp => vector-stores/how-to/snippets}/VectorStoresExamples/VectorStoresExamples.csproj (100%) rename docs/ai/{ => vector-stores}/how-to/use-vector-stores.md (96%) create mode 100644 docs/ai/vector-stores/how-to/vector-store-data-ingestion.md create mode 100644 docs/ai/vector-stores/hybrid-search.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md rename docs/ai/{conceptual/vector-databases.md => vector-stores/overview.md} (56%) create mode 100644 docs/ai/vector-stores/schema-with-record-definition.md rename docs/ai/{tutorials/tutorial-ai-vector-search.md => vector-stores/tutorial-vector-search.md} (94%) create mode 100644 docs/ai/vector-stores/vector-search.md diff --git a/docs/ai/toc.yml b/docs/ai/toc.yml index 771bca2fc6190..4b57ee1d0be7e 100644 --- a/docs/ai/toc.yml +++ b/docs/ai/toc.yml @@ -77,7 +77,7 @@ items: - name: Get started with the RAG sample href: get-started-app-chat-template.md - name: Implement RAG using vector search - href: tutorials/tutorial-ai-vector-search.md + href: vector-stores/tutorial-vector-search.md - name: Scale Azure OpenAI with Azure Container Apps href: get-started-app-chat-scaling-with-azure-container-apps.md - name: MCP client/server @@ -94,10 +94,30 @@ items: href: how-to/use-tokenizers.md - name: Vector stores items: - - name: Vector databases overview - href: conceptual/vector-databases.md + - name: Vector databases overview + href: vector-stores/overview.md + - name: How-to + items: - name: Use vector stores - href: how-to/use-vector-stores.md + href: vector-stores/how-to/use-vector-stores.md + - name: Build a vector search app + href: vector-stores/how-to/build-vector-search-app.md + - name: Ingest data into a vector store + href: vector-stores/how-to/vector-store-data-ingestion.md + - name: Build a vector store connector + href: vector-stores/how-to/build-your-own-connector.md + - name: Define your data model + href: vector-stores/defining-your-data-model.md + - name: Define schema with record definitions + href: vector-stores/schema-with-record-definition.md + - name: Dynamic data model + href: vector-stores/dynamic-data-model.md + - name: Generate embeddings + href: vector-stores/embedding-generation.md + - name: Vector search + href: vector-stores/vector-search.md + - name: Hybrid search + href: vector-stores/hybrid-search.md - name: Security and content safety items: - name: Authentication for Azure-hosted apps and services diff --git a/docs/ai/vector-stores/defining-your-data-model.md b/docs/ai/vector-stores/defining-your-data-model.md new file mode 100644 index 0000000000000..f8135c4390943 --- /dev/null +++ b/docs/ai/vector-stores/defining-your-data-model.md @@ -0,0 +1,117 @@ +--- +title: Define your Vector Store data model +description: Describes how to create a data model with Microsoft.Extensions.VectorData to use when writing to or reading from a Vector Store. +ms.topic: reference +ms.date: 07/08/2024 +--- +# Define your data model + +## Overview + +The Vector Store connectors use a model-first approach to interacting with databases. + +All methods to upsert or get records use strongly typed model classes. +The properties on these classes are decorated with attributes that indicate the purpose of each property. + +> [!TIP] +> For an alternative to using attributes, see [defining your schema with a record definition](./schema-with-record-definition.md). +> [!TIP] +> For an alternative to defining your own data model, see [using Vector Store abstractions without defining your own data model](./generic-data-model.md). + +Here is an example of a model that is decorated with these attributes. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string[] Tags { get; set; } +} +``` + +## Attributes + +### VectorStoreKeyAttribute + +Use the attribute to indicate that your property is the key of the record. + +```csharp +[VectorStoreKey] +public ulong HotelId { get; set; } +``` + +#### VectorStoreKeyAttribute parameters + +| Parameter | Required | Description | +|---------------|:--------:|-------------| +| | No | Can be used to supply an alternative name for the property in the database. This parameter isn't supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` are supported. | + +### VectorStoreDataAttribute + +Use the attribute to indicate that your property contains general data that is not a key or a vector. + +```csharp +[VectorStoreData(IsIndexed = true)] +public string HotelName { get; set; } +``` + +#### VectorStoreDataAttribute parameters + +| Parameter | Required | Description | +|-------------|:--------:|-------------| +| | No | Indicates whether the property should be indexed for filtering in cases where a database requires opting in to indexing per property. The default is `false`. | +| | No | Indicates whether the property should be indexed for full text search for databases that support full text search. The default is `false`. | +| | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` are supported. | + +> [!TIP] +> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). + +### VectorStoreVectorAttribute + +Use the attribute to indicate that your property contains a vector. + +```csharp +[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] +public ReadOnlyMemory? DescriptionEmbedding { get; set; } +``` + +It's also possible to use on properties that dont' have a vector type, for example, a property of type `string`. +When a property is decorated in this way, you need to provide an instance to the vector store. +When upserting the record, the text that is in the `string` property is automatically converted and stored as a vector in the database. +It's not possible to retrieve a vector using this mechanism. + +```csharp +[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] +public string DescriptionEmbedding { get; set; } +``` + +> [!TIP] +> For more information on how to use built-in embedding generation, see [Let the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). + +#### VectorStoreVectorAttribute parameters + +| Parameter | Required | Description | +|------------|:--------:|-------------| +| `Dimensions` | Yes | The number of dimensions that the vector has. This is required when creating a vector index for a collection. | +| | No | The type of index to index the vector with. Default varies by vector store type. | +| | No | The type of function to use when doing vector comparison during vector search over this vector. Default varies by vector store type. | +| | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | + +Common index kinds and distance function types are supplied as static values on the and classes. +Individual Vector Store implementations might also use their own index kinds and distance functions, where the database supports unusual types. + +> [!TIP] +> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). \ No newline at end of file diff --git a/docs/ai/vector-stores/dynamic-data-model.md b/docs/ai/vector-stores/dynamic-data-model.md new file mode 100644 index 0000000000000..2fc296aafbdb9 --- /dev/null +++ b/docs/ai/vector-stores/dynamic-data-model.md @@ -0,0 +1,77 @@ +--- +title: Using Vector Store abstractions without defining your own data model +description: Describes how to use Vector Store abstractions without defining your own data model. +ms.topic: reference +ms.date: 10/16/2024 +--- +# Use Vector Store abstractions without defining your own data model + +The Vector Store connectors use a model-first approach to interact with databases. This makes using the connectors easy and simple, since +your data model reflects the schema of your database records and to add any additional schema information required, you can simply add attributes to your data model properties. + +However, there are cases where it isn't desirable or possible to define your own data model. For example, imagine that you don't know at compile time what your +database schema looks like, and the schema is only provided via configuration. Creating a data model that reflects the schema would be impossible in this case. + +In this case, you can use a `Dictionary` for the record type. Properties are added to the `Dictionary` with key as the property name and the value as the property value. + +## Supply schema information when using `Dictionary` + +When using a `Dictionary`, connectors still need to know what the database schema looks like. Without the schema information +the connector would not be able to create a collection, or know how to map to and from the storage representation that each database uses. + +A record definition can be used to provide the schema information. Unlike a data model, a record definition can be created from configuration at runtime, providing a solution for when schema information is not known at compile time. + +> [!TIP] +> To see how to create a record definition, see [defining your schema with a record definition](./schema-with-record-definition.md). + +## Example + +To use the `Dictionary` with a connector, simply specify it as your data model when creating a collection, and simultaneously provide a record definition. + +```csharp +// Create the definition to define the schema. +VectorStoreCollectionDefinition definition = new() +{ + Properties = new List + { + new VectorStoreKeyProperty("Key", typeof(string)), + new VectorStoreDataProperty("Term", typeof(string)), + new VectorStoreDataProperty("Definition", typeof(string)), + new VectorStoreVectorProperty("DefinitionEmbedding", typeof(ReadOnlyMemory), dimensions: 1536) + } +}; + +// When getting your collection instance from a vector store instance +// specify the Dictionary, using object as the key type for your database +// and also pass your record definition. +// Note that you have to use GetDynamicCollection instead of the regular GetCollection method +// to get an instance of a collection using Dictionary. +var dynamicDataModelCollection = vectorStore.GetDynamicCollection( + "glossary", + definition); + +// Since schema information is available from the record definition +// it's possible to create a collection with the right vectors, +// dimensions, indexes, and distance functions. +await dynamicDataModelCollection.EnsureCollectionExistsAsync(); + +// When retrieving a record from the collection, key, data, and vector values can +// now be accessed via the dictionary entries. +var record = await dynamicDataModelCollection.GetAsync("SK"); +Console.WriteLine(record["Definition"]); +``` + +When constructing a collection instance directly, the record definition +is passed as an option. For example, here is an example of constructing +an Azure AI Search collection instance with `Dictionary`. + +Each vector store collection implementation has a separate `*DynamicCollection` +class that can be used with `Dictionary`. +This is because these implementations might support NativeAOT/trimming. + +```csharp +new AzureAISearchDynamicCollection( + searchIndexClient, + "glossary", + new() { Definition = definition }); +``` diff --git a/docs/ai/vector-stores/embedding-generation.md b/docs/ai/vector-stores/embedding-generation.md new file mode 100644 index 0000000000000..a7fb2cda0642c --- /dev/null +++ b/docs/ai/vector-stores/embedding-generation.md @@ -0,0 +1,276 @@ +--- +title: Generating embeddings for Vector Store connectors +description: Describes how you can generate embeddings to use with Vector Store connectors. +ms.topic: concept-article +ms.date: 09/23/2024 +--- + +# Generating embeddings for Vector Store connectors + +Vector Store connectors support multiple ways of generating embeddings. +Embeddings can be generated by the developer and passed as part of a record when using a or can be generated internally to the . + +## Let the Vector Store generate embeddings + +You can configure an embedding generator on your vector store, allowing embeddings to be automatically generated during both upsert and search operations, eliminating the need for manual preprocessing. + +To enable generating vectors automatically on upsert, the vector property on your data model is defined as the source type, for example, string but still decorated with a . + +```csharp +[VectorStoreVector(1536)] +public string Embedding { get; set; } +``` + +Before upsert, the `Embedding` property should contain the string from which a vector should be generated. The type of the vector stored in the database (for example, float32, float16, etc.) will be derived from the configured embedding generator. + +> [!IMPORTANT] +> These vector properties do not support retrieving either the generated vector or the original text that the vector was generated from. They also do not store the original text. +> If the original text needs to be stored, a separate Data property should be added to store it. + +Embedding generators implementing the `Microsoft.Extensions.AI` abstractions are supported and can be configured at various levels: + +1. **On the Vector Store**: + + You can set a default embedding generator for the entire vector store. This generator will be used for all collections and properties unless overridden. + + ```csharp + using Microsoft.Extensions.AI; + using Microsoft.SemanticKernel.Connectors.Qdrant; + using OpenAI; + using Qdrant.Client; + + var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + + var vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + ownsClient: true, + new QdrantVectorStoreOptions + { + EmbeddingGenerator = embeddingGenerator + }); + ``` + +2. **On a Collection**: + + You can configure an embedding generator for a specific collection, overriding the store-level generator. + + ```csharp + using Microsoft.Extensions.AI; + using Microsoft.SemanticKernel.Connectors.Qdrant; + using OpenAI; + using Qdrant.Client; + + var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + + var collectionOptions = new QdrantCollectionOptions + { + EmbeddingGenerator = embeddingGenerator + }; + var collection = new QdrantCollection( + new QdrantClient("localhost"), + "myCollection", + ownsClient: true, + collectionOptions); + ``` + +3. **On a Record Definition**: + When defining properties programmatically using , you can specify an embedding generator for all properties. + + ```csharp + using Microsoft.Extensions.AI; + using Microsoft.Extensions.VectorData; + using Microsoft.SemanticKernel.Connectors.Qdrant; + using OpenAI; + using Qdrant.Client; + + var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + + var definition = new VectorStoreCollectionDefinition + { + EmbeddingGenerator = embeddingGenerator, + Properties = new List + { + new VectorStoreKeyProperty("Key", typeof(ulong)), + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) + } + }; + + var collectionOptions = new QdrantCollectionOptions + { + Definition = definition + }; + var collection = new QdrantCollection( + new QdrantClient("localhost"), + "myCollection", + ownsClient: true, + collectionOptions); + ``` + +4. **On a Vector Property Definition**: + When defining properties programmatically, you can set an embedding generator directly on the property. + + ```csharp + using Microsoft.Extensions.AI; + using Microsoft.Extensions.VectorData; + using OpenAI; + + var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + + var vectorProperty = new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) + { + EmbeddingGenerator = embeddingGenerator + }; + ``` + +### Example usage + +The following example demonstrates how to use the embedding generator to automatically generate vectors during both upsert and search operations. This approach simplifies workflows by eliminating the need to precompute embeddings manually. + +```csharp + +// The data model +internal class FinanceInfo +{ + [VectorStoreKey] + public string Key { get; set; } = string.Empty; + + [VectorStoreData] + public string Text { get; set; } = string.Empty; + + // Note that the vector property is typed as a string, and + // its value is derived from the Text property. The string + // value will however be converted to a vector on upsert and + // stored in the database as a vector. + [VectorStoreVector(1536)] + public string Embedding => this.Text; +} + +// Create an OpenAI embedding generator. +var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + +// Use the embedding generator with the vector store. +var vectorStore = new InMemoryVectorStore(new() { EmbeddingGenerator = embeddingGenerator }); +var collection = vectorStore.GetCollection("finances"); +await collection.EnsureCollectionExistsAsync(); + +// Create some test data. +string[] budgetInfo = +{ + "The budget for 2020 is EUR 100 000", + "The budget for 2021 is EUR 120 000", + "The budget for 2022 is EUR 150 000", + "The budget for 2023 is EUR 200 000", + "The budget for 2024 is EUR 364 000" +}; + +// Embeddings are generated automatically on upsert. +var records = budgetInfo.Select((input, index) => new FinanceInfo { Key = index.ToString(), Text = input }); +await collection.UpsertAsync(records); + +// Embeddings for the search is automatically generated on search. +var searchResult = collection.SearchAsync( + "What is my budget for 2024?", + top: 1); + +// Output the matching result. +await foreach (var result in searchResult) +{ + Console.WriteLine($"Key: {result.Record.Key}, Text: {result.Record.Text}"); +} +``` + +## Generate embeddings yourself + +### Construct an embedding generator + +For information on how to construct `Microsoft.Extensions.AI` embedding generators, see [Microsoft.Extensions.AI.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.AI.Abstractions). + +### Generate embeddings on upsert with `IEmbeddingGenerator` + +```csharp +public async Task GenerateEmbeddingsAndUpsertAsync( + IEmbeddingGenerator> embeddingGenerator, + VectorStoreCollection collection) +{ + // Upsert a record. + string descriptionText = "A place where everyone can be happy."; + ulong hotelId = 1; + + // Generate the embedding. + ReadOnlyMemory embedding = + (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; + + // Create a record and upsert with the already generated embedding. + await collection.UpsertAsync(new Hotel + { + HotelId = hotelId, + HotelName = "Hotel Happy", + Description = descriptionText, + DescriptionEmbedding = embedding, + Tags = new[] { "luxury", "pool" } + }); +} +``` + +### Generate embeddings on search with `IEmbeddingGenerator` + +```csharp +public async Task GenerateEmbeddingsAndSearchAsync( + IEmbeddingGenerator> embeddingGenerator, + VectorStoreCollection collection) +{ + // Upsert a record. + string descriptionText = "Find me a hotel with happiness in mind."; + + // Generate the embedding. + ReadOnlyMemory searchEmbedding = + (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; + + // Search using the already generated embedding. + IAsyncEnumerable> searchResult = collection.SearchAsync(searchEmbedding, top: 1); + List> resultItems = await searchResult.ToListAsync(); + + // Print the first search result. + Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); + Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); +} +``` + +## Embedding dimensions + +Vector databases typically require you to specify the number of dimensions that each vector has when creating the collection. +Different embedding models typically support generating vectors with various dimension sizes. For example, OpenAI `text-embedding-ada-002` +generates vectors with 1536 dimensions. Some models also allow a developer to choose the number of dimensions they want in the +output vector. For example, Google `text-embedding-004` produces vectors with 768 dimensions by default, but allows a developer to +choose any number of dimensions between 1 and 768. + +It's important to ensure that the vectors generated by the embedding model have the same number of dimensions as the +matching vector in the database. + +If creating a collection using the Vector Store abstractions, you need to specify the number of dimensions +required for each vector property either via annotations or via the record definition. Here are examples of both setting +the number of dimensions to 1536. + +```csharp +[VectorStoreVector(Dimensions: 1536)] +public ReadOnlyMemory? DescriptionEmbedding { get; set; } +``` + +```csharp +new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 1536); +``` + +## See also + +- [Defining your data model](./defining-your-data-model.md) +- [Defining your schema with a record definition](./schema-with-record-definition.md) \ No newline at end of file diff --git a/docs/ai/quickstarts/build-vector-search-app.md b/docs/ai/vector-stores/how-to/build-vector-search-app.md similarity index 100% rename from docs/ai/quickstarts/build-vector-search-app.md rename to docs/ai/vector-stores/how-to/build-vector-search-app.md diff --git a/docs/ai/vector-stores/how-to/build-your-own-connector.md b/docs/ai/vector-stores/how-to/build-your-own-connector.md new file mode 100644 index 0000000000000..5b39688ccfe1d --- /dev/null +++ b/docs/ai/vector-stores/how-to/build-your-own-connector.md @@ -0,0 +1,276 @@ +--- +title: How to build your own Vector Store connector (Preview) +description: Describes how to build your own Vector Store connector connector +ms.topic: tutorial +ms.date: 07/08/2024 +--- +# How to build your own Vector Store connector (Preview) + +This article provides guidance for building a Vector Store connector. This article can be used by database providers that want to build and maintain their own implementation, or for anyone that wants to build and maintain an unofficial connector for a database that lacks support. + +To contribute your connector to the Semantic Kernel code base: + +1. Create an issue in the [Semantic Kernel Github repository](https://github.com/microsoft/semantic-kernel/issues). +1. Review the [Semantic Kernel contribution guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md). + +## Overview + +Vector Store connectors are implementations of the [Vector Store abstraction](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions). Some of the decisions that were made when designing the Vector Store abstraction mean that a Vector Store connector requires certain features to provide users with a good experience. + +A key design decision is that the Vector Store abstraction takes a strongly typed approach to working with database records. +This means that takes a strongly typed record as input, while returns a strongly typed record. +The design uses C# generics to achieve the strong typing. This means that a connector has to be able to map from this data model to the storage model used by the underlying database. It also means that a connector might need to find out certain information about the record properties in order to know how to map each of these properties. For example, some vector databases (such as Chroma, Qdrant and Weaviate) require vectors to be stored in a specific structure and non-vectors in a different structure, or require record keys to be stored in a specific field. + +At the same time, the Vector Store abstraction also provides a generic data model that allows a developer to work with a database without needing to create a custom data model. + +It's important for connectors to support different types of model and provide developers with flexibility around how they use the connector. The following section deep dives into each of these requirements. + +## Requirements + +To be considered a full implementation of the Vector Store abstractions, the following set of requirements must be met. + +### 1. Implement the core abstract base clases and interfaces + +1.1 The three core abstract base classes and interfaces that need to be implemented are: + +- +- +- + + implements `IVectorSearchable`, so only two inheriting classes are required. Use the following naming convention: + +- {database type}VectorStore : VectorStore +- {database type}Collection : VectorStoreCollection\ + +For example: + +- MyDbVectorStore : VectorStore +- MyDbCollection : VectorStoreCollection\ + +The implementation should accept the name of the collection as a constructor parameter and each instance of it is therefore tied to a specific collection instance in the database. + +Here follows specific requirements for individual methods on these abstract base classes and interfaces. + +1.2 *`VectorStore.GetCollection`* implementations should not do any checks to verify whether a collection exists or not. +The method should simply construct a collection object and return it. The user can optionally use the `CollectionExistsAsync` method to check if the collection exists in cases where this is not known. Doing checks on each invocation of `GetCollection` might add unwanted overhead for users when they are working with a collection that they know exists. + +1.3 *`VectorStoreCollection.DeleteAsync`* that takes a single key as input should succeed if the record does not exist and for any other failures an exception should be thrown. +For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. + +1.4 *`VectorStoreCollection.DeleteAsync`* that takes multiple keys as input should succeed if any of the requested records do not exist and for any other failures an exception should be thrown. +For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. + +1.5 *`VectorStoreCollection.GetAsync`* that takes a single key as input should return null and not throw if a record is not found. For any other failures an exception should be thrown. +For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. + +1.6 *`VectorStoreCollection.GetAsync`* that takes multiple keys as input should return the subset of records that were found and not throw if any of the requested records were not found. For any other failures an exception should be thrown. +For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. + +1.7 *`VectorStoreCollection.GetAsync`* implementations should respect the `IncludeVectors` option provided via `RecordRetrievalOptions` where possible. +Vectors are often most useful in the database itself, since that is where vector comparison happens during vector searches and downloading them can be costly due to their size. +There might be cases where the database doesn't support excluding vectors in which case returning them is acceptable. + +1.8 *`IVectorSearchable.SearchAsync`* implementations should also respect the `IncludeVectors` option provided via `VectorSearchOptions` where possible. + +1.9 *`IVectorSearchable.SearchAsync`* implementations should simulate the `Top` and `Skip` functionality requested via `VectorSearchOptions` if the database +does not support this natively. To simulate this behavior, the implementation should fetch a number of results equal to Top + Skip, and then skip the first Skip number of results +before returning the remaining results. + +1.10 *`IVectorSearchable.SearchAsync`* implementations should not require `VectorPropertyName` or `VectorProperty` to be specified if only one vector exists on the data model. +In this case that single vector should automatically become the search target. If no vector or multiple vectors exists on the data model, and no `VectorPropertyName` or `VectorProperty` is provided the search method should throw. + +When using `VectorPropertyName`, if a user does provide this value, the expected name should be the +property name from the data model and not any customized name that the property might be stored under +in the database. For example, imagine that the user has a data model property called `TextEmbedding` and they +decorated the property with a `JsonPropertyNameAttribute` that indicates that it should be serialized +as `text_embedding`. Assuming that the database is json based, it means that the property should be +stored in the database with the name `text_embedding`. + When specifying the `VectorPropertyName` option, the user should always provide +`TextEmbedding` as the value. This is to ensure that where different connectors are used with the +same data model, the user can always use the same property names, even though the storage name +of the property might be different. + +### 2. Support data model attributes + +The Vector Store abstraction allows a user to use attributes to decorate their data model to indicate the type of each property and to configure the type of indexing required +for each vector property. + +This information is typically required for + +- Mapping between a data model and the underlying database's storage model. +- Creating a collection or index. +- Vector search. + +If the user does not provide a , read this information from the data model attributes using reflection. If the user did provide a +, don't use the data model as the source of truth. + +> [!TIP] +> For a detailed list of all attributes and settings that need to be supported, see [Defining your data model](../defining-your-data-model.md). + +### 3. Support record definitions + +As mentioned in [Support data model attributes](#2-support-data-model-attributes), you need information about each property to build out a connector. This information can also +be supplied via a and if supplied, the connector should avoid trying to read this information from the data model or try and validate that the +data model matches the definition in any way. + +The user should be able to provide a to the implementation via options. + +> [!TIP] +> Refer to [Defining your storage schema using a record definition](../schema-with-record-definition.md) for a detailed list of > all record definition settings that need to be supported. + +### 4. Collection and index creation + +4.1 A user can optionally choose an index kind and distance function for each vector property. +These are specified via string based settings, but where available a connector should expect +the strings that are provided as string consts on `Microsoft.Extensions.VectorData.IndexKind` +and `Microsoft.Extensions.VectorData.DistanceFunction`. Where the connector requires +index kinds and distance functions that are not available on the above mentioned static classes +additional custom strings might be accepted. + +For example, the goal is for a user to be able to specify a standard distance function, like `DotProductSimilarity` +for any connector that supports this distance function, without needing to use different naming for each connector. + +```csharp +[VectorStoreVector(1536, DistanceFunction = DistanceFunction.DotProductSimilarity] +public ReadOnlyMemory? Embedding { get; set; } +``` + +4.2 A user can optionally choose whether each data property should be indexed or full text indexed. +In some databases, all properties might already be filterable or full text searchable by default, however +in many databases, special indexing is required to achieve this. If special indexing is required +this also means that adding this indexing will most likely incur extra cost. +The and settings allow a user to control whether to enable +this additional indexing per property. + +### 5. Data model validation + +Every database doesn't support every data type. To improve the user experience it's important to validate +the data types of any record properties and to do so early, for example, when an +instance is constructed. This way the user will be notified of any potential failures before starting to use the database. + +### 6. Storage property naming + +The naming conventions used for properties in code doesn't always match the preferred naming +for matching fields in a database. +It's therefore valuable to support customized storage names for properties. +Some databases might support storage formats that already have their own mechanism for +specifying storage names, for example, when using JSON as the storage format you can use +a `JsonPropertyNameAttribute` to provide a custom name. + +6.1 Where the database has a storage format that supports its own mechanism for specifying storage +names, the connector should preferably use that mechanism. + +6.2 Where the database does not use a storage format that supports its own mechanism for specifying +storage names, the connector must support the settings from the data model +attributes or the . + +### 7. Mapper support + +Connectors should provide the ability to map between the user supplied data model and the +storage model that the database requires, but should also provide some flexibility in how +that mapping is done. Most connectors would typically need to support the following two +mappers. + +7.1 All connectors should come with a built in mapper that can map between the user supplied +data model and the storage model required by the underlying database. + +7.2. All connectors should have a built in mapper that works with the . +See [Support GenericDataModel](#8-support-genericdatamodel) for more information. + +### 8. Support GenericDataModel + +While it is very useful for users to be able to define their own data model, in some cases +it might not be desirable, for example, when the database schema is not known at coding time and driven +by configuration. + +To support this scenario, connectors should have out of the box support for the generic data +model supplied by the abstraction package: `Microsoft.Extensions.VectorData.VectorStoreGenericDataModel`. + +In practice this means that the connector must implement a special mapper +to support the generic data model. The connector should automatically use this mapper +if the user specified the generic data model as their data model. + +### 9. Divergent data model and database schema + +The only divergence required to be supported by connector implementations are +customizing storage property names for any properties. + +Any more complex divergence is not supported, since this causes additional +complexity for filtering. For example, if the user has a filter expression that references +the data model, but the underlying schema is different to the data model, the +filter expression cannot be used against the underlying schema. + +### 10. Standard Exceptions + +The database operation methods provided by the connector should throw a set of standard +exceptions so that users of the abstraction know what exceptions they need to handle, +instead of having to catch a different set for each provider. For example, if the underlying +database client throws a `MyDBClientException` when a call to the database fails, this +should be caught and wrapped in a , preferably preserving +the original exception as an inner exception. + +11.1 For failures relating to service call or database failures the connector should throw: +`Microsoft.Extensions.VectorData.VectorStoreOperationException` + +11.2 For mapping failures, the connector should throw: +`Microsoft.Extensions.VectorData.VectorStoreRecordMappingException` + +11.3 For cases where a certain setting or feature is not supported, for example, an unsupported index type, use: +`System.NotSupportedException`. + +11.4 In addition, use `System.ArgumentException`, `System.ArgumentNullException` for argument validation. + +### 11. Batching + +The abstract base class includes batching overloads for Get, Upsert and Delete. +Not all underlying database clients might have the same level of support for batching. + +The base batch method implementations on calls the abstract non-batch implementations in serial. +If the database supports batching natively, these base batch implementations should be overridden and implemented +using the native database support. + +## Recommended common patterns and practices + +- Keep and implementations sealed. It's recommended to use a decorator pattern to override a default vector store behavior. +- Always use options classes for optional settings with smart defaults. +- Keep required parameters on the main signature and move optional parameters to options. + +Here is an example of an constructor following this pattern. + +```csharp +public sealed class MyDBCollection : VectorStoreCollection +{ + public MyDBCollection(MyDBClient myDBClient, string collectionName, MyDBCollectionOptions? options = default) + { + } + + ... +} + +public class MyDBCollectionOptions : VectorStoreCollectionOptions +{ +} +``` + +## SDK changes + +See the following articles for a history of changes to the SDK and therefore implementation requirements: + +- [Vector Store Changes March 2025](../../../support/migration/vectorstore-march-2025.md) +- [Vector Store Changes April 2025](../../../support/migration/vectorstore-april-2025.md) +- [Vector Store Changes May 2025](../../../support/migration/vectorstore-may-2025.md) + +## Documentation + +To share the features and limitations of your implementation, you can contribute an article to this +Microsoft Learn website. For the documentation on existing connectors, see [Out-of-the-box Vector Store connectors](../out-of-the-box-connectors/index.md). + +To create your article, create a pull request in the [.NET docs GitHub repository](https://github.com/dotnet/docs). + +Areas to cover: + +1. An `Overview` with a standard table describing the main features of the connector. +1. An optional `Limitations` section with any limitations for your connector. +1. A `Getting started` section that describes how to import your NuGet and construct your and . +1. A `Data mapping` section showing the connector's default data mapping mechanism to the database storage model, including any property renaming it might support. +1. Information about additional features your connector supports. \ No newline at end of file diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AutoEmbedding.cs similarity index 100% rename from docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs rename to docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AutoEmbedding.cs diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Hotel.cs similarity index 100% rename from docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs rename to docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Hotel.cs diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Program.cs similarity index 100% rename from docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs rename to docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Program.cs diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/RecordDefinition.cs similarity index 100% rename from docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs rename to docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/RecordDefinition.cs diff --git a/docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/VectorStoresExamples.csproj b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/VectorStoresExamples.csproj similarity index 100% rename from docs/ai/how-to/snippets/use-vector-stores/csharp/VectorStoresExamples/VectorStoresExamples.csproj rename to docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/VectorStoresExamples.csproj diff --git a/docs/ai/how-to/use-vector-stores.md b/docs/ai/vector-stores/how-to/use-vector-stores.md similarity index 96% rename from docs/ai/how-to/use-vector-stores.md rename to docs/ai/vector-stores/how-to/use-vector-stores.md index f9c39e03adaa7..36e34286cd623 100644 --- a/docs/ai/how-to/use-vector-stores.md +++ b/docs/ai/vector-stores/how-to/use-vector-stores.md @@ -15,7 +15,7 @@ This article shows you how to use the key features of the library. ## Prerequisites - [.NET 9 SDK](https://dotnet.microsoft.com/download/dotnet/9.0) or later -- An understanding of [embeddings](../conceptual/embeddings.md) and [vector databases](../conceptual/vector-databases.md) +- An understanding of [embeddings](../conceptual/embeddings.md) and [vector databases](../overview.md) ## Install the packages @@ -26,7 +26,7 @@ dotnet add package Microsoft.Extensions.VectorData.Abstractions dotnet add package Microsoft.SemanticKernel.Connectors.InMemory --prerelease ``` -For production scenarios, replace `Microsoft.SemanticKernel.Connectors.InMemory` with the connector for your database. For a full list of available connectors, see [Available vector store connectors](../conceptual/vector-databases.md#available-vector-store-connectors). +For production scenarios, replace `Microsoft.SemanticKernel.Connectors.InMemory` with the connector for your database. For a full list of available connectors, see [Available vector store connectors](../overview.md#available-vector-store-connectors). ## Define a data model @@ -75,7 +75,7 @@ As an alternative to using attributes, you can define your schema programmatical ## Create a vector store -Create an instance of the `VectorStore` implementation for your chosen database. The following example creates an in-memory vector store: +Create an instance of the implementation for your chosen database. The following example creates an in-memory vector store: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="CreateVectorStore"::: @@ -192,7 +192,7 @@ Then configure the embedding generator when creating the vector store: :::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs" id="AutoEmbeddingVectorStore"::: -With this approach, `SearchAsync` also accepts a `string` query directly—you don't need to generate a query embedding manually. +With this approach, also accepts a `string` query directly—you don't need to generate a query embedding manually. > [!IMPORTANT] > Automatic embedding generation doesn't retrieve or store the original source text. If you need to access the original text later, store it in a separate `[VectorStoreData]` property on your model. @@ -255,7 +255,7 @@ This approach lets you develop and test locally without any external services, t ## Related content -- [Vector databases for .NET AI apps](../conceptual/vector-databases.md) +- [Vector databases for .NET AI apps](../overview.md) - [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md) - [Data ingestion](../conceptual/data-ingestion.md) -- [Embeddings in .NET](../conceptual/embeddings.md) +- [Embeddings in .NET](../conceptual/embeddings.md) \ No newline at end of file diff --git a/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md b/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md new file mode 100644 index 0000000000000..4fa824f03e40b --- /dev/null +++ b/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md @@ -0,0 +1,278 @@ +--- +title: How to ingest data into a Vector Store +description: Step by step instructions on how to ingest data into a Vector Store. +ms.topic: tutorial +ms.date: 07/08/2024 +--- +# How to ingest data into a Vector Store + +This article demonstrates how to create an application to: + +1. Take text from each paragraph in a Microsoft Word document. +2. Generate an embedding for each paragraph. +3. Upsert the text, embedding, and a reference to the original location into a Redis instance. + +## Prerequisites + +For this sample you need: + +- An embedding generation model hosted in Azure or another provider of your choice. +- An instance of Redis or Docker Desktop so that you can run Redis locally. +- A Word document to parse and load. Here is a zip containing a sample Word document you can download and use: [vector-store-data-ingestion-input.zip](../../../media/vector-store-data-ingestion-input.zip). + +## Set up Redis + +If you already have a Redis instance, you can use that. If you prefer to test your project locally, +you can easily start a Redis container using Docker: + +```docker +docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latest +``` + +To verify that it's running successfully, navigate to [http://localhost:8001/redis-stack/browser](http://localhost:8001/redis-stack/browser) in your browser. + +The rest of these instructions assume that you're using a container with these settings. + +## Create your project + +Create a new project and add NuGet package references for the Redis connector, the OpenXml package to read the Word document, and the Azure OpenAI packages for generating embeddings. + +```dotnetcli +dotnet new console --framework net8.0 --name SKVectorIngest +cd SKVectorIngest +dotnet add package Azure.AI.OpenAI +dotnet add package Microsoft.Extensions.AI.OpenAI +dotnet add package Microsoft.SemanticKernel.Connectors.Redis --prerelease +dotnet add package DocumentFormat.OpenXml +``` + +## Add a data model + +To upload data, you must first describe what format the data should have in the database. +You can do this by creating a data model with attributes that describe the function of each property. + +Add a new file to the project called `TextParagraph.cs` and add the following model to it. + +```csharp +using Microsoft.Extensions.VectorData; + +namespace SKVectorIngest; + +internal class TextParagraph +{ + /// A unique key for the text paragraph. + [VectorStoreKey] + public required string Key { get; init; } + + /// A uri that points at the original location of the document containing the text. + [VectorStoreData] + public required string DocumentUri { get; init; } + + /// The id of the paragraph from the document containing the text. + [VectorStoreData] + public required string ParagraphId { get; init; } + + /// The text of the paragraph. + [VectorStoreData] + public required string Text { get; init; } + + /// The embedding generated from the Text. + [VectorStoreVector(1536)] + public ReadOnlyMemory TextEmbedding { get; set; } +} +``` + +The value `1536`, which is the dimension size of the vector, is passed to the . This value must match the size of vector that your chosen embedding generator produces. + +> [!TIP] +> For more information on how to annotate your data model and what additional options are available for each attribute, see [defining your data model](../../../concepts/vector-store-connectors/defining-your-data-model.md). + +## Read the paragraphs in the document + +Next, you add the code to read the Word document and find the text of each paragraph in it. + +Add a new file to the project called `DocumentReader.cs` and add the following class to read the paragraphs from a document. + +```csharp +using System.Text; +using System.Xml; +using DocumentFormat.OpenXml.Packaging; + +namespace SKVectorIngest; + +internal class DocumentReader +{ + public static IEnumerable ReadParagraphs(Stream documentContents, string documentUri) + { + // Open the document. + using WordprocessingDocument wordDoc = WordprocessingDocument.Open(documentContents, false); + if (wordDoc.MainDocumentPart == null) + { + yield break; + } + + // Create an XmlDocument to hold the document contents + // and load the document contents into the XmlDocument. + XmlDocument xmlDoc = new(); + XmlNamespaceManager nsManager = new(xmlDoc.NameTable); + nsManager.AddNamespace("w", "http://schemas.openxmlformats.org/wordprocessingml/2006/main"); + nsManager.AddNamespace("w14", "http://schemas.microsoft.com/office/word/2010/wordml"); + + xmlDoc.Load(wordDoc.MainDocumentPart.GetStream()); + + // Select all paragraphs in the document and break if none found. + XmlNodeList? paragraphs = xmlDoc.SelectNodes("//w:p", nsManager); + if (paragraphs == null) + { + yield break; + } + + // Iterate over each paragraph. + foreach (XmlNode paragraph in paragraphs) + { + // Select all text nodes in the paragraph and continue if none found. + XmlNodeList? texts = paragraph.SelectNodes(".//w:t", nsManager); + if (texts == null) + { + continue; + } + + // Combine all non-empty text nodes into a single string. + var textBuilder = new StringBuilder(); + foreach (XmlNode text in texts) + { + if (!string.IsNullOrWhiteSpace(text.InnerText)) + { + textBuilder.Append(text.InnerText); + } + } + + // Yield a new TextParagraph if the combined text is not empty. + var combinedText = textBuilder.ToString(); + if (!string.IsNullOrWhiteSpace(combinedText)) + { + Console.WriteLine("Found paragraph:"); + Console.WriteLine(combinedText); + Console.WriteLine(); + + yield return new TextParagraph + { + Key = Guid.NewGuid().ToString(), + DocumentUri = documentUri, + ParagraphId = paragraph.Attributes?["w14:paraId"]?.Value ?? string.Empty, + Text = combinedText + }; + } + } + } +} +``` + +## Generate embeddings and upload the data + +Next, you add a new class to generate embeddings and upload the paragraphs to Redis. + +Add a new file called `DataUploader.cs` with the following contents: + +```csharp +using Microsoft.Extensions.AI; +using Microsoft.Extensions.VectorData; + +namespace SKVectorIngest; + +internal class DataUploader(VectorStore vectorStore, IEmbeddingGenerator> embeddingGenerator) +{ + /// + /// Generate an embedding for each text paragraph and upload it to the specified collection. + /// + /// The name of the collection to upload the text paragraphs to. + /// The text paragraphs to upload. + /// An async task. + public async Task GenerateEmbeddingsAndUpload(string collectionName, IEnumerable textParagraphs) + { + var collection = vectorStore.GetCollection(collectionName); + await collection.EnsureCollectionExistsAsync(); + + foreach (var paragraph in textParagraphs) + { + // Generate the text embedding. + Console.WriteLine($"Generating embedding for paragraph: {paragraph.ParagraphId}"); + paragraph.TextEmbedding = (await embeddingGenerator.GenerateEmbeddingAsync(paragraph.Text)).Vector; + + // Upload the text paragraph. + Console.WriteLine($"Upserting paragraph: {paragraph.ParagraphId}"); + await collection.UpsertAsync(paragraph); + + Console.WriteLine(); + } + } +} +``` + +## Put it all together + +Finally, you put together the different pieces. +In this example, you use standard .NET dependency injection to register the Redis vector store and the embedding generator. + +Add the following code to your `Program.cs` file to set up the container, register the Redis vector store, and register the embedding service. Replace the text embedding generation settings with your own values. + +```csharp +using Azure; +using Azure.AI.OpenAI; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.Redis; +using SKVectorIngest; + +// Replace with your values. +var deploymentName = "text-embedding-ada-002"; +var endpoint = "https://sksample.openai.azure.com/"; +var apiKey = "your-api-key"; + +// Register Azure OpenAI embedding generator and Redis vector store. +var services = new ServiceCollection(); +services.AddSingleton>>( + new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey)) + .GetEmbeddingClient(deploymentName) + .AsIEmbeddingGenerator()); + +services.AddRedisVectorStore("localhost:6379"); + +// Register the data uploader. +services.AddSingleton(); + +// Build the service provider and get the data uploader. +var serviceProvider = services.BuildServiceProvider(); +var dataUploader = serviceProvider.GetRequiredService(); +``` + +Lastly, add code to read the paragraphs from the Word document and call the data uploader +to generate the embeddings and upload the paragraphs. + +```csharp +// Load the data. +var textParagraphs = DocumentReader.ReadParagraphs( + new FileStream( + "vector-store-data-ingestion-input.docx", + FileMode.Open), + "file:///c:/vector-store-data-ingestion-input.docx"); + +await dataUploader.GenerateEmbeddingsAndUpload( + "sk-documentation", + textParagraphs); +``` + +## See your data in Redis + +Navigate to the Redis stack browser, for example, [http://localhost:8001/redis-stack/browser](http://localhost:8001/redis-stack/browser), where you should now be able to see +your uploaded paragraphs. Following is an example of what you should see for one of the uploaded paragraphs. + +```json +{ + "DocumentUri" : "file:///c:/vector-store-data-ingestion-input.docx", + "ParagraphId" : "14CA7304", + "Text" : "Version 1.0+ support across C#, Python, and Java means it’s reliable, committed to non breaking changes. Any existing chat-based APIs are easily expanded to support additional modalities like voice and video.", + "TextEmbedding" : [...] +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/hybrid-search.md b/docs/ai/vector-stores/hybrid-search.md new file mode 100644 index 0000000000000..55cefd1e66bb2 --- /dev/null +++ b/docs/ai/vector-stores/hybrid-search.md @@ -0,0 +1,243 @@ +--- +title: Hybrid search using Vector Store connectors (Preview) +description: Describes the different options you can use when doing a hybrid search using Vector Store connectors. +ms.topic: concept-article +ms.date: 03/06/2025 +--- +# Hybrid search using Vector Store connectors (Preview) + +The Microsoft.Extensions.VectorData library provides hybrid search capabilities as part of its Vector Store abstractions. This supports filtering and many other options, which this article will explain in more detail. + +Currently the type of hybrid search supported is based on a vector search, plus a keyword search, both of which are executed in parallel, after which a union of the two result sets +are returned. Sparse vector based hybrid search is not currently supported. + +To execute a hybrid search, your database schema needs to have a vector field and a string field with full text search capabilities enabled. +If you are creating a collection using the Vector Store connectors, make sure to enable the option +on the string field that you want to target for the keyword search. + +> [!TIP] +> For more information on how to enable refer to [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstoredataattribute-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md#vectorstoredataproperty-configuration-settings) + +## Hybrid search + +The method allows searching using a vector and an `ICollection` of string keywords. It also takes an optional `HybridSearchOptions` class as input. +This method is available on the following interface: + +1. `IKeywordHybridSearchable` + +Only connectors for databases that currently support vector plus keyword hybrid search are implementing this interface. + +Assuming you have a collection that already contains data, you can easily do a hybrid search on it. Here is an example using Qdrant. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +// Placeholder embedding generation method. +async Task> GenerateEmbeddingAsync(string textToVectorize) +{ + // your logic here +} + +// Create a Qdrant VectorStore object and choose an existing collection that already contains records. +VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +IKeywordHybridSearchable collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skhotels"); + +// Generate a vector for your search text, using your chosen embedding generation implementation. +ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); + +// Do the search, passing an options object with a Top value to limit results to the single top match. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 1); + +// Inspect the returned hotel. +await foreach (var record in searchResult) +{ + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); +} +``` + +> [!TIP] +> For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). + +## Supported vector types + + takes a generic type as the vector parameter. +The types of vectors supported by each data store vary. +See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. + +It's also important for the search vector type to match the target vector that is being searched, for example, if you have two vectors +on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector +you are targeting. +See [VectorProperty and AdditionalProperty](#vectorproperty-and-additionalproperty) for how to pick a target vector if you have more than one per record. + +## Hybrid search options + +The following options can be provided using the `HybridSearchOptions` class. + +### VectorProperty and AdditionalProperty + +The `VectorProperty` and `AdditionalProperty` options can be used to specify the vector property and full text search property to target during the search. + +If no `VectorProperty` is provided and the data model contains only one vector, that vector will be used. +If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. + +If no `AdditionalProperty` is provided and the data model contains only one full text search property, that property will be used. +If the data model contains no full text search property or multiple full text search properties and `AdditionalProperty` is not provided, the search method will throw. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +var collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skproducts"); + +// Create the hybrid search options and indicate that you want +// to search the DescriptionEmbedding vector property and the +// Description full text search property. +var hybridSearchOptions = new HybridSearchOptions +{ + VectorProperty = r => r.DescriptionEmbedding, + AdditionalProperty = r => r.Description +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + +public sealed class Product +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Name { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreData] + public List FeatureList { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } +} +``` + +### `Top` and `Skip` + +The `Top` and `Skip` options allow you to limit the number of results to the Top n results and +to skip a number of results from the top of the resultset. +Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. + +```csharp +// Create the vector search options and indicate that you want to skip the first 40 results and then pass 20 to search to get the next 20. +var hybridSearchOptions = new HybridSearchOptions +{ + Skip = 40 +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 20, hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.Description); +} +``` + +The default values for `Skip` is 0. + +### IncludeVectors + +The `IncludeVectors` option allows you to specify whether you wish to return vectors in the search results. +If `false`, the vector properties on the returned model will be left null. +Using `false` can significantly reduce the amount of data retrieved from the vector store during search, +making searches more efficient. + +The default value for `IncludeVectors` is `false`. + +```csharp +// Create the hybrid search options and indicate that you want to include vectors in the search results. +var hybridSearchOptions = new HybridSearchOptions +{ + IncludeVectors = true +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.FeatureList); +} +``` + +### Filter + +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection +before applying the vector search. + +This has multiple benefits: + +- Reduce latency and processing cost, since only records remaining after filtering need to be compared with the search vector and therefore fewer vector comparisons have to be done. +- Limit the resultset for for example, access control purposes, by excluding data that the user shouldn't have access to. + +Note that in order for fields to be used for filtering, many vector stores require those fields to be indexed first. +Some vector stores will allow filtering using any field, but might optionally allow indexing to improve filtering performance. + +If creating a collection via the Vector Store abstractions and you wish to enable filtering on a field, +set the property to true when defining your data model or when creating your record definition. + +> [!TIP] +> For more information on how to set the property, see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddatafield-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md). + +Filters are expressed using LINQ expressions based on the type of the data model. +The set of LINQ expressions supported will vary depending on the functionality supported +by each database, but all databases support a broad base of common expressions, for example, `equals`, `not equals`, `and`, and `or`. + +```csharp +// Create the hybrid search options and set the filter on the options. +var hybridSearchOptions = new HybridSearchOptions +{ + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.Definition); +} + +sealed class Glossary +{ + [VectorStoreKey] + public ulong Key { get; set; } + + // Category is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public string Category { get; set; } + + // Tags is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public List Tags { get; set; } + + [VectorStoreData] + public string Term { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Definition { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md new file mode 100644 index 0000000000000..654f43d28d64d --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md @@ -0,0 +1,146 @@ +--- +title: Using the Azure AI Search Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Azure AI Search. +ms.topic: concept-article +ms.date: 07/08/2024 +--- +# Using the Azure AI Search Vector Store connector (Preview) + +> [!WARNING] +> The Azure AI Search Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Azure AI Search Vector Store connector can be used to access and manage data in Azure AI Search. The connector has the following characteristics. + +| Feature Area | Support | +|-------------------------------|-----------------------| +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StorageName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | | + +## Limitations + +Notable Azure AI Search connector functionality limitations. + +| Feature Area | Workaround | +|--------------|------------| +| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | + +## Get started + +Add the Azure AI Search Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.AzureAISearch --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Azure; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); +``` + +```csharp +using Azure; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); +``` + +Extension methods that take no parameters are also provided. These require an instance of the Azure AI Search `SearchIndexClient` to be separately registered with the dependency injection container. + +```csharp +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret))); +services.AddAzureAISearchVectorStore(); +``` + +```csharp +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret))); +builder.Services.AddAzureAISearchVectorStore(); +``` + +You can construct an Azure AI Search Vector Store instance directly. + +```csharp +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.SemanticKernel.Connectors.AzureAISearch; + +var vectorStore = new AzureAISearchVectorStore( + new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret))); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.SemanticKernel.Connectors.AzureAISearch; + +var collection = new AzureAISearchCollection( + new SearchIndexClient(new Uri(azureAISearchUri), new AzureKeyCredential(secret)), + "skhotels"); +``` + +## Data mapping + +The default mapper used by the Azure AI Search connector when mapping data from the data model to storage is the one provided by the Azure AI Search SDK. + +This mapper does a direct conversion of the list of properties on the data model to the fields in Azure AI Search and uses `System.Text.Json.JsonSerializer` +to convert to the storage schema. This means that usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the +data model property name is required. + +It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` +must be passed to both the `SearchIndexClient` and the `AzureAISearchCollection` on construction. + +```csharp +var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; +var collection = new AzureAISearchCollection( + new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret), + new() { Serializer = new JsonObjectSerializer(jsonSerializerOptions) }), + "skhotels", + new() { JsonSerializerOptions = jsonSerializerOptions }); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md new file mode 100644 index 0000000000000..f3d20baaa305c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md @@ -0,0 +1,155 @@ +--- +title: Using the Azure CosmosDB MongoDB (vCore) Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Azure CosmosDB MongoDB (vCore). +ms.topic: concept-article +ms.date: 09/23/2024 +--- +# Using the Azure CosmosDB MongoDB (vCore) Vector Store connector (Preview) + +> [!WARNING] +> The Azure CosmosDB MongoDB (vCore) Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Azure CosmosDB MongoDB Vector Store connector can be used to access and manage data in Azure CosmosDB MongoDB (vCore). The connector has the following characteristics. + +| Feature Area | Support | +|------------------------------|----------------------------------------------------| +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | No | +| StorageName supported? | No, use BsonElementAttribute instead. For more information, see [Data mapping](#data-mapping). | +| HybridSearch supported? | No | + +## Limitations + +This connector is compatible with Azure Cosmos DB MongoDB (vCore) and is *not* designed to be compatible with Azure Cosmos DB MongoDB (RU). + +## Get started + +Add the Azure CosmosDB MongoDB Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.CosmosMongoDB --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddCosmosMongoVectorStore(connectionString, databaseName); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddCosmosMongoVectorStore(connectionString, databaseName); +``` + +Extension methods that take no parameters are also provided. These require an instance of `MongoDB.Driver.IMongoDatabase` to be separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using MongoDB.Driver; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => + { + var mongoClient = new MongoClient(connectionString); + return mongoClient.GetDatabase(databaseName); + }); +services.AddCosmosMongoVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using MongoDB.Driver; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => + { + var mongoClient = new MongoClient(connectionString); + return mongoClient.GetDatabase(databaseName); + }); +builder.Services.AddCosmosMongoVectorStore(); +``` + +You can construct an Azure CosmosDB MongoDB Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var vectorStore = new CosmosMongoVectorStore(database); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var collection = new CosmosMongoCollection( + database, + "skhotels"); +``` + +## Data mapping + +The Azure CosmosDB MongoDB Vector Store connector provides a default mapper when mapping data from the data model to storage. + +This mapper does a direct conversion of the list of properties on the data model to the fields in Azure CosmosDB MongoDB and uses `MongoDB.Bson.Serialization` to convert to the storage schema. This means that usage of the `MongoDB.Bson.Serialization.Attributes.BsonElement` is supported if a different storage name to the data model property name is required. The only exception is the key of the record which is mapped to a database field named `_id`, since all CosmosDB MongoDB records must use this name for ids. + +### Property name override + +For data properties and vector properties, you can provide override field names to use in storage that is different to the property names on the data model. This is not supported for keys, since a key has a fixed name in MongoDB. + +The property name override is done by setting the `BsonElement` attribute on the data model properties. + +Here is an example of a data model with `BsonElement` set. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [BsonElement("hotel_name")] + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [BsonElement("hotel_description")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [BsonElement("hotel_description_embedding")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md new file mode 100644 index 0000000000000..30648bd7785d5 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md @@ -0,0 +1,245 @@ +--- +title: Using the Azure CosmosDB NoSQL Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Azure CosmosDB NoSQL. +ms.topic: concept-article +ms.date: 09/23/2024 +--- +# Using the Azure CosmosDB NoSQL Vector Store connector (Preview) + +> [!WARNING] +> The Azure CosmosDB NoSQL Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage data in Azure CosmosDB NoSQL. The connector has the following characteristics. + +| Feature Area | Support | +|--------------------|---------------------------------| +| Collection maps to | Azure Cosmos DB NoSQL Container | +| Supported key property types |
  • string
  • CosmosNoSqlCompositeKey
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • byte[]
  • ReadOnlyMemory\
  • Embedding\
  • sbyte[]
| +| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StorageName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +## Limitations + +When initializing `CosmosClient` manually, it is necessary to specify `CosmosClientOptions.UseSystemTextJsonSerializerWithOptions` due to limitations in the default serializer. This option can be set to `JsonSerializerOptions.Default` or customized with other serializer options to meet specific configuration needs. + +```csharp +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, +}); +``` + +## Get started + +Add the Azure CosmosDB NoSQL Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.CosmosNoSql --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddCosmosNoSqlVectorStore(connectionString, databaseName); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddCosmosNoSqlVectorStore(connectionString, databaseName); +``` + +Extension methods that take no parameters are also provided. These require an instance of `Microsoft.Azure.Cosmos.Database` to be separately registered with the dependency injection container. + +```csharp +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => + { + var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() + { + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, + }); + + return cosmosClient.GetDatabase(databaseName); + }); +services.AddCosmosNoSqlVectorStore(); +``` + +```csharp +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => + { + var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() + { + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, + }); + + return cosmosClient.GetDatabase(databaseName); + }); +builder.Services.AddCosmosNoSqlVectorStore(); +``` + +You can construct an Azure CosmosDB NoSQL Vector Store instance directly. + +```csharp +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.SemanticKernel.Connectors.CosmosNoSql; + +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, +}); + +var database = cosmosClient.GetDatabase(databaseName); +var vectorStore = new CosmosNoSqlVectorStore(database); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.SemanticKernel.Connectors.CosmosNoSql; + +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, +}); + +var database = cosmosClient.GetDatabase(databaseName); +var collection = new CosmosNoSqlCollection( + database, + "skhotels"); +``` + +## Data mapping + +The Azure CosmosDB NoSQL Vector Store connector provides a default mapper when mapping from the data model to storage. + +This mapper does a direct conversion of the list of properties on the data model to the fields in Azure CosmosDB NoSQL and uses `System.Text.Json.JsonSerializer` +to convert to the storage schema. This means that usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the +data model property name is required. The only exception is the key of the record which is mapped to a database field named `id`, since all CosmosDB NoSQL +records must use this name for ids. + +It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` +must be passed to the `CosmosNoSqlCollection` on construction. + +```csharp +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.SemanticKernel.Connectors.CosmosNoSql; + +var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; + +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = jsonSerializerOptions +}); + +var database = cosmosClient.GetDatabase(databaseName); +var collection = new CosmosNoSqlCollection( + database, + "skhotels", + new() { JsonSerializerOptions = jsonSerializerOptions }); +``` + +Using the above custom `JsonSerializerOptions` which is using `SnakeCaseUpper`, the following data model will be mapped to the below json. + +```csharp +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.EuclideanDistance, IndexKind = IndexKind.QuantizedFlat)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```json +{ + "id": "1", + "HOTEL_NAME": "Hotel Happy", + "DESCRIPTION": "A place where everyone can be happy.", + "HOTEL_DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1], +} +``` + +## Using partition key + +In the Azure Cosmos DB for NoSQL connector, the partition key property defaults to the key property - `id`. The `PartitionKeyPropertyName` property in `CosmosNoSqlCollectionOptions` class allows specifying a different property as the partition key. + +The `CosmosNoSqlCollection` class supports two key types: `string` and `CosmosNoSqlCompositeKey`. The `CosmosNoSqlCompositeKey` consists of `RecordKey` and `PartitionKey`. + +If the partition key property is not set (and the default key property is used), `string` keys can be used for operations with database records. However, if a partition key property is specified, it is recommended to use `CosmosNoSqlCompositeKey` to provide both the key and partition key values. + +Specify partition key property name: + +```csharp +var options = new CosmosNoSqlCollectionOptions +{ + PartitionKeyPropertyName = nameof(Hotel.HotelName) +}; + +var collection = new CosmosNoSqlCollection(database, "collection-name", options) + as VectorStoreCollection; +``` + +Get with partition key: + +```csharp +var record = await collection.GetAsync(new CosmosNoSqlCompositeKey("hotel-id", "hotel-name")); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md new file mode 100644 index 0000000000000..a8461b71b4c61 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md @@ -0,0 +1,347 @@ +--- +title: Using the Couchbase Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Couchbase. +author: azaddhirajkumar +ms.topic: concept-article +ms.date: 11/03/2025 +--- + +# Using the Couchbase connector (Preview) + +> [!WARNING] +> The Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Couchbase Vector Store connector can be used to access and manage data in Couchbase. The connector has the +following characteristics. + +| Feature Area | Support | +|------------------------------|------------------------------| +| Collection maps to | Couchbase collection + index | +| Supported key property types |
  • `string`
| +| Supported data property types | All types that are supported by `System.Text.Json` (either built-in or by using a custom converter) | +| Supported vector property types |
  • `ReadOnlyMemory`
  • `Embedding`
  • `float[]`
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • `AnyTagEqualTo`
  • `EqualTo`
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +## Get started + +Add the Couchbase Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package CouchbaseConnector.SemanticKernel --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.SemanticKernel; +using Couchbase.SemanticKernel; + +// Using a ServiceCollection. +var kernelBuilder = Kernel + .CreateBuilder() + .AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name"); +``` + +```csharp +using Couchbase.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name"); +``` + +## Configure index type + +The vector store defaults to using Hyperscale indexes. You can specify a different index type by passing `CouchbaseVectorStoreOptions`: + +```csharp +using Couchbase.SemanticKernel; + +var builder = WebApplication.CreateBuilder(args); + +// Option 1: Use Hyperscale index +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name", + options: new CouchbaseVectorStoreOptions + { + IndexType = CouchbaseIndexType.Hyperscale + }); + +// Option 2: Use Composite index +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name", + options: new CouchbaseVectorStoreOptions + { + IndexType = CouchbaseIndexType.Composite + }); + +// Option 3: Use Search vector index +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name", + options: new CouchbaseVectorStoreOptions + { + IndexType = CouchbaseIndexType.Search + }); +``` + +Extension methods that take no parameters are also provided. These require an instance of the `IScope` class to be +separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Couchbase; +using Couchbase.KeyValue; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => +{ + var clusterOptions = new ClusterOptions + { + ConnectionString = "couchbases://your-cluster-address", + UserName = "username", + Password = "password" + }; + + return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); +}); + +services.AddSingleton(sp => +{ + var cluster = sp.GetRequiredService(); + var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); + return bucket.Scope("scope-name"); +}); + +// Add Couchbase Vector Store +services.AddCouchbaseVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Couchbase.KeyValue; +using Couchbase; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); + +builder.Services.AddSingleton(sp => +{ + var clusterOptions = new ClusterOptions + { + ConnectionString = "couchbases://your-cluster-address", + UserName = "username", + Password = "password" + }; + + return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); +}); + +builder.Services.AddSingleton(sp => +{ + var cluster = sp.GetRequiredService(); + var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); + return bucket.Scope("scope-name"); +}); + +// Add Couchbase Vector Store +builder.Services.AddCouchbaseVectorStore(); +``` + +You can construct a Couchbase Vector Store instance directly. + +```csharp +using Couchbase; +using Couchbase.KeyValue; +using Couchbase.SemanticKernel; + +var clusterOptions = new ClusterOptions +{ + ConnectionString = "couchbases://your-cluster-address", + UserName = "username", + Password = "password" +}; + +var cluster = await Cluster.ConnectAsync(clusterOptions); +var bucket = await cluster.BucketAsync("bucket-name"); +var scope = bucket.Scope("scope-name"); + +var vectorStore = new CouchbaseVectorStore(scope); +``` + +It's possible to construct a direct reference to a named collection. + +## Use query collection (Hyperscale or composite index) + +For high-performance vector search with Hyperscale indexes: + +```csharp +using Couchbase.SemanticKernel; +using Couchbase; +using Couchbase.KeyValue; + +var cluster = await Cluster.ConnectAsync(clusterOptions); +var bucket = await cluster.BucketAsync("bucket-name"); +var scope = bucket.Scope("scope-name"); + +// Using Hyperscale index (default) +var collection = new CouchbaseQueryCollection( + scope, + "skhotels", + indexType: CouchbaseIndexType.Hyperscale); + +// Or using Composite index +var collectionComposite = new CouchbaseQueryCollection( + scope, + "skhotels", + indexType: CouchbaseIndexType.Composite); +``` + +### Use search collection (seach vector index) + +For hybrid search scenarios combining full-text search: + +```csharp +using Couchbase.SemanticKernel; +using Couchbase; +using Couchbase.KeyValue; + +var cluster = await Cluster.ConnectAsync(clusterOptions); +var bucket = await cluster.BucketAsync("bucket-name"); +var scope = bucket.Scope("scope-name"); + +var collection = new CouchbaseSearchCollection( + scope, + "skhotels"); +``` + +### Index type comparison + +Couchbase offers three types of indexes for vector search: + +**Hyperscale vector indexes** + +- Best for pure vector searches - content discovery, recommendations, semantic search +- High performance with low memory footprint - designed to scale to billions of vectors +- Optimized for concurrent operations - supports simultaneous searches and inserts +- **Use when:** You primarily perform vector-only queries without complex scalar filtering +- **Ideal for:** Large-scale semantic search, recommendation systems, content discovery +- **Requires:** Couchbase Server 8.0+ or Capella + +**Composite vector indexes** + +- Best for filtered vector searches - combines vector search with scalar value filtering +- Efficient pre-filtering - scalar attributes reduce the vector comparison scope +- **Use when:** Your queries combine vector similarity with scalar filters that eliminate large portions of data +- **Ideal for:** Compliance-based filtering, user-specific searches, time-bounded queries +- **Requires:** Couchbase Server 8.0+ or Capella + +**Search vector indexes** + +- Best for hybrid searches combining full-text search with vector similarity +- Allows semantic search alongside traditional keyword matching +- Supports geospatial searches in addition to vector and text +- **Use when:** You need to combine traditional keyword search with vector similarity search in the same query +- **Ideal for:** E-commerce product search, travel recommendations, content discovery with multiple search criteria +- **Requires:** Couchbase Server 7.6+ or Capella + +#### Choose the right index type + +- Start with **Hyperscale Index** for pure vector searches and large datasets (scales to billions) +- Choose **Composite Index** when scalar filters significantly reduce your search space (works well for tens of millions to billions of vectors) +- Use **Search Vector Index** for hybrid search combining text and vectors + +[Detailed comparison of vector index types](https://docs.couchbase.com/server/current/vector-index/use-vector-indexes.html) + +## Data mapping + +The Couchbase connector will use `System.Text.Json.JsonSerializer` to do mapping. Properties in the data model are serialized into a JSON object and stored as the document value in Couchbase. + +Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. + +```csharp +using Couchbase.SemanticKernel; +using Couchbase.KeyValue; +using System.Text.Json; + +var jsonSerializerOptions = new JsonSerializerOptions +{ + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper +}; + +var options = new CouchbaseQueryCollectionOptions +{ + JsonSerializerOptions = jsonSerializerOptions +}; + +var collection = new CouchbaseQueryCollection(scope, "skhotelsjson", options); +``` + +Since a naming policy of snake case upper was chosen, here is an example of how this data type will be stored in Couchbase. Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. + +```csharp +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```json +{ + "_id" : "h1", + "HOTEL_ID" : "h1", + "HOTEL_NAME" : "Hotel Happy", + "HOTEL_DESCRIPTION" : "A place where everyone can be happy.", + "DESCRIPTION_EMBEDDING" : [ + 0.9, + 0.1, + 0.1, + 0.1 + ] +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md new file mode 100644 index 0000000000000..a254eed5a0d9c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md @@ -0,0 +1,197 @@ +--- +title: Using the Elasticsearch Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Elasticsearch. +author: flobernd +ms.topic: concept-article +ms.date: 11/04/2024 +--- +# Using the Elasticsearch connector (Preview) + +> [!WARNING] +> The Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Elasticsearch Vector Store connector can be used to access and manage data in Elasticsearch. The connector has the following characteristics. + +| Feature area | Support | +|--------------------|---------------------| +| Collection maps to | Elasticsearch index | +| Supported key property types |
  • `string`
  • `long`
  • `Guid`
| +| Supported data property types | All types that are supported by `System.Text.Json` (etiher built-in or by using a custom converter) | +| Supported vector property types |
  • `ReadOnlyMemory`
  • `IEnumerable`
  • `IReadOnlyCollection`
  • `ICollection`
  • `IReadOnlyList`
  • `IList`
  • `float[]`
| +| Supported index types |
  • HNSW (32, 8, or 4 bit or BBQ)
  • FLAT (32, 8, or 4 bit or BBQ)
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • MaxInnerProduct
| +| Supported filter clauses |
  • `AnyTagEqualTo`
  • `EqualTo`
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +## Get started + +To [run Elasticsearch locally](https://www.elastic.co/guide/en/elasticsearch/reference/current/run-elasticsearch-locally.html) for local development or testing run the `start-local` script with one command: + +```bash +curl -fsSL https://elastic.co/start-local | sh +``` + +Add the Elasticsearch Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Elastic.SemanticKernel.Connectors.Elasticsearch --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using a ServiceCollection. +var kernelBuilder = Kernel + .CreateBuilder() + .AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); +``` + +```csharp +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); +``` + +Extension methods that take no parameters are also provided. These require an instance of the `Elastic.Clients.Elasticsearch.ElasticsearchClient` class to be separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); +services.AddElasticsearchVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); +builder.Services.AddElasticsearchVectorStore(); +``` + +You can construct an Elasticsearch Vector Store instance directly. + +```csharp +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; + +var vectorStore = new ElasticsearchVectorStore( + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; + +var collection = new ElasticsearchVectorStoreRecordCollection( + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))), + "skhotels"); +``` + +## Data mapping + +The Elasticsearch connector will use `System.Text.Json.JsonSerializer` to do mapping. +Since Elasticsearch stores documents with a separate key/id and value, the mapper will serialize all properties except for the key to a JSON object +and use that as the value. + +Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the +data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, +a custom source serializer must be configured. + +```csharp +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; +using Elastic.Clients.Elasticsearch.Serialization; +using Elastic.Transport; + +var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); +var settings = new ElasticsearchClientSettings( + nodePool, + sourceSerializer: (defaultSerializer, settings) => + new DefaultSourceSerializer(settings, options => + options.PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper)); +var client = new ElasticsearchClient(settings); + +var collection = new ElasticsearchVectorStoreRecordCollection( + client, + "skhotelsjson"); +``` + +As an alternative, the `DefaultFieldNameInferrer` lambda function can be configured to achieve the same result or to even further customize property naming based on dynamic conditions. + +```csharp +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; + +var settings = new ElasticsearchClientSettings(new Uri("http://localhost:9200")); +settings.DefaultFieldNameInferrer(name => JsonNamingPolicy.SnakeCaseUpper.ConvertName(name)); +var client = new ElasticsearchClient(settings); + +var collection = new ElasticsearchVectorStoreRecordCollection( + client, + "skhotelsjson"); +``` + +Since a naming policy of snake case upper was chosen, here is an example of how this data type will be set in Elasticsearch. +Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. + +```csharp +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity, IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```json +{ + "_index" : "skhotelsjson", + "_id" : "h1", + "_source" : { + "HOTEL_NAME" : "Hotel Happy", + "HOTEL_DESCRIPTION" : "A place where everyone can be happy.", + "DESCRIPTION_EMBEDDING" : [ + 0.9, + 0.1, + 0.1, + 0.1 + ] + } +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md new file mode 100644 index 0000000000000..8f9b6616bed2d --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md @@ -0,0 +1,76 @@ +--- +title: Using the In-Memory Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in an in-memory vector store. +ms.topic: concept-article +ms.date: 11/10/2024 +--- +# Using the In-Memory connector (Preview) + +> [!WARNING] +> The In-Memory Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The In-Memory Vector Store connector is a Vector Store implementation that uses no external database and stores data in memory. +This Vector Store is useful for prototyping scenarios or where high-speed in-memory operations are required. + +The connector has the following characteristics. + +| Feature Area | Support | +|-------------------------------|-------------------------------| +| Collection maps to | In-memory dictionary | +| Supported key property types | Any type that can be compared | +| Supported data property types | Any type | +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types | Flat | +| Supported distance functions |
  • CosineSimilarity
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StorageName supported? | No, since storage is in-memory and data reuse is therefore not possible, custom naming is not applicable. | +| HybridSearch supported? | No | + +## Get started + +Add the Semantic Kernel Core NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.InMemory --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddInMemoryVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddInMemoryVectorStore(); +``` + +You can construct an InMemory Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.InMemory; + +var vectorStore = new InMemoryVectorStore(); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.InMemory; + +var collection = new InMemoryCollection("skhotels"); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md new file mode 100644 index 0000000000000..ba56bfb7d0859 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md @@ -0,0 +1,131 @@ +--- +title: Using the MongoDB Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in MongoDB. +author: dmytrostruk +ms.topic: concept-article +ms.author: dmytrostruk +ms.date: 10/25/2024 +--- +# Using the MongoDB Vector Store connector (Preview) + +> [!WARNING] +> The MongoDB Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The MongoDB Vector Store connector can be used to access and manage data in MongoDB. The connector has the following characteristics. + +| Feature Area | Support | +|--|--| +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types | N/A | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | No | +| StorageName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +## Get started + +Add the MongoDB Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.MongoDB --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMongoVectorStore(connectionString, databaseName); +``` + +Extension methods that take no parameters are also provided. These require an instance of `MongoDB.Driver.IMongoDatabase` to be separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using MongoDB.Driver; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => + { + var mongoClient = new MongoClient(connectionString); + return mongoClient.GetDatabase(databaseName); + }); +builder.Services.AddMongoVectorStore(); +``` + +You can construct a MongoDB Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.MongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var vectorStore = new MongoVectorStore(database); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.MongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var collection = new MongoCollection( + database, + "skhotels"); +``` + +## Data mapping + +The MongoDB Vector Store connector provides a default mapper when mapping data from the data model to storage. + +This mapper does a direct conversion of the list of properties on the data model to the fields in MongoDB and uses `MongoDB.Bson.Serialization` +to convert to the storage schema. This means that usage of the `MongoDB.Bson.Serialization.Attributes.BsonElement` is supported if a different storage name to the +data model property name is required. The only exception is the key of the record which is mapped to a database field named `_id`, since all MongoDB +records must use this name for ids. + +### Property name override + +For data properties and vector properties, you can provide override field names to use in storage that is different to the +property names on the data model. This is not supported for keys, since a key has a fixed name in MongoDB. + +The property name override is done by setting the `BsonElement` attribute on the data model properties. + +Here is an example of a data model with `BsonElement` set. + +```csharp +using Microsoft.Extensions.VectorData; +using MongoDB.Bson.Serialization.Attributes; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [BsonElement("hotel_name")] + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [BsonElement("hotel_description")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [BsonElement("hotel_description_embedding")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md new file mode 100644 index 0000000000000..f8aedecab0eb6 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md @@ -0,0 +1,225 @@ +--- +title: Using the Oracle Database Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Oracle Database. +author: minal-agashe-oracle +ms.topic: article +ms.date: 08/14/2025 +--- +# Using the Oracle Database Vector Store connector (Preview) + +> [!WARNING] +> The Oracle Database Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Oracle Database Vector Store Connector can be used to access and manage data in Oracle Database. The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Oracle database table | +| Supported key property types |
  • short
  • int
  • long
  • string
  • Guid
| +| Supported data property types |
  • bool
  • byte
  • short
  • int
  • decimal
  • long
  • float
  • double
  • DateTime
  • DateTimeOffset
  • TimeSpan
  • char
  • char[]
  • byte[]
  • String
  • Guid
  • *and nullable type of the above types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • double[]
  • ReadOnlyMemory\
  • Embedding\
  • short[]
  • ReadOnlyMemory\
  • Embedding\
  • byte[]
  • BitArray
  • BinaryEmbedding
| +| Supported index types |
  • Flat (default)
  • HNSW
  • IVF
| +| Supported distance functions |
  • CosineDistance
    • FLOAT32, FLOAT64, and INT8 vector default
  • CosineSimilarity
  • DotProductSimilarity
  • NegativeDotProductSimilarity
  • EuclideanDistance
  • EuclideanSquaredDistance
  • HammingDistance
    • BINARY vector default
  • ManhattanDistance
  • JaccardSimilarity
    To use Jaccard similarity, set the DistanceFunction string to "JACCARD" or "JACCARDSIMILARITY" (for example, DistanceFunction = "JACCARDSIMILARITY"). This value is case sensitive. Jaccard similarity requires BINARY numeric format vectors.
| +| Supported filter clauses |
  • ==
  • !=
  • <
  • <=
  • >
  • >=
  • List.Contains()
    • Only when checking if the model property is in the list
| +| Supports zero, one, or multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | No | +| StorageName supported? | Yes | +| HybridSearch supported? | No | + +> [!IMPORTANT] +> Vector data searches require Oracle Database 23ai or higher. All other Oracle connector features are available using Oracle Database 19c or higher. + +## Get started + +Add the Oracle Database Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Oracle.VectorData --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. In this case, an instance of the `Oracle.VectorData.OracleVectorStore` class also gets registered with the container. + +```csharp +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddOracleVectorStore(""); +``` + +```csharp +using Microsoft.AspNetCore.Builder; +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddOracleVectorStore(""); +``` + +Extension methods that take no parameters are also available. These require an instance of the `Oracle.ManagedDataAccess.Client.OracleDataSource` class to be separately registered with the dependency injection container. + +```csharp +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; +using Oracle.ManagedDataAccess.Client; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => +{ + OracleDataSourceBuilder dataSourceBuilder = new(""); + return dataSourceBuilder.Build(); +}); + +services.AddOracleVectorStore(); +``` + +```csharp +using Microsoft.AspNetCore.Builder; +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; +using Oracle.ManagedDataAccess.Client; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => +{ + OracleDataSourceBuilder dataSourceBuilder = new(""); + return dataSourceBuilder.Build(); +}); + +builder.Services.AddOracleVectorStore(); +``` + +You can construct an Oracle Database Vector Store instance directly with a custom data source or with a connection string. + +```csharp +using Oracle.VectorData; +using Oracle.ManagedDataAccess.Client; + +OracleDataSourceBuilder dataSourceBuilder = new(""); +var dataSource = dataSourceBuilder.Build(); + +var connection = new OracleVectorStore(dataSource); +``` + +```csharp +using Oracle.VectorData; + +var connection = new OracleVectorStore(""); +``` + +It's possible to construct a direct reference to a named collection with a custom data source or with a connection string. + +```csharp +using Oracle.VectorData; +using Oracle.ManagedDataAccess.Client; + +OracleDataSourceBuilder dataSourceBuilder = new(""); +var dataSource = dataSourceBuilder.Build(); + +var collection = new OracleCollection(dataSource, "skhotels"); +``` + +```csharp +using Oracle.VectorData; + +var collection = new OracleCollection("", "skhotels"); +``` + +## Data mapping + +The Oracle Database Vector Store connector provides a default mapper when mapping data from the data model to storage. This mapper does a direct conversion of the data model properties list to the Oracle database columns to convert to the storage schema. + +The Oracle Database Vector Store connector supports data model annotations and record definitions.Using annotations, the information can be provided to the data model for creating indexes and database column mapping. Using [record definitions](../schema-with-record-definition.md), the information can be defined and supplied separately from the data model. + +The following table shows the default primary key data type mapping between Oracle Database and C#: + +| C# Data Type | Database Type | +|--------------|-----------------| +| short/int16 | NUMBER(5) | +| int/int32 | NUMBER(10) | +| long/int64 | NUMBER(19) | +| string | NVARCHAR2(2000) | +| Guid | RAW(16) | + +The following table shows the default data property type mapping, including nullable types: + +| C# data type | Database type | +|------------------|--------------------------------| +| `bool` | `BOOLEAN` for Oracle Database 23ai and higher
`NUMBER(1)` for earlier versions | +| `byte` | `NUMBER(3)` | +| `short`/`int16` | `NUMBER(5)` | +| `int`/`int32` | `NUMBER(10)` | +| `decimal` | `NUMBER(18,2)` | +| `long`/`int64` | `NUMBER(19)` | +| `float` | `BINARY_FLOAT` | +| `double` | `BINARY_DOUBLE` | +| `DateTime` | `TIMESTAMP(7)` | +| `DateTimeOffset` | `TIMESTAMP(7) WITH TIME ZONE` | +| `TimeSpan` | `INTERVAL DAY(8) TO SECOND(7)` | +| `char` | `NVARCHAR2(1)` | +| `char[]` | `NVARCHAR2(2000)` | +| `byte[]` | `RAW(2000)` | +| `string` | `NVARCHAR2(2000)` | +| `Guid` | `RAW(16)` | + +Starting with Oracle Database 23ai, database vectors can be mapped to .NET data types. Multiple vector columns are supported. The following table shows the default vector property type mapping, including nullable types: + +| C# data type | Database type | +|--|--| +|
  • ReadOnlyMemory\
  • Embedding\
  • BinaryEmbedding
  • Embedding\
  • byte[]
  • System.Byte[]
  • BitArray
| VECTOR(dimensions, BINARY) | +|
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • Embedding\
  • Embedding\
  • short[]
  • System.Int16[]
| VECTOR(dimensions, INT8) | +|
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • Embedding\
  • Embedding\
  • double[]
  • System.Double[]
| VECTOR(dimensions, FLOAT64) | +|
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • Embedding\
  • Embedding\
  • float[]
  • System.Float[]
| VECTOR(dimensions, FLOAT32) | + +### Property name override + +For data properties and vector properties, you can override names to use in storage that are different from the data model property names. The property name override occurs when setting the option either in the data model properties or record definition. + +Here is a data model with set code sample and how that will be represented in an Oracle SQL command. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public long HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string? HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 384, DistanceFunction = DistanceFunction.CosineDistance)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```SQL +CREATE TABLE "MYSCHEMA"."Hotels" + ("HotelId" NUMBER(10), + "hotel_name" NVARCHAR2(2000), + "hotel_description" NVARCHAR2(2000), + "DescriptionEmbedding" VECTOR(384, FLOAT32), + PRIMARY KEY ( "HotelId" ) +); +``` + +## See also + +To learn more, see the following Oracle Database Vector Store connector resources: + +- [Introducing the Oracle Database Vector Store Connector for Semantic Kernel](https://medium.com/oracledevs/announcing-the-oracle-database-vector-store-connector-for-semantic-kernel-adb83e806d4e) +Describes key connector features, classes, and guides the reader through a sample AI vector search application using the connector. +- [Documentation: Oracle Database Vector Store Connector Classes for Semantic Kernel (.NET) APIs](https://docs.oracle.com/en/database/oracle/oracle-database/23/odpnt/VSConnector4SKClasses.html) +Contains information on Oracle Database Vector Store connector classes for adding data, retrieving data, and performing vector search in the Oracle vector database. +- [Documentation: Oracle Data Provider for .NET](https://docs.oracle.com/en/database/oracle/oracle-database/23/odpnt/intro.html) +Contains information on Oracle Data Provider for .NET (ODP.NET), the ADO.NET data provider for Oracle Database Vector Store connector. \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md new file mode 100644 index 0000000000000..7c62105cc0b51 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md @@ -0,0 +1,168 @@ +--- +title: Using the Pinecone Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Pinecone. +ms.topic: concept-article +ms.date: 07/08/2024 +--- +# Using the Pinecone connector (Preview) + +> [!WARNING] +> The Pinecone Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Pinecone Vector Store connector can be used to access and manage data in Pinecone. The connector has the following characteristics. + +| Feature Area | Support | +|------------------------------|---------------------------| +| Collection maps to | Pinecone serverless Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • decimal
  • *enumerables of type* string
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types | PGA (Pinecone Graph Algorithm) | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | No | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | No | +| StorageName supported? | Yes | +| HybridSearch supported? | No | +| Integrated Embeddings supported? | No | + +## Get started + +Add the Pinecone Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.Pinecone --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddPineconeVectorStore(pineconeApiKey); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddPineconeVectorStore(pineconeApiKey); +``` + +Extension methods that take no parameters are also provided. These require an instance of the `PineconeClient` to be separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using PineconeClient = Pinecone.PineconeClient; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => new PineconeClient(pineconeApiKey)); +services.AddPineconeVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using PineconeClient = Pinecone.PineconeClient; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => new PineconeClient(pineconeApiKey)); +builder.Services.AddPineconeVectorStore(); +``` + +You can construct a Pinecone Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.Pinecone; +using PineconeClient = Pinecone.PineconeClient; + +var vectorStore = new PineconeVectorStore( + new PineconeClient(pineconeApiKey)); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.Pinecone; +using PineconeClient = Pinecone.PineconeClient; + +var collection = new PineconeCollection( + new PineconeClient(pineconeApiKey), + "skhotels"); +``` + +## Index namespace + +The Vector Store abstraction does not support a multi tiered record grouping mechanism. Collections in the abstraction map to a Pinecone serverless index +and no second level exists in the abstraction. Pinecone does support a second level of grouping called namespaces. + +By default the Pinecone connector will pass null as the namespace for all operations. However it is possible to pass a single namespace to the +Pinecone collection when constructing it and use this instead for all operations. + +```csharp +using Microsoft.SemanticKernel.Connectors.Pinecone; +using PineconeClient = Pinecone.PineconeClient; + +var collection = new PineconeCollection( + new PineconeClient(pineconeApiKey), + "skhotels", + new() { IndexNamespace = "seasidehotels" }); +``` + +## Data mapping + +The Pinecone connector provides a default mapper when mapping data from the data model to storage. +Pinecone requires properties to be mapped into id, metadata and values groupings. +The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping. + +- The data model property annotated as a key will be mapped to the Pinecone id property. +- The data model properties annotated as data will be mapped to the Pinecone metadata object. +- The data model property annotated as a vector will be mapped to the Pinecone vector property. + +### Property name override + +For data properties, you can provide override field names to use in storage that is different to the +property names on the data model. This is not supported for keys, since a key has a fixed name in Pinecone. +It's also not supported for vectors, since the vector is stored under a fixed name `values`. +The property name override is done by setting the option via the data model attributes or record definition. + +Here is an example of a data model with set on its attributes and how that will be represented in Pinecone. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```json +{ + "id": "h1", + "values": [0.9, 0.1, 0.1, 0.1], + "metadata": { "hotel_name": "Hotel Happy", "hotel_description": "A place where everyone can be happy." } +} +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md new file mode 100644 index 0000000000000..4170a219471a0 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md @@ -0,0 +1,327 @@ +--- +title: Using the Postgres Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Postgres. +ms.topic: concept-article +ms.date: 10/24/2024 +--- +# Using the Postgres Vector Store connector (Preview) + +> [!WARNING] +> The Postgres Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Postgres Vector Store connector can be used to access and manage data in Postgres and also supports [Neon Serverless Postgres](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/neon1722366567200.neon_serverless_postgres_azure_prod). + +The connector has the following characteristics. + +| Feature Area | Support | +|--|--| +| Collection maps to | Postgres table | +| Supported key property types |
  • short
  • int
  • long
  • string
  • Guid
| +| Supported data property types |
  • bool
  • short
  • int
  • long
  • float
  • double
  • decimal
  • string
  • DateTime
  • DateTimeOffset
  • Guid
  • byte[]
  • bool Enumerables
  • short Enumerables
  • int Enumerables
  • long Enumerables
  • float Enumerables
  • double Enumerables
  • decimal Enumerables
  • string Enumerables
  • DateTime Enumerables
  • DateTimeOffset Enumerables
  • Guid Enumerables
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • Half[]
  • BitArray
  • Pgvector.SparseVector
| +| Supported index types | Hnsw | +| Supported distance functions |
  • CosineDistance
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | No | +| IsFullTextIndexed supported? | No | +| StorageName supported? | Yes | +| HybridSearch supported? | No | + +## Limitations + +> [!IMPORTANT] +> When initializing `NpgsqlDataSource` manually, it is necessary to call `UseVector` on the `NpgsqlDataSourceBuilder`. This enables vector support. Without this, usage of the VectorStore implementation will fail. + +Here is an example of how to call `UseVector`. + +```csharp +NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +dataSourceBuilder.UseVector(); +NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +``` + +When using the `AddPostgresVectorStore` dependency injection registration method with a connection string, the datasource will be constructed by this method and will automatically have `UseVector` applied. + +## Get started + +Add the Postgres Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.PgVector --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; + +var services = new ServiceCollection(); +services.AddPostgresVectorStore(""); +``` + +Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects, for example `Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres`. + +Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that `UseVector` must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddPostgresVectorStore(""); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Npgsql; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => +{ + NpgsqlDataSourceBuilder dataSourceBuilder = new(""); + dataSourceBuilder.UseVector(); + return dataSourceBuilder.Build(); +}); +builder.Services.AddPostgresVectorStore(); +``` + +You can construct a Postgres Vector Store instance directly with a custom data source or with a connection string. + +```csharp +using Microsoft.SemanticKernel.Connectors.PgVector; +using Npgsql; + +NpgsqlDataSourceBuilder dataSourceBuilder = new(""); +dataSourceBuilder.UseVector(); +NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); +``` + +```csharp +using Microsoft.SemanticKernel.Connectors.PgVector; + +var connection = new PostgresVectorStore(""); +``` + +It's possible to construct a direct reference to a named collection with a custom data source or with a connection string. + +```csharp +using Microsoft.SemanticKernel.Connectors.PgVector; +using Npgsql; + +NpgsqlDataSourceBuilder dataSourceBuilder = new(""); +dataSourceBuilder.UseVector(); +var dataSource = dataSourceBuilder.Build(); + +var collection = new PostgresCollection(dataSource, "skhotels", ownsDataSource: true); +``` + +```csharp +using Microsoft.SemanticKernel.Connectors.PgVector; + +var collection = new PostgresCollection("", "skhotels"); +``` + +## Data mapping + +The Postgres connector provides a default mapper when mapping data from the data model to storage. +The default mapper uses the model annotations or record definition to determine the type of each property and to map the model +into a Dictionary that can be serialized to Postgres. + +- The data model property annotated as a key will be mapped to the PRIMARY KEY in the Postgres table. +- The data model properties annotated as data will be mapped to a table column in Postgres. +- The data model properties annotated as vectors will be mapped to a table column that has the pgvector `VECTOR` type in Postgres. + +### Property name override + +You can provide override field names to use in storage that is different from the +property names on the data model. This allows you to match table column names even +if they don't match the property names on the data model. + +The property name override is done by setting the option via the data model attributes or record definition. + +Here is an example of a data model with set on its attributes and how it will be represented in Postgres as a table, assuming the Collection name is `Hotels`. + +```csharp +using System; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey(StorageName = "hotel_id")] + public int HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```sql +CREATE TABLE IF NOT EXISTS public."Hotels" ( + "hotel_id" INTEGER PRIMARY KEY NOT NULL, + hotel_name TEXT, + hotel_description TEXT, + hotel_description_embedding VECTOR(4) +); +``` + +## Vector indexing + +The `hotel_description_embedding` in the above `Hotel` model is a vector property with `IndexKind.HNSW` +indexing. This index will be created automatically when the collection is created. +HNSW is the only index type supported for index creation. IVFFlat index building requires that data already +exist in the table at index creation time, and so it is not appropriate for the creation of an empty table. +You are free to create and modify indexes on tables outside of the connector, which will +be used by the connector when performing queries. + +## Use with Entra authentication + +Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](/azure/postgresql/flexible-server/concepts-azure-ad-authentication). +This removes the need to store a username and password in your connection string. +To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following Npgsql extension method and set a connection string that does not have a username or password: + +```csharp +using System.Text; +using System.Text.Json; +using Azure.Core; +using Azure.Identity; +using Npgsql; + +namespace Program; + +public static class NpgsqlDataSourceBuilderExtensions +{ + private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new(["https://ossrdbms-aad.database.windows.net/.default"]); + + public static NpgsqlDataSourceBuilder UseEntraAuthentication(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default) + { + credential ??= new DefaultAzureCredential(); + + if (dataSourceBuilder.ConnectionStringBuilder.Username == null) + { + var token = credential.GetToken(s_azureDBForPostgresTokenRequestContext, default); + SetUsernameFromToken(dataSourceBuilder, token.Token); + } + + SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); + + return dataSourceBuilder; + } + + public static async Task UseEntraAuthenticationAsync(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default, CancellationToken cancellationToken = default) + { + credential ??= new DefaultAzureCredential(); + + if (dataSourceBuilder.ConnectionStringBuilder.Username == null) + { + var token = await credential.GetTokenAsync(s_azureDBForPostgresTokenRequestContext, cancellationToken).ConfigureAwait(false); + SetUsernameFromToken(dataSourceBuilder, token.Token); + } + + SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); + + return dataSourceBuilder; + } + + private static void SetPasswordProvider(NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential credential, TokenRequestContext tokenRequestContext) + { + dataSourceBuilder.UsePasswordProvider(_ => + { + var token = credential.GetToken(tokenRequestContext, default); + return token.Token; + }, async (_, ct) => + { + var token = await credential.GetTokenAsync(tokenRequestContext, ct).ConfigureAwait(false); + return token.Token; + }); + } + + private static void SetUsernameFromToken(NpgsqlDataSourceBuilder dataSourceBuilder, string token) + { + var username = TryGetUsernameFromToken(token); + + if (username != null) + { + dataSourceBuilder.ConnectionStringBuilder.Username = username; + } + else + { + throw new Exception("Could not determine username from token claims"); + } + } + + private static string? TryGetUsernameFromToken(string jwtToken) + { + // Split the token into its parts (Header, Payload, Signature) + var tokenParts = jwtToken.Split('.'); + if (tokenParts.Length != 3) + { + return null; + } + + // The payload is the second part, Base64Url encoded + var payload = tokenParts[1]; + + // Add padding if necessary + payload = AddBase64Padding(payload); + + // Decode the payload from Base64Url + var decodedBytes = Convert.FromBase64String(payload); + var decodedPayload = Encoding.UTF8.GetString(decodedBytes); + + // Parse the decoded payload as JSON + var payloadJson = JsonSerializer.Deserialize(decodedPayload); + + // Try to get the username from 'upn', 'preferred_username', or 'unique_name' claims + if (payloadJson.TryGetProperty("upn", out var upn)) + { + return upn.GetString(); + } + else if (payloadJson.TryGetProperty("preferred_username", out var preferredUsername)) + { + return preferredUsername.GetString(); + } + else if (payloadJson.TryGetProperty("unique_name", out var uniqueName)) + { + return uniqueName.GetString(); + } + + return null; + } + + private static string AddBase64Padding(string base64) => (base64.Length % 4) switch + { + 2 => base64 + "==", + 3 => base64 + "=", + _ => base64, + }; +} +``` + +Now you can use the `UseEntraAuthentication` method to set up the connection string for the Postgres connector: + +```csharp +using Microsoft.SemanticKernel.Connectors.Postgres; + +var connectionString = "Host=mydb.postgres.database.azure.com;Port=5432;Database=postgres;SSL Mode=Require;"; // No Username or Password +var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); +dataSourceBuilder.UseEntraAuthentication(); +dataSourceBuilder.UseVector(); +var dataSource = dataSourceBuilder.Build(); + +var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); +``` + +By default, the `UseEntraAuthentication` method uses the [DefaultAzureCredential](/dotnet/api/azure.identity.defaultazurecredential) to authenticate with Azure AD. You can also provide a custom `TokenCredential` implementation if needed. \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md new file mode 100644 index 0000000000000..f523600f15dae --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md @@ -0,0 +1,224 @@ +--- +title: Using the Qdrant Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Qdrant. +ms.topic: concept-article +ms.date: 07/08/2024 +--- +# Using the Qdrant connector (Preview) + +> [!WARNING] +> The Qdrant Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Qdrant Vector Store connector can be used to access and manage data in Qdrant. The connector has the following characteristics. + +| Feature Area | Support | +|------------------------------|-------------------------------------------------------------------| +| Collection maps to | Qdrant collection with payload indices for filterable data fields | +| Supported key property types |
  • ulong
  • Guid
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types | Hnsw | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes (configurable) | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StorageName supported? | Yes | +| HybridSearch supported? | Yes | + +## Get started + +Add the Qdrant Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.Qdrant --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddQdrantVectorStore("localhost"); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddQdrantVectorStore("localhost"); +``` + +Extension methods that take no parameters are also provided. These require an instance of the `Qdrant.Client.QdrantClient` class to be separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Qdrant.Client; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => new QdrantClient("localhost")); +services.AddQdrantVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Qdrant.Client; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => new QdrantClient("localhost")); +builder.Services.AddQdrantVectorStore(); +``` + +You can construct a Qdrant Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +var collection = new QdrantCollection( + new QdrantClient("localhost"), + "skhotels", + ownsClient: true); +``` + +## Data mapping + +The Qdrant connector provides a default mapper when mapping data from the data model to storage. +Qdrant requires properties to be mapped into id, payload and vector(s) groupings. +The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping. + +- The data model property annotated as a key will be mapped to the Qdrant point id. +- The data model properties annotated as data will be mapped to the Qdrant point payload object. +- The data model properties annotated as vectors will be mapped to the Qdrant point vector object. + +### Property name override + +For data properties and vector properties (if using named vectors mode), you can provide override field names to use in storage that is different to the +property names on the data model. This is not supported for keys, since a key has a fixed name in Qdrant. It's also not supported for vectors in *single +unnamed vector* mode, since the vector is stored under a fixed name. + +The property name override is done by setting the option via the data model attributes or record definition. + +Here is an example of a data model with set on its attributes and how that will be represented in Qdrant. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```json +{ + "id": 1, + "payload": { "hotel_name": "Hotel Happy", "hotel_description": "A place where everyone can be happy." }, + "vector": { + "hotel_description_embedding": [0.9, 0.1, 0.1, 0.1], + } +} +``` + +### Qdrant vector modes + +Qdrant supports two modes for vector storage and the Qdrant Connector with default mapper supports both modes. +The default mode is *single unnamed vector*. + +#### Single unnamed vector + +With this option a collection might only contain a single vector and it will be unnamed in the storage model in Qdrant. +Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: + +```csharp +new Hotel +{ + HotelId = 1, + HotelName = "Hotel Happy", + Description = "A place where everyone can be happy.", + DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } +}; +``` + +```json +{ + "id": 1, + "payload": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." }, + "vector": [0.9, 0.1, 0.1, 0.1] +} +``` + +#### Named vectors + +If using the named vectors mode, it means that each point in a collection might contain more than one vector, and each will be named. +Here is an example of how an object is represented in Qdrant when using *named vectors* mode: + +```csharp +new Hotel +{ + HotelId = 1, + HotelName = "Hotel Happy", + Description = "A place where everyone can be happy.", + HotelNameEmbedding = new float[4] { 0.9f, 0.5f, 0.5f, 0.5f } + DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } +}; +``` + +```json +{ + "id": 1, + "payload": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." }, + "vector": { + "HotelNameEmbedding": [0.9, 0.5, 0.5, 0.5], + "DescriptionEmbedding": [0.9, 0.1, 0.1, 0.1], + } +} +``` + +To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. +The same options can also be passed to any of the provided dependency injection container extension methods. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + ownsClient: true, + new() { HasNamedVectors = true }); + +var collection = new QdrantCollection( + new QdrantClient("localhost"), + "skhotels", + ownsClient: true, + new() { HasNamedVectors = true }); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md new file mode 100644 index 0000000000000..5b8bbac4ebe58 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md @@ -0,0 +1,239 @@ +--- +title: Using the Redis Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Redis. +ms.topic: concept-article +ms.date: 07/08/2024 +--- +# Using the Redis connector (Preview) + +> [!WARNING] +> The Redis Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Redis Vector Store connector can be used to access and manage data in Redis. The connector supports both Hashes and JSON modes and which mode you pick will determine what other features are supported. + +The connector has the following characteristics. + +| Feature Area | Support | +|--|--| +| Collection maps to | Redis index with prefix set to `:` | +| Supported key property types | string | +| Supported data property types | **When using Hashes:**
  • string
  • int
  • uint
  • long
  • ulong
  • double
  • float
  • bool
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • double[]
| +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StorageName supported? | **When using Hashes:** Yes
**When using JSON:** No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | + +## Get started + +Add the Redis Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.Redis --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddRedisVectorStore("localhost:6379"); +``` + +```csharp +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddRedisVectorStore("localhost:6379"); +``` + +Extension methods that take no parameters are also provided. These require an instance of the Redis `IDatabase` to be separately registered with the dependency injection container. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using StackExchange.Redis; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); +services.AddRedisVectorStore(); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using StackExchange.Redis; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); +builder.Services.AddRedisVectorStore(); +``` + +You can construct a Redis Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +var vectorStore = new RedisVectorStore(ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); +``` + +It's possible to construct a direct reference to a named collection. +When doing so, you have to choose between the JSON or Hashes instance depending on how you wish to store data in Redis. + +```csharp +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +// Using Hashes. +var hashesCollection = new RedisHashSetCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelshashes"); +``` + +```csharp +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +// Using JSON. +var jsonCollection = new RedisJsonCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelsjson"); +``` + +When constructing a `RedisVectorStore` or registering it with the dependency injection container, it's possible to pass a `RedisVectorStoreOptions` instance +that configures the preferred storage type / mode used: Hashes or JSON. If not specified, the default is JSON. + +```csharp +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +var vectorStore = new RedisVectorStore( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + new() { StorageType = RedisStorageType.HashSet }); +``` + +## Index prefixes + +Redis uses a system of key prefixing to associate a record with an index. +When creating an index you can specify one or more prefixes to use with that index. +If you want to associate a record with that index, you have to add the prefix to the key of that record. + +For example, If you create a index called `skhotelsjson` with a prefix of `skhotelsjson:`, when setting a record +with key `h1`, the record key will need to be prefixed like this `skhotelsjson:h1` to be added to the index. + +When creating a new collection using the Redis connector, the connector will create an index in Redis with a +prefix consisting of the collection name and a colon, like this `:`. +By default, the connector will also prefix all keys with the this prefix when doing record operations like Get, Upsert, and Delete. + +If you didn't want to use a prefix consisting of the collection name and a colon, it is possible to switch +off the prefixing behavior and pass in the fully prefixed key to the record operations. + +```csharp +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +var collection = new RedisJsonCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelsjson", + new() { PrefixCollectionNameToKeyNames = false }); + +await collection.GetAsync("myprefix_h1"); +``` + +## Data mapping + +Redis supports two modes for storing data: JSON and Hashes. The Redis connector supports both storage types, and mapping differs depending on the chosen storage type. + +### Data mapping when using the JSON storage type + +When using the JSON storage type, the Redis connector will use `System.Text.Json.JsonSerializer` to do mapping. +Since Redis stores records with a separate key and value, the mapper will serialize all properties except for the key to a JSON object +and use that as the value. + +Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the +data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` +must be passed to the `RedisJsonCollection` on construction. + +```csharp +var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; +var collection = new RedisJsonCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelsjson", + new() { JsonSerializerOptions = jsonSerializerOptions }); +``` + +Since a naming policy of snake case upper was chosen, here is an example of how this data type will be set in Redis. +Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. + +```csharp +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```redis +JSON.SET skhotelsjson:h1 $ '{ "HOTEL_NAME": "Hotel Happy", "HOTEL_DESCRIPTION": "A place where everyone can be happy.", "DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1] }' +``` + +### Data mapping when using the Hashes storage type + +When using the Hashes storage type, the Redis connector provides its own mapper to do mapping. +This mapper will map each property to a field-value pair as supported by the Redis `HSET` command. + +For data properties and vector properties, you can provide override field names to use in storage that is different to the +property names on the data model. This is not supported for keys, since keys cannot be named in Redis. + +Property name overriding is done by setting the option via the data model attributes or record definition. + +Here is an example of a data model with set on its attributes and how these are set in Redis. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```redis +HSET skhotelshashes:h1 hotel_name "Hotel Happy" hotel_description 'A place where everyone can be happy.' hotel_description_embedding +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md new file mode 100644 index 0000000000000..2dd3ac1ac504c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md @@ -0,0 +1,114 @@ +--- +title: Using the SQL Server Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in SQL Server. +author: eavanvalkenburg +ms.topic: concept-article +ms.author: edvan +ms.date: 03/21/2024 +--- +# Using the SQL Server Vector Store connector (Preview) + +> [!WARNING] +> The Sql Server Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The SQL Server Vector Store connector can be used to access and manage data in SQL Server. The connector has the following characteristics. + +| Feature Area | Support | +|--|--| +| Collection maps to | SQL Server table | +| Supported key property types |
  • int
  • long
  • string
  • Guid
  • DateTime
  • byte[]
| +| Supported data property types |
  • int
  • short
  • byte
  • long
  • Guid
  • string
  • bool
  • float
  • double
  • decimal
  • byte[]
  • DateTime
  • TimeOnly
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types |
  • Flat
| +| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanDistance
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | No | +| StorageName supported? | Yes | +| HybridSearch supported? | No | + +## Get started + +Add the SQL Sever Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.SqlServer --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqlServerVectorStore(_ => ""); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqlServerVectorStore(_ => "") +``` + +You can construct a Sql Server Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.SqlServer; + +var vectorStore = new SqlServerVectorStore(""); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.SqlServer; + +var collection = new SqlServerCollection("", "skhotels"); +``` + +## Data mapping + +The SQL Server Vector Store connector provides a default mapper when mapping from the data model to storage. +This mapper does a direct conversion of the list of properties on the data model to the columns in SQL Server. + +### Property name override + +You can provide override property names to use in storage that is different to the property names on the data model. +The property name override is done by setting the option via the data model property attributes or record definition. + +Here is an example of a data model with set on its attributes and how that will be represented in a SQL Server command. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string? HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```sql +CREATE TABLE Hotel ( +[HotelId] BIGINT NOT NULL, +[hotel_name] NVARCHAR(MAX), +[hotel_description] NVARCHAR(MAX), +[DescriptionEmbedding] VECTOR(4), +PRIMARY KEY ([HotelId]) +); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md new file mode 100644 index 0000000000000..e13c5a5461bbd --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md @@ -0,0 +1,121 @@ +--- +title: Using the SQLite Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in SQLite. +author: dmytrostruk +ms.topic: concept-article +ms.author: dmytrostruk +ms.date: 10/24/2024 +--- +# Using the SQLite Vector Store connector (Preview) + +> [!WARNING] +> The Sqlite Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The SQLite Vector Store connector can be used to access and manage data in SQLite. The connector has the following characteristics. + +| Feature Area | Support | +|--|--| +| Collection maps to | SQLite table | +| Supported key property types |
  • int
  • long
  • string
| +| Supported data property types |
  • int
  • long
  • short
  • string
  • bool
  • float
  • double
  • byte[]
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types | N/A | +| Supported distance functions |
  • CosineDistance
  • ManhattanDistance
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | No | +| IsFullTextIndexed supported? | No | +| StorageName supported? | Yes | +| HybridSearch supported? | No | + +## Get started + +Add the SQLite Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.SqliteVec --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. + +```csharp +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:"); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:") +``` + +You can construct a SQLite Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Connectors.SqliteVec; + +var vectorStore = new SqliteVectorStore("Data Source=:memory:"); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Connectors.SqliteVec; + +var collection = new SqliteCollection("Data Source=:memory:", "skhotels"); +``` + +## Data mapping + +The SQLite Vector Store connector provides a default mapper when mapping from the data model to storage. +This mapper does a direct conversion of the list of properties on the data model to the columns in SQLite. + +With the vector search extension, vectors are stored in virtual tables, separately from key and data properties. +By default, the virtual table with vectors will use the same name as the table with key and data properties, but with a `vec_` prefix. For example, if the collection name in `SqliteCollection` is `skhotels`, the name of the virtual table with vectors will be `vec_skhotels`. It's possible to override the virtual table name by using the `SqliteVectorStoreOptions.VectorVirtualTableName` or `SqliteCollectionOptions.VectorVirtualTableName` properties. + +### Property name override + +You can provide override property names to use in storage that is different to the property names on the data model. +The property name override is done by setting the option via the data model property attributes or record definition. + +Here is an example of a data model with set on its attributes and how that will be represented in a SQLite command. + +```csharp +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string? HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```tsql +CREATE TABLE Hotels ( + HotelId INTEGER PRIMARY KEY, + hotel_name TEXT, + hotel_description TEXT +); + +CREATE VIRTUAL TABLE vec_Hotels ( + HotelId INTEGER PRIMARY KEY, + DescriptionEmbedding FLOAT[4] distance_metric=cosine +); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md new file mode 100644 index 0000000000000..01d87816bbd1d --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md @@ -0,0 +1,73 @@ +--- +title: Using the Volatile (In-Memory) Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data using the Volatile (in-memory) vector store. +ms.topic: concept-article +ms.date: 07/08/2024 +--- +# Using the Volatile (In-Memory) connector (Preview) + +> [!WARNING] +> The C# VolatileVectorStore is obsolete and has been replaced with a new package. See [InMemory Connector](./inmemory-connector.md) + +## Overview + +The Volatile Vector Store connector is a Vector Store implementation that uses no external database and stores data in memory. +This Vector Store is useful for prototyping scenarios or where high-speed in-memory operations are required. + +The connector has the following characteristics. + +| Feature Area | Support | +|---------------------------------------|-------------------------------| +| Collection maps to | In-memory dictionary | +| Supported key property types | Any type that can be compared | +| Supported data property types | Any type | +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types | N/A | +| Supported distance functions | N/A | +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StoragePropertyName supported? | No, since storage is volatile and data reuse is therefore not possible, custom naming is not useful and not supported. | + +## Get started + +Add the Semantic Kernel Core NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Core +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. + +```csharp +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var kernelBuilder = Kernel + .CreateBuilder() + .AddVolatileVectorStore(); +``` + +```csharp +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddVolatileVectorStore(); +``` + +You can construct a Volatile Vector Store instance directly. + +```csharp +using Microsoft.SemanticKernel.Data; + +var vectorStore = new VolatileVectorStore(); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using Microsoft.SemanticKernel.Data; + +var collection = new VolatileVectorStoreRecordCollection("skhotels"); +``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md new file mode 100644 index 0000000000000..d1acf98dbcfe6 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md @@ -0,0 +1,173 @@ +--- +title: Using the Weaviate Vector Store connector (Preview) +description: Contains information on how to use a Vector Store connector to access and manipulate data in Weaviate. +ms.topic: concept-article +ms.date: 09/23/2024 +--- +# Using the Weaviate Vector Store connector (Preview) + +> [!WARNING] +> The Weaviate Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. + +## Overview + +The Weaviate Vector Store connector can be used to access and manage data in Weaviate. The connector has the following characteristics. + +| Feature Area | Support | +|------------------------------|---------------------| +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • DateTimeOffset
  • Guid
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| +| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| +| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • HammingDistance
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsIndexed supported? | Yes | +| IsFullTextIndexed supported? | Yes | +| StorageName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +## Limitations + +Notable Weaviate connector functionality limitations. + +| Feature Area | Workaround | +|--|--| +| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | + +> [!WARNING] +> Weaviate requires collection names to start with an upper case letter. If you do not provide a collection name with an upper case letter, Weaviate will return an error when you try and create your collection. The error that you will see is `Cannot query field "mycollection" on type "GetObjectsObj". Did you mean "Mycollection"?` where `mycollection` is your collection name. In this example, if you change your collection name to `Mycollection` instead, this will fix the error. + +## Get started + +Add the Weaviate Vector Store connector NuGet package to your project. + +```dotnetcli +dotnet add package Microsoft.SemanticKernel.Connectors.Weaviate --prerelease +``` + +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. +The Weaviate vector store uses an `HttpClient` to communicate with the Weaviate service. There are two options for providing the URL/endpoint for the Weaviate service. +It can be provided via options or by setting the base address of the `HttpClient`. + +This first example shows how to set the service URL via options. +Also note that these methods will retrieve an `HttpClient` instance for making calls to the Weaviate service from the dependency injection service provider. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); +``` + +```csharp +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); +``` + +Overloads where you can specify your own `HttpClient` are also provided. +In this case it's possible to set the service url via the `HttpClient` `BaseAddress` option. + +```csharp +using System.Net.Http; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; +services.AddWeaviateVectorStore(_ => client); +``` + +```csharp +using System.Net.Http; +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; +builder.Services.AddWeaviateVectorStore(_ => client); +``` + +You can construct a Weaviate Vector Store instance directly as well. + +```csharp +using System.Net.Http; +using Microsoft.SemanticKernel.Connectors.Weaviate; + +var vectorStore = new WeaviateVectorStore( + new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }); +``` + +It's possible to construct a direct reference to a named collection. + +```csharp +using System.Net.Http; +using Microsoft.SemanticKernel.Connectors.Weaviate; + +var collection = new WeaviateCollection( + new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }, + "Skhotels"); +``` + +If needed, it is possible to pass an API key, as an option, when using any of the previously mentioned mechanisms. For example: + +```csharp +using Microsoft.SemanticKernel; + +var services = new ServiceCollection(); +services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), secretVar); +``` + +## Data mapping + +The Weaviate Vector Store connector provides a default mapper when mapping from the data model to storage. +Weaviate requires properties to be mapped into id, payload and vectors groupings. +The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping. + +- The data model property annotated as a key will be mapped to the Weaviate `id` property. +- The data model properties annotated as data will be mapped to the Weaviate `properties` object. +- The data model properties annotated as vectors will be mapped to the Weaviate `vectors` object. + +The default mapper uses `System.Text.Json.JsonSerializer` to convert to the storage schema. +This means that usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the +data model property name is required. + +Here is an example of a data model with `JsonPropertyNameAttribute` set and how that will be represented in Weaviate. + +```csharp +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public Guid HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.QuantizedFlat)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +``` + +```json +{ + "id": "11111111-1111-1111-1111-111111111111", + "properties": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." }, + "vectors": { + "HOTEL_DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1], + } +} +``` \ No newline at end of file diff --git a/docs/ai/conceptual/vector-databases.md b/docs/ai/vector-stores/overview.md similarity index 56% rename from docs/ai/conceptual/vector-databases.md rename to docs/ai/vector-stores/overview.md index d8b5728f2ebb4..f2653e514a757 100644 --- a/docs/ai/conceptual/vector-databases.md +++ b/docs/ai/vector-stores/overview.md @@ -34,7 +34,7 @@ Vector databases and their search features are especially useful in [RAG pattern 1. Run a vector search across your data, comparing the user prompt embedding to the embeddings in your database. 1. Use a language model such as GPT-4 to assemble a user-friendly completion from the vector search results. -Visit the [Implement Azure OpenAI with RAG using vector search in a .NET app](../tutorials/tutorial-ai-vector-search.md) tutorial for a hands-on example of this flow. +For a hands-on example of this flow, see the [Implement Azure OpenAI with RAG using vector search in a .NET app](../tutorials/tutorial-ai-vector-search.md) tutorial. Other benefits of the RAG pattern include: @@ -53,61 +53,6 @@ The library provides the following key capabilities: - **Vector and text search**: Query records by semantic similarity using vector search, or by keyword using text search. - **Collection management**: Create, list, and delete collections (tables or indices) in a vector store. -### Define a data model - -To store records in a vector store, define a .NET class and annotate its properties with the following attributes from the namespace: - -- : Marks the property that uniquely identifies each record (the primary key). -- : Marks properties that contain regular data (strings, numbers, and so on) to store and optionally filter on. -- : Marks properties that store embedding vectors. You specify the number of dimensions and the distance function to use for similarity comparisons. - -The following example defines a data model for a hotel: - -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public int HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string? HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 1536, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string[]? Tags { get; set; } -} -``` - -The `Dimensions` parameter must match the output size of the embedding model you use. For example, `text-embedding-3-small` produces 1536-dimensional vectors, while `text-embedding-3-large` produces 3072-dimensional vectors. - -### Automatic embedding generation - -Instead of generating embeddings yourself before each upsert, you can declare your vector property as a `string` type and configure an `IEmbeddingGenerator` on the vector store. The store then generates the embedding automatically when you upsert a record, using the string value you provide: - -```csharp -public class FinanceInfo -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData] - public string Text { get; set; } = ""; - - // Declare as string to enable automatic embedding generation on upsert. - [VectorStoreVector(1536)] - public string EmbeddingSource { get; set; } = ""; -} -``` - -You can configure the `IEmbeddingGenerator` at the vector store or collection level, or on individual vector properties. With auto-embedding, you can also pass a `string` directly to `SearchAsync` instead of a precomputed vector—the store generates the search embedding for you. For more information, see [Use built-in embedding generation](../how-to/use-vector-stores.md#use-built-in-embedding-generation). - ### Key abstractions The `Microsoft.Extensions.VectorData.Abstractions` library exposes the following main abstract classes: @@ -116,84 +61,7 @@ The `Microsoft.Extensions.VectorData.Abstractions` library exposes the following - : Represents a named collection of records within a vector store. Use it to perform CRUD and search operations. Also inherits from `IVectorSearchable`. - `IKeywordHybridSearchable`: Implemented by collections that support hybrid search combining vector similarity with keyword matching. -The following example shows how to get a collection from a vector store and upsert (insert or update) records: - -```csharp -// Get or create a collection named "hotels". -VectorStoreCollection collection = - vectorStore.GetCollection("hotels"); - -// Ensure the collection exists in the database. -await collection.EnsureCollectionExistsAsync(); - -// Upsert a record. -await collection.UpsertAsync(new Hotel -{ - HotelId = 1, - HotelName = "Seaside Retreat", - Description = "A peaceful hotel on the coast with stunning ocean views.", - DescriptionEmbedding = await embeddingGenerator.GenerateVectorAsync( - "A peaceful hotel on the coast with stunning ocean views."), - Tags = ["beach", "ocean", "relaxation"] -}); -``` - -### Perform vector search - -Use the `SearchAsync` method to search for semantically similar records. Pass in an embedding vector for your query and specify the number of results to return: - -```csharp -// Generate an embedding for the search query. -ReadOnlyMemory queryEmbedding = - await embeddingGenerator.GenerateVectorAsync("beachfront hotel"); - -// Search for the top 3 most similar hotels. -IAsyncEnumerable> results = - collection.SearchAsync(queryEmbedding, top: 3); - -await foreach (VectorSearchResult result in results) -{ - Console.WriteLine($"Hotel: {result.Record.HotelName}"); - Console.WriteLine($"Score: {result.Score}"); -} -``` - -### Filter search results - -Use the class to filter vector search results by field values. Only properties marked with `IsIndexed = true` in `[VectorStoreData]` can be used in filters: - -```csharp -var searchOptions = new VectorSearchOptions -{ - Filter = hotel => hotel.HotelName == "Seaside Retreat" -}; - -IAsyncEnumerable> results = - collection.SearchAsync(queryEmbedding, top: 3, searchOptions); -``` - -Filters are expressed as LINQ expressions and compiled into the query syntax of the underlying database. The supported operations vary by connector. - -### Hybrid search - -Some connectors support *hybrid search*, which combines vector similarity with full-text keyword matching. To use hybrid search, check whether your collection implements `IKeywordHybridSearchable` and use the `HybridSearchAsync` method: - -```csharp -if (collection is IKeywordHybridSearchable hybridSearch) -{ - var results = hybridSearch.HybridSearchAsync( - queryEmbedding, - keywords: ["ocean", "beach"], - top: 3); - - await foreach (var result in results) - { - Console.WriteLine($"Hotel: {result.Record.HotelName}, Score: {result.Score}"); - } -} -``` - -For hybrid search to work, the data model must have at least one vector property and one string property with `IsFullTextIndexed = true`. +For a step-by-step guide covering data model definition, CRUD operations, vector search, filtering, hybrid search, and embedding generation, see [Use vector stores in .NET AI apps](how-to/use-vector-stores.md). ## Available vector store connectors @@ -217,18 +85,18 @@ The `Microsoft.Extensions.VectorData.Abstractions` package defines the abstracti | SQLite | [Microsoft.SemanticKernel.Connectors.SqliteVec](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqliteVec) | | Weaviate | [Microsoft.SemanticKernel.Connectors.Weaviate](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Weaviate) | -All connectors implement the same `VectorStore` and `VectorStoreCollection` abstract classes, so you can switch between them without changing your application logic. +All connectors implement the same and abstract classes, so you can switch between them without changing your application logic. > [!TIP] > Use the in-memory connector (`Microsoft.SemanticKernel.Connectors.InMemory`) during development and testing. It doesn't require any external service or configuration, and you can swap it out for a production connector later. > [!IMPORTANT] -> Not all connectors are maintained by the Microsoft Semantic Kernel project. When evaluating a connector, review its quality, licensing, support, and compatibility to ensure it meets your requirements. +> Not all connectors are maintained by the Microsoft Semantic Kernel team. When evaluating a connector, review its quality, licensing, support, and compatibility to ensure it meets your requirements. ## Related content +- [Use vector stores in .NET AI apps](how-to/use-vector-stores.md) - [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md) - [Implement Azure OpenAI with RAG using vector search in a .NET app](../tutorials/tutorial-ai-vector-search.md) -- [Use vector stores in .NET AI apps](../how-to/use-vector-stores.md) -- [Data ingestion](data-ingestion.md) -- [Embeddings in .NET](embeddings.md) +- [Data ingestion](../conceptual/data-ingestion.md) +- [Embeddings in .NET](../conceptual/embeddings.md) \ No newline at end of file diff --git a/docs/ai/vector-stores/schema-with-record-definition.md b/docs/ai/vector-stores/schema-with-record-definition.md new file mode 100644 index 0000000000000..28b108ff4cf58 --- /dev/null +++ b/docs/ai/vector-stores/schema-with-record-definition.md @@ -0,0 +1,106 @@ +--- +title: Defining your storage schema using a record definition (Preview) +description: Describes how to create a record definition to use when writing to or reading from a Vector Store. +ms.topic: reference +ms.date: 07/08/2024 +--- +# Define your storage schema using a record definition (Preview) + +The Vector Store connectors use a model first approach to interacting with databases and allows annotating data +models with information that is needed for creating indexes or mapping data to the database schema. + +Another way of providing this information is via record definitions, that can be defined and supplied separately to the data model. +This can be useful in multiple scenarios: + +- There might be a case where a developer wants to use the same data model with more than one configuration. +- There might be a case where a developer wants to use a built-in type, like a dict, or a optimized format like a dataframe and still wants to leverage the vector store functionality. + +Here is an example of how to create a record definition. + +```csharp +using Microsoft.Extensions.VectorData; + +var hotelDefinition = new VectorStoreCollectionDefinition +{ + Properties = new List + { + new VectorStoreKeyProperty("HotelId", typeof(ulong)), + new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, + new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, + } +}; +``` + +When creating a definition you always have to provide a name and type for each property in your schema, since this is required for index creation and data mapping. + +To use the definition, pass it to the GetCollection method. + +```csharp +var collection = vectorStore.GetCollection("skhotels", hotelDefinition); +``` + +## Record property configuration classes + +### VectorStoreKeyProperty + +Use this class to indicate that your property is the key of the record. + +```csharp +new VectorStoreKeyProperty("HotelId", typeof(ulong)), +``` + +#### VectorStoreKeyProperty configuration settings + +| Parameter | Required | Description | +|--|:-:|--| +| Name | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| Type | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| StorageName | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | + +> [!TIP] +> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). + +### VectorStoreDataProperty + +Use this class to indicate that your property contains general data that is not a key or a vector. + +```csharp +new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, +``` + +#### VectorStoreDataProperty configuration settings + +| Parameter | Required | Description | +|--|:-:|--| +| Name | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| Type | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| IsIndexed | No | Indicates whether the property should be indexed for filtering in cases where a database requires opting in to indexing per property. Default is false. | +| IsFullTextIndexed | No | Indicates whether the property should be indexed for full text search for databases that support full text search. Default is false. | +| StorageName | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | + +> [!TIP] +> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). + +### VectorStoreVectorProperty + +Use this class to indicate that your property contains a vector. + +```csharp +new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, +``` + +#### VectorStoreVectorProperty configuration settings + +| Parameter | Required | Description | +|--|:-:|--| +| Name | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| Type | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| Dimensions | Yes | The number of dimensions that the vector has. This is required for creating a vector index for a collection. | +| IndexKind | No | The type of index to index the vector with. Default varies by vector store type. | +| DistanceFunction | No | The type of function to use when doing vector comparison during vector search over this vector. Default varies by vector store type. | +| StorageName | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | +| EmbeddingGenerator | No | Allows specifying a `Microsoft.Extensions.AI.IEmbeddingGenerator` instance to use for generating embeddings automatically for the decorated property. | + +> [!TIP] +> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). \ No newline at end of file diff --git a/docs/ai/tutorials/tutorial-ai-vector-search.md b/docs/ai/vector-stores/tutorial-vector-search.md similarity index 94% rename from docs/ai/tutorials/tutorial-ai-vector-search.md rename to docs/ai/vector-stores/tutorial-vector-search.md index 4d712d9624b20..a3f2f50775278 100644 --- a/docs/ai/tutorials/tutorial-ai-vector-search.md +++ b/docs/ai/vector-stores/tutorial-vector-search.md @@ -9,7 +9,7 @@ ms.author: alexwolf # Implement Azure OpenAI with RAG using vector search in a .NET app -This tutorial explores integration of the RAG pattern using Open AI models and vector search capabilities in a .NET app. The sample application performs vector searches on custom data stored in Azure Cosmos DB for MongoDB and further refines the responses using generative AI models, such as GPT-35 and GPT-4. In the sections that follow, you'll set up a sample application and explore key code examples that demonstrate these concepts. +This tutorial explores integration of the RAG pattern using OpenAI models and vector search capabilities in a .NET app. The sample application performs vector searches on custom data stored in Azure Cosmos DB for MongoDB and further refines the responses using generative AI models, such as GPT-35 and GPT-4. In the sections that follow, you'll set up a sample application and explore key code examples that demonstrate these concepts. ## Prerequisites @@ -186,7 +186,7 @@ When you run the app for the first time, it connects to Azure Cosmos DB and repo 1. Select the **Ask AI Assistant (search for a recipe by name or description, or ask a question)** option in the application to run a user query. - The user query is converted to an embedding using the Open AI service and the embedding model. The embedding is then sent to Azure Cosmos DB for MongoDB and is used to perform a vector search. The `VectorSearchAsync` method in the _VCoreMongoService.cs_ file performs a vector search to find vectors that are close to the supplied vector and returns a list of documents from Azure Cosmos DB for MongoDB vCore. + The user query is converted to an embedding using the Open AI service and the embedding model. The embedding is then sent to Azure Cosmos DB for MongoDB and is used to perform a vector search. The method in the _VCoreMongoService.cs_ file performs a vector search to find vectors that are close to the supplied vector and returns a list of documents from Azure Cosmos DB for MongoDB vCore. ```C# public async Task> VectorSearchAsync(float[] queryVector) @@ -293,4 +293,4 @@ When you run the app for the first time, it connects to Azure Cosmos DB and repo - Assume the user is not an expert in cooking. - Format the content so that it can be printed to the Command Line console. - In case there is more than one recipe you find, let the user pick the most appropriate recipe."; - ``` + ``` \ No newline at end of file diff --git a/docs/ai/vector-stores/vector-search.md b/docs/ai/vector-stores/vector-search.md new file mode 100644 index 0000000000000..2ad12b696fba4 --- /dev/null +++ b/docs/ai/vector-stores/vector-search.md @@ -0,0 +1,228 @@ +--- +title: Vector search using Vector Store connectors (Preview) +description: Describes the different options you can use when doing a vector search using Vector Store connectors. +ms.topic: concept-article +ms.date: 09/23/2024 +--- +# Vector search using Vector Store connectors (Preview) + +The Microsoft.Extensions.VectorData library provides vector search capabilities as part of its Vector Store abstractions. This supports filtering and many other options, which this article will explain in more detail. + +> [!TIP] +> To see how you can search without generating embeddings yourself, see [Letting the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). + +## Vector search + +The method allows searching using data that has already been vectorized. This method takes a vector and an optional class as input. is available on the following types: + +- +- + +Note that implements from `IVectorSearchable`. + +Assuming you have a collection that already contains data, you can easily search it. Here is an example using Qdrant. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +// Placeholder embedding generation method. +async Task> GenerateEmbeddingAsync(string textToVectorize) +{ + // your logic here +} + +// Create a Qdrant VectorStore object and choose an existing collection that already contains records. +VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +VectorStoreCollection collection = vectorStore.GetCollection("skhotels"); + +// Generate a vector for your search text, using your chosen embedding generation implementation. +ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); + +// Do the search, passing an options object with a Top value to limit results to the single top match. +var searchResult = collection.SearchAsync(searchVector, top: 1); + +// Inspect the returned hotel. +await foreach (var record in searchResult) +{ + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); +} +``` + +> [!TIP] +> For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). + +## Supported vector types + + takes a generic type as the vector parameter. +The types of vectors supported by each data store vary. +See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. + +It's also important for the search vector type to match the target vector that is being searched, for example, if you have two vectors +on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector +you are targeting. +See [VectorProperty](#vectorproperty) for how to pick a target vector if you have more than one per record. + +## Vector search options + +The following options can be provided using the `VectorSearchOptions` class. + +### VectorProperty + +The `VectorProperty` option can be used to specify the vector property to target during the search. +If none is provided and the data model contains only one vector, that vector will be used. +If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. + +```csharp +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; + +var vectorStore = new InMemoryVectorStore(); +var collection = vectorStore.GetCollection("skproducts"); + +// Create the vector search options and indicate that you want to search the FeatureListEmbedding property. +var vectorSearchOptions = new VectorSearchOptions +{ + VectorProperty = r => r.FeatureListEmbedding +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + +public sealed class Product +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public string Description { get; set; } + + [VectorStoreData] + public List FeatureList { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } +} +``` + +### `Top` and `Skip` + +The `Top` and `Skip` options allow you to limit the number of results to the Top n results and +to skip a number of results from the top of the resultset. +Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. + +```csharp +// Create the vector search options and indicate that you want to skip the first 40 results. +var vectorSearchOptions = new VectorSearchOptions +{ + Skip = 40 +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +// Pass 'top: 20' to indicate that you want to retrieve the next 20 results after skipping +// the first 40 +var searchResult = collection.SearchAsync(searchVector, top: 20, vectorSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.FeatureList); +} +``` + +The default value `Skip` is 0. + +### IncludeVectors + +The `IncludeVectors` option allows you to specify whether you wish to return vectors in the search results. +If `false`, the vector properties on the returned model will be left null. +Using `false` can significantly reduce the amount of data retrieved from the vector store during search, +making searches more efficient. + +The default value for `IncludeVectors` is `false`. + +```csharp +// Create the vector search options and indicate that you want to include vectors in the search results. +var vectorSearchOptions = new VectorSearchOptions +{ + IncludeVectors = true +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.FeatureList); +} +``` + +### Filter + +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection +before applying the vector search. + +This has multiple benefits: + +- Reduce latency and processing cost, since only records remaining after filtering need to be compared with the search vector and therefore fewer vector comparisons have to be done. +- Limit the resultset for for example, access control purposes, by excluding data that the user shouldn't have access to. + +Note that in order for fields to be used for filtering, many vector stores require those fields to be indexed first. +Some vector stores will allow filtering using any field, but might optionally allow indexing to improve filtering performance. + +If creating a collection via the Vector Store abstractions and you wish to enable filtering on a field, +set the property to true when defining your data model or when creating your record definition. + +> [!TIP] +> For more information on how to set the property, see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstoredataattribute-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md#vectorstoredataproperty-configuration-settings). + +Filters are expressed using LINQ expressions based on the type of the data model. +The set of LINQ expressions supported will vary depending on the functionality supported +by each database, but all databases support a broad base of common expressions, for example, equals, +not equals, and, or, etc. + +```csharp +// Create the vector search options and set the filter on the options. +var vectorSearchOptions = new VectorSearchOptions +{ + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.Definition); +} + +sealed class Glossary +{ + [VectorStoreKey] + public ulong Key { get; set; } + + // Category is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public string Category { get; set; } + + // Tags is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public List Tags { get; set; } + + [VectorStoreData] + public string Term { get; set; } + + [VectorStoreData] + public string Definition { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } +} +``` \ No newline at end of file From c523c87f6fd6abf021948cbcabc6a05fe9ba0dcb Mon Sep 17 00:00:00 2001 From: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Date: Sat, 28 Feb 2026 09:49:20 -0800 Subject: [PATCH 08/11] add project files --- .../vector-stores/defining-your-data-model.md | 42 +--- docs/ai/vector-stores/dynamic-data-model.md | 40 +--- docs/ai/vector-stores/embedding-generation.md | 203 +--------------- .../how-to/build-your-own-connector.md | 20 +- .../how-to/snippets/HowToSnippets.csproj | 19 ++ .../AdditionalExamples.cs | 70 ++++++ .../snippets/build-your-own-connector.cs | 19 ++ .../snippets/vector-store-data-ingestion.cs | 182 ++++++++++++++ .../vector-stores/how-to/use-vector-stores.md | 71 +----- .../how-to/vector-store-data-ingestion.md | 183 +------------- docs/ai/vector-stores/hybrid-search.md | 145 +---------- .../azure-ai-search-connector.md | 82 +------ .../azure-cosmosdb-mongodb-connector.md | 92 +------ .../azure-cosmosdb-nosql-connector.md | 161 +------------ .../couchbase-connector.md | 226 +----------------- .../elasticsearch-connector.md | 113 +-------- .../inmemory-connector.md | 30 +-- .../mongodb-connector.md | 66 +---- .../oracle-connector.md | 103 +------- .../pinecone-connector.md | 87 +------ .../postgres-connector.md | 221 +---------------- .../qdrant-connector.md | 112 +-------- .../redis-connector.md | 135 +---------- .../snippets/ConnectorSnippets.csproj | 46 ++++ .../snippets/azure-ai-search-connector.cs | 81 +++++++ .../azure-cosmosdb-mongodb-connector.cs | 91 +++++++ .../azure-cosmosdb-nosql-connector.cs | 160 +++++++++++++ .../snippets/couchbase-connector.cs | 225 +++++++++++++++++ .../snippets/elasticsearch-connector.cs | 112 +++++++++ .../snippets/inmemory-connector.cs | 29 +++ .../snippets/mongodb-connector.cs | 65 +++++ .../snippets/oracle-connector.cs | 102 ++++++++ .../snippets/pinecone-connector.cs | 86 +++++++ .../snippets/postgres-connector.cs | 220 +++++++++++++++++ .../snippets/qdrant-connector.cs | 111 +++++++++ .../snippets/redis-connector.cs | 134 +++++++++++ .../snippets/sql-connector.cs | 47 ++++ .../snippets/sqlite-connector.cs | 47 ++++ .../snippets/volatile-connector.cs | 28 +++ .../snippets/weaviate-connector.cs | 82 +++++++ .../sql-connector.md | 48 +--- .../sqlite-connector.md | 48 +--- .../volatile-connector.md | 29 +-- .../weaviate-connector.md | 83 +------ .../schema-with-record-definition.md | 31 +-- .../snippets/VectorStoreSnippets.csproj | 21 ++ .../snippets/defining-your-data-model.cs | 41 ++++ .../snippets/dynamic-data-model.cs | 39 +++ .../snippets/embedding-generation.cs | 202 ++++++++++++++++ .../vector-stores/snippets/hybrid-search.cs | 144 +++++++++++ .../snippets/schema-with-record-definition.cs | 30 +++ .../vector-stores/snippets/vector-search.cs | 139 +++++++++++ docs/ai/vector-stores/vector-search.md | 140 +---------- 53 files changed, 2740 insertions(+), 2343 deletions(-) create mode 100644 docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj create mode 100644 docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs create mode 100644 docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs create mode 100644 docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs create mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs create mode 100644 docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj create mode 100644 docs/ai/vector-stores/snippets/defining-your-data-model.cs create mode 100644 docs/ai/vector-stores/snippets/dynamic-data-model.cs create mode 100644 docs/ai/vector-stores/snippets/embedding-generation.cs create mode 100644 docs/ai/vector-stores/snippets/hybrid-search.cs create mode 100644 docs/ai/vector-stores/snippets/schema-with-record-definition.cs create mode 100644 docs/ai/vector-stores/snippets/vector-search.cs diff --git a/docs/ai/vector-stores/defining-your-data-model.md b/docs/ai/vector-stores/defining-your-data-model.md index f8135c4390943..5fb16481c84b7 100644 --- a/docs/ai/vector-stores/defining-your-data-model.md +++ b/docs/ai/vector-stores/defining-your-data-model.md @@ -20,27 +20,7 @@ The properties on these classes are decorated with attributes that indicate the Here is an example of a model that is decorated with these attributes. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string[] Tags { get; set; } -} -``` +:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="Overview"::: ## Attributes @@ -48,10 +28,7 @@ public class Hotel Use the attribute to indicate that your property is the key of the record. -```csharp -[VectorStoreKey] -public ulong HotelId { get; set; } -``` +:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreKeyAttribute"::: #### VectorStoreKeyAttribute parameters @@ -63,10 +40,7 @@ public ulong HotelId { get; set; } Use the attribute to indicate that your property contains general data that is not a key or a vector. -```csharp -[VectorStoreData(IsIndexed = true)] -public string HotelName { get; set; } -``` +:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreDataAttribute"::: #### VectorStoreDataAttribute parameters @@ -83,20 +57,14 @@ public string HotelName { get; set; } Use the attribute to indicate that your property contains a vector. -```csharp -[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] -public ReadOnlyMemory? DescriptionEmbedding { get; set; } -``` +:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreVectorAttribute1"::: It's also possible to use on properties that dont' have a vector type, for example, a property of type `string`. When a property is decorated in this way, you need to provide an instance to the vector store. When upserting the record, the text that is in the `string` property is automatically converted and stored as a vector in the database. It's not possible to retrieve a vector using this mechanism. -```csharp -[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] -public string DescriptionEmbedding { get; set; } -``` +:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreVectorAttribute2"::: > [!TIP] > For more information on how to use built-in embedding generation, see [Let the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). diff --git a/docs/ai/vector-stores/dynamic-data-model.md b/docs/ai/vector-stores/dynamic-data-model.md index 2fc296aafbdb9..8594e38093373 100644 --- a/docs/ai/vector-stores/dynamic-data-model.md +++ b/docs/ai/vector-stores/dynamic-data-model.md @@ -28,38 +28,7 @@ A record definition can be used to provide the schema information. Unlike a data To use the `Dictionary` with a connector, simply specify it as your data model when creating a collection, and simultaneously provide a record definition. -```csharp -// Create the definition to define the schema. -VectorStoreCollectionDefinition definition = new() -{ - Properties = new List - { - new VectorStoreKeyProperty("Key", typeof(string)), - new VectorStoreDataProperty("Term", typeof(string)), - new VectorStoreDataProperty("Definition", typeof(string)), - new VectorStoreVectorProperty("DefinitionEmbedding", typeof(ReadOnlyMemory), dimensions: 1536) - } -}; - -// When getting your collection instance from a vector store instance -// specify the Dictionary, using object as the key type for your database -// and also pass your record definition. -// Note that you have to use GetDynamicCollection instead of the regular GetCollection method -// to get an instance of a collection using Dictionary. -var dynamicDataModelCollection = vectorStore.GetDynamicCollection( - "glossary", - definition); - -// Since schema information is available from the record definition -// it's possible to create a collection with the right vectors, -// dimensions, indexes, and distance functions. -await dynamicDataModelCollection.EnsureCollectionExistsAsync(); - -// When retrieving a record from the collection, key, data, and vector values can -// now be accessed via the dictionary entries. -var record = await dynamicDataModelCollection.GetAsync("SK"); -Console.WriteLine(record["Definition"]); -``` +:::code language="csharp" source="./snippets/dynamic-data-model.cs" id="Example1"::: When constructing a collection instance directly, the record definition is passed as an option. For example, here is an example of constructing @@ -69,9 +38,4 @@ Each vector store collection implementation has a separate `*DynamicCollection` class that can be used with `Dictionary`. This is because these implementations might support NativeAOT/trimming. -```csharp -new AzureAISearchDynamicCollection( - searchIndexClient, - "glossary", - new() { Definition = definition }); -``` +:::code language="csharp" source="./snippets/dynamic-data-model.cs" id="Example2"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/embedding-generation.md b/docs/ai/vector-stores/embedding-generation.md index a7fb2cda0642c..3cc3236657837 100644 --- a/docs/ai/vector-stores/embedding-generation.md +++ b/docs/ai/vector-stores/embedding-generation.md @@ -16,10 +16,7 @@ You can configure an embedding generator on your vector store, allowing embeddin To enable generating vectors automatically on upsert, the vector property on your data model is defined as the source type, for example, string but still decorated with a . -```csharp -[VectorStoreVector(1536)] -public string Embedding { get; set; } -``` +:::code language="csharp" source="./snippets/embedding-generation.cs" id="LetTheVectorStoreGenerateEmbeddings"::: Before upsert, the `Embedding` property should contain the string from which a vector should be generated. The type of the vector stored in the database (for example, float32, float16, etc.) will be derived from the configured embedding generator. @@ -33,161 +30,29 @@ Embedding generators implementing the `Microsoft.Extensions.AI` abstractions are You can set a default embedding generator for the entire vector store. This generator will be used for all collections and properties unless overridden. - ```csharp - using Microsoft.Extensions.AI; - using Microsoft.SemanticKernel.Connectors.Qdrant; - using OpenAI; - using Qdrant.Client; - - var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - - var vectorStore = new QdrantVectorStore( - new QdrantClient("localhost"), - ownsClient: true, - new QdrantVectorStoreOptions - { - EmbeddingGenerator = embeddingGenerator - }); - ``` + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnTheVectorStore"::: 2. **On a Collection**: You can configure an embedding generator for a specific collection, overriding the store-level generator. - ```csharp - using Microsoft.Extensions.AI; - using Microsoft.SemanticKernel.Connectors.Qdrant; - using OpenAI; - using Qdrant.Client; - - var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - - var collectionOptions = new QdrantCollectionOptions - { - EmbeddingGenerator = embeddingGenerator - }; - var collection = new QdrantCollection( - new QdrantClient("localhost"), - "myCollection", - ownsClient: true, - collectionOptions); - ``` + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnACollection"::: 3. **On a Record Definition**: When defining properties programmatically using , you can specify an embedding generator for all properties. - ```csharp - using Microsoft.Extensions.AI; - using Microsoft.Extensions.VectorData; - using Microsoft.SemanticKernel.Connectors.Qdrant; - using OpenAI; - using Qdrant.Client; - - var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - - var definition = new VectorStoreCollectionDefinition - { - EmbeddingGenerator = embeddingGenerator, - Properties = new List - { - new VectorStoreKeyProperty("Key", typeof(ulong)), - new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) - } - }; - - var collectionOptions = new QdrantCollectionOptions - { - Definition = definition - }; - var collection = new QdrantCollection( - new QdrantClient("localhost"), - "myCollection", - ownsClient: true, - collectionOptions); - ``` + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnARecordDefinition"::: 4. **On a Vector Property Definition**: When defining properties programmatically, you can set an embedding generator directly on the property. - ```csharp - using Microsoft.Extensions.AI; - using Microsoft.Extensions.VectorData; - using OpenAI; - - var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - - var vectorProperty = new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) - { - EmbeddingGenerator = embeddingGenerator - }; - ``` + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnAVectorPropertyDefinition"::: ### Example usage The following example demonstrates how to use the embedding generator to automatically generate vectors during both upsert and search operations. This approach simplifies workflows by eliminating the need to precompute embeddings manually. -```csharp - -// The data model -internal class FinanceInfo -{ - [VectorStoreKey] - public string Key { get; set; } = string.Empty; - - [VectorStoreData] - public string Text { get; set; } = string.Empty; - - // Note that the vector property is typed as a string, and - // its value is derived from the Text property. The string - // value will however be converted to a vector on upsert and - // stored in the database as a vector. - [VectorStoreVector(1536)] - public string Embedding => this.Text; -} - -// Create an OpenAI embedding generator. -var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - -// Use the embedding generator with the vector store. -var vectorStore = new InMemoryVectorStore(new() { EmbeddingGenerator = embeddingGenerator }); -var collection = vectorStore.GetCollection("finances"); -await collection.EnsureCollectionExistsAsync(); - -// Create some test data. -string[] budgetInfo = -{ - "The budget for 2020 is EUR 100 000", - "The budget for 2021 is EUR 120 000", - "The budget for 2022 is EUR 150 000", - "The budget for 2023 is EUR 200 000", - "The budget for 2024 is EUR 364 000" -}; - -// Embeddings are generated automatically on upsert. -var records = budgetInfo.Select((input, index) => new FinanceInfo { Key = index.ToString(), Text = input }); -await collection.UpsertAsync(records); - -// Embeddings for the search is automatically generated on search. -var searchResult = collection.SearchAsync( - "What is my budget for 2024?", - top: 1); - -// Output the matching result. -await foreach (var result in searchResult) -{ - Console.WriteLine($"Key: {result.Record.Key}, Text: {result.Record.Text}"); -} -``` +:::code language="csharp" source="./snippets/embedding-generation.cs" id="ExampleUsage"::: ## Generate embeddings yourself @@ -197,54 +62,11 @@ For information on how to construct `Microsoft.Extensions.AI` embedding generato ### Generate embeddings on upsert with `IEmbeddingGenerator` -```csharp -public async Task GenerateEmbeddingsAndUpsertAsync( - IEmbeddingGenerator> embeddingGenerator, - VectorStoreCollection collection) -{ - // Upsert a record. - string descriptionText = "A place where everyone can be happy."; - ulong hotelId = 1; - - // Generate the embedding. - ReadOnlyMemory embedding = - (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; - - // Create a record and upsert with the already generated embedding. - await collection.UpsertAsync(new Hotel - { - HotelId = hotelId, - HotelName = "Hotel Happy", - Description = descriptionText, - DescriptionEmbedding = embedding, - Tags = new[] { "luxury", "pool" } - }); -} -``` +:::code language="csharp" source="./snippets/embedding-generation.cs" id="GenerateEmbeddingsOnUpsertWithIEmbedding"::: ### Generate embeddings on search with `IEmbeddingGenerator` -```csharp -public async Task GenerateEmbeddingsAndSearchAsync( - IEmbeddingGenerator> embeddingGenerator, - VectorStoreCollection collection) -{ - // Upsert a record. - string descriptionText = "Find me a hotel with happiness in mind."; - - // Generate the embedding. - ReadOnlyMemory searchEmbedding = - (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; - - // Search using the already generated embedding. - IAsyncEnumerable> searchResult = collection.SearchAsync(searchEmbedding, top: 1); - List> resultItems = await searchResult.ToListAsync(); - - // Print the first search result. - Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); - Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); -} -``` +:::code language="csharp" source="./snippets/embedding-generation.cs" id="GenerateEmbeddingsOnSearchWithIEmbedding"::: ## Embedding dimensions @@ -261,14 +83,9 @@ If creating a collection using the Vector Store abstractions, you need to specif required for each vector property either via annotations or via the record definition. Here are examples of both setting the number of dimensions to 1536. -```csharp -[VectorStoreVector(Dimensions: 1536)] -public ReadOnlyMemory? DescriptionEmbedding { get; set; } -``` +:::code language="csharp" source="./snippets/embedding-generation.cs" id="EmbeddingDimensions1"::: -```csharp -new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 1536); -``` +:::code language="csharp" source="./snippets/embedding-generation.cs" id="EmbeddingDimensions2"::: ## See also diff --git a/docs/ai/vector-stores/how-to/build-your-own-connector.md b/docs/ai/vector-stores/how-to/build-your-own-connector.md index 5b39688ccfe1d..e21633733b9c5 100644 --- a/docs/ai/vector-stores/how-to/build-your-own-connector.md +++ b/docs/ai/vector-stores/how-to/build-your-own-connector.md @@ -130,10 +130,7 @@ additional custom strings might be accepted. For example, the goal is for a user to be able to specify a standard distance function, like `DotProductSimilarity` for any connector that supports this distance function, without needing to use different naming for each connector. -```csharp -[VectorStoreVector(1536, DistanceFunction = DistanceFunction.DotProductSimilarity] -public ReadOnlyMemory? Embedding { get; set; } -``` +:::code language="csharp" source="./snippets/build-your-own-connector.cs" id="4CollectionAndIndexCreation"::: 4.2 A user can optionally choose whether each data property should be indexed or full text indexed. In some databases, all properties might already be filterable or full text searchable by default, however @@ -237,20 +234,7 @@ using the native database support. Here is an example of an constructor following this pattern. -```csharp -public sealed class MyDBCollection : VectorStoreCollection -{ - public MyDBCollection(MyDBClient myDBClient, string collectionName, MyDBCollectionOptions? options = default) - { - } - - ... -} - -public class MyDBCollectionOptions : VectorStoreCollectionOptions -{ -} -``` +:::code language="csharp" source="./snippets/build-your-own-connector.cs" id="RecommendedCommonPatternsAndPractices"::: ## SDK changes diff --git a/docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj b/docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj new file mode 100644 index 0000000000000..ba1408b632897 --- /dev/null +++ b/docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj @@ -0,0 +1,19 @@ + + + + net10.0 + enable + enable + $(NoWarn);CS8019;CS0219;CS1591;CS8602 + + + + + + + + + + + + diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs new file mode 100644 index 0000000000000..5b30ff9e212a3 --- /dev/null +++ b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs @@ -0,0 +1,70 @@ +// +// A data model with two vector properties. +public class Hotel +{ + [VectorStoreKey] + public int HotelId { get; set; } + + [VectorStoreData] + public string? HotelName { get; set; } + + // Two separate embeddings for different aspects of the hotel. + [VectorStoreVector(1536)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory? AmenitiesEmbedding { get; set; } +} + +// Target the amenities embedding specifically. +var options = new VectorSearchOptions +{ + VectorProperty = r => r.AmenitiesEmbedding +}; + +IAsyncEnumerable> results = + collection.SearchAsync(queryEmbedding, top: 3, options); +// + +// +public class FinanceInfo +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public string Text { get; set; } = ""; + + // Use a string type to trigger automatic embedding generation on upsert. + [VectorStoreVector(1536)] + public string EmbeddingSource { get; set; } = ""; +} +// + +// +// Check whether the collection supports hybrid search. +if (collection is IKeywordHybridSearchable hybridCollection) +{ + ReadOnlyMemory queryEmbedding = + await embeddingGenerator.GenerateVectorAsync("peaceful beachfront hotel"); + + // Provide both a vector and keywords for hybrid search. + var results = hybridCollection.HybridSearchAsync( + queryEmbedding, + keywords: ["ocean", "beach"], + top: 3); + + await foreach (var result in results) + { + Console.WriteLine($"Hotel: {result.Record.HotelName}, Score: {result.Score}"); + } +} +// + +// +// Development: in-memory +var vectorStore = new InMemoryVectorStore(); + +// Production: Azure AI Search (swap in at startup) +// var vectorStore = new AzureAISearchVectorStore(new SearchIndexClient(...)); +// diff --git a/docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs b/docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs new file mode 100644 index 0000000000000..12e70cb414260 --- /dev/null +++ b/docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs @@ -0,0 +1,19 @@ +// <4CollectionAndIndexCreation> +[VectorStoreVector(1536, DistanceFunction = DistanceFunction.DotProductSimilarity] +public ReadOnlyMemory? Embedding { get; set; } +// + +// +public sealed class MyDBCollection : VectorStoreCollection +{ + public MyDBCollection(MyDBClient myDBClient, string collectionName, MyDBCollectionOptions? options = default) + { + } + + ... +} + +public class MyDBCollectionOptions : VectorStoreCollectionOptions +{ +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs b/docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs new file mode 100644 index 0000000000000..b567b806db996 --- /dev/null +++ b/docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs @@ -0,0 +1,182 @@ +// +using Microsoft.Extensions.VectorData; + +namespace SKVectorIngest; + +internal class TextParagraph +{ + /// A unique key for the text paragraph. + [VectorStoreKey] + public required string Key { get; init; } + + /// A uri that points at the original location of the document containing the text. + [VectorStoreData] + public required string DocumentUri { get; init; } + + /// The id of the paragraph from the document containing the text. + [VectorStoreData] + public required string ParagraphId { get; init; } + + /// The text of the paragraph. + [VectorStoreData] + public required string Text { get; init; } + + /// The embedding generated from the Text. + [VectorStoreVector(1536)] + public ReadOnlyMemory TextEmbedding { get; set; } +} +// + +// +using System.Text; +using System.Xml; +using DocumentFormat.OpenXml.Packaging; + +namespace SKVectorIngest; + +internal class DocumentReader +{ + public static IEnumerable ReadParagraphs(Stream documentContents, string documentUri) + { + // Open the document. + using WordprocessingDocument wordDoc = WordprocessingDocument.Open(documentContents, false); + if (wordDoc.MainDocumentPart == null) + { + yield break; + } + + // Create an XmlDocument to hold the document contents + // and load the document contents into the XmlDocument. + XmlDocument xmlDoc = new(); + XmlNamespaceManager nsManager = new(xmlDoc.NameTable); + nsManager.AddNamespace("w", "http://schemas.openxmlformats.org/wordprocessingml/2006/main"); + nsManager.AddNamespace("w14", "http://schemas.microsoft.com/office/word/2010/wordml"); + + xmlDoc.Load(wordDoc.MainDocumentPart.GetStream()); + + // Select all paragraphs in the document and break if none found. + XmlNodeList? paragraphs = xmlDoc.SelectNodes("//w:p", nsManager); + if (paragraphs == null) + { + yield break; + } + + // Iterate over each paragraph. + foreach (XmlNode paragraph in paragraphs) + { + // Select all text nodes in the paragraph and continue if none found. + XmlNodeList? texts = paragraph.SelectNodes(".//w:t", nsManager); + if (texts == null) + { + continue; + } + + // Combine all non-empty text nodes into a single string. + var textBuilder = new StringBuilder(); + foreach (XmlNode text in texts) + { + if (!string.IsNullOrWhiteSpace(text.InnerText)) + { + textBuilder.Append(text.InnerText); + } + } + + // Yield a new TextParagraph if the combined text is not empty. + var combinedText = textBuilder.ToString(); + if (!string.IsNullOrWhiteSpace(combinedText)) + { + Console.WriteLine("Found paragraph:"); + Console.WriteLine(combinedText); + Console.WriteLine(); + + yield return new TextParagraph + { + Key = Guid.NewGuid().ToString(), + DocumentUri = documentUri, + ParagraphId = paragraph.Attributes?["w14:paraId"]?.Value ?? string.Empty, + Text = combinedText + }; + } + } + } +} +// + +// +using Microsoft.Extensions.AI; +using Microsoft.Extensions.VectorData; + +namespace SKVectorIngest; + +internal class DataUploader(VectorStore vectorStore, IEmbeddingGenerator> embeddingGenerator) +{ + /// + /// Generate an embedding for each text paragraph and upload it to the specified collection. + /// + /// The name of the collection to upload the text paragraphs to. + /// The text paragraphs to upload. + /// An async task. + public async Task GenerateEmbeddingsAndUpload(string collectionName, IEnumerable textParagraphs) + { + var collection = vectorStore.GetCollection(collectionName); + await collection.EnsureCollectionExistsAsync(); + + foreach (var paragraph in textParagraphs) + { + // Generate the text embedding. + Console.WriteLine($"Generating embedding for paragraph: {paragraph.ParagraphId}"); + paragraph.TextEmbedding = (await embeddingGenerator.GenerateEmbeddingAsync(paragraph.Text)).Vector; + + // Upload the text paragraph. + Console.WriteLine($"Upserting paragraph: {paragraph.ParagraphId}"); + await collection.UpsertAsync(paragraph); + + Console.WriteLine(); + } + } +} +// + +// +using Azure; +using Azure.AI.OpenAI; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.Redis; +using SKVectorIngest; + +// Replace with your values. +var deploymentName = "text-embedding-ada-002"; +var endpoint = "https://sksample.openai.azure.com/"; +var apiKey = "your-api-key"; + +// Register Azure OpenAI embedding generator and Redis vector store. +var services = new ServiceCollection(); +services.AddSingleton>>( + new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey)) + .GetEmbeddingClient(deploymentName) + .AsIEmbeddingGenerator()); + +services.AddRedisVectorStore("localhost:6379"); + +// Register the data uploader. +services.AddSingleton(); + +// Build the service provider and get the data uploader. +var serviceProvider = services.BuildServiceProvider(); +var dataUploader = serviceProvider.GetRequiredService(); +// + +// +// Load the data. +var textParagraphs = DocumentReader.ReadParagraphs( + new FileStream( + "vector-store-data-ingestion-input.docx", + FileMode.Open), + "file:///c:/vector-store-data-ingestion-input.docx"); + +await dataUploader.GenerateEmbeddingsAndUpload( + "sk-documentation", + textParagraphs); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/how-to/use-vector-stores.md b/docs/ai/vector-stores/how-to/use-vector-stores.md index 36e34286cd623..8fc7d6f1262fb 100644 --- a/docs/ai/vector-stores/how-to/use-vector-stores.md +++ b/docs/ai/vector-stores/how-to/use-vector-stores.md @@ -141,52 +141,13 @@ The following table describes the available options: When your data model has multiple vector properties, use `VectorProperty` to specify which one to search: -```csharp -// A data model with two vector properties. -public class Hotel -{ - [VectorStoreKey] - public int HotelId { get; set; } - - [VectorStoreData] - public string? HotelName { get; set; } - - // Two separate embeddings for different aspects of the hotel. - [VectorStoreVector(1536)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory? AmenitiesEmbedding { get; set; } -} - -// Target the amenities embedding specifically. -var options = new VectorSearchOptions -{ - VectorProperty = r => r.AmenitiesEmbedding -}; - -IAsyncEnumerable> results = - collection.SearchAsync(queryEmbedding, top: 3, options); -``` +:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="TargetVectorProperty"::: ## Use built-in embedding generation Instead of generating embeddings manually before each upsert, you can configure an `IEmbeddingGenerator` on the vector store or collection. When you do, declare your vector property as a `string` type (the source text) and the store generates the embedding automatically. -```csharp -public class FinanceInfo -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData] - public string Text { get; set; } = ""; - - // Use a string type to trigger automatic embedding generation on upsert. - [VectorStoreVector(1536)] - public string EmbeddingSource { get; set; } = ""; -} -``` +:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="EmbeddingSourceModel"::: Then configure the embedding generator when creating the vector store: @@ -203,25 +164,7 @@ Some vector stores support *hybrid search*, which combines vector similarity wit To use hybrid search, check whether your collection implements . Only connectors for databases that support this feature implement this interface. -```csharp -// Check whether the collection supports hybrid search. -if (collection is IKeywordHybridSearchable hybridCollection) -{ - ReadOnlyMemory queryEmbedding = - await embeddingGenerator.GenerateVectorAsync("peaceful beachfront hotel"); - - // Provide both a vector and keywords for hybrid search. - var results = hybridCollection.HybridSearchAsync( - queryEmbedding, - keywords: ["ocean", "beach"], - top: 3); - - await foreach (var result in results) - { - Console.WriteLine($"Hotel: {result.Record.HotelName}, Score: {result.Score}"); - } -} -``` +:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="HybridSearch"::: For hybrid search to work, the data model must have at least one vector property and one text property with `IsFullTextIndexed = true`. For a list of connectors that support hybrid search, see the documentation for each connector. @@ -243,13 +186,7 @@ Because all connectors implement the same A unique key for the text paragraph. - [VectorStoreKey] - public required string Key { get; init; } - - /// A uri that points at the original location of the document containing the text. - [VectorStoreData] - public required string DocumentUri { get; init; } - - /// The id of the paragraph from the document containing the text. - [VectorStoreData] - public required string ParagraphId { get; init; } - - /// The text of the paragraph. - [VectorStoreData] - public required string Text { get; init; } - - /// The embedding generated from the Text. - [VectorStoreVector(1536)] - public ReadOnlyMemory TextEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="AddADataModel"::: The value `1536`, which is the dimension size of the vector, is passed to the . This value must match the size of vector that your chosen embedding generator produces. @@ -93,80 +66,7 @@ Next, you add the code to read the Word document and find the text of each parag Add a new file to the project called `DocumentReader.cs` and add the following class to read the paragraphs from a document. -```csharp -using System.Text; -using System.Xml; -using DocumentFormat.OpenXml.Packaging; - -namespace SKVectorIngest; - -internal class DocumentReader -{ - public static IEnumerable ReadParagraphs(Stream documentContents, string documentUri) - { - // Open the document. - using WordprocessingDocument wordDoc = WordprocessingDocument.Open(documentContents, false); - if (wordDoc.MainDocumentPart == null) - { - yield break; - } - - // Create an XmlDocument to hold the document contents - // and load the document contents into the XmlDocument. - XmlDocument xmlDoc = new(); - XmlNamespaceManager nsManager = new(xmlDoc.NameTable); - nsManager.AddNamespace("w", "http://schemas.openxmlformats.org/wordprocessingml/2006/main"); - nsManager.AddNamespace("w14", "http://schemas.microsoft.com/office/word/2010/wordml"); - - xmlDoc.Load(wordDoc.MainDocumentPart.GetStream()); - - // Select all paragraphs in the document and break if none found. - XmlNodeList? paragraphs = xmlDoc.SelectNodes("//w:p", nsManager); - if (paragraphs == null) - { - yield break; - } - - // Iterate over each paragraph. - foreach (XmlNode paragraph in paragraphs) - { - // Select all text nodes in the paragraph and continue if none found. - XmlNodeList? texts = paragraph.SelectNodes(".//w:t", nsManager); - if (texts == null) - { - continue; - } - - // Combine all non-empty text nodes into a single string. - var textBuilder = new StringBuilder(); - foreach (XmlNode text in texts) - { - if (!string.IsNullOrWhiteSpace(text.InnerText)) - { - textBuilder.Append(text.InnerText); - } - } - - // Yield a new TextParagraph if the combined text is not empty. - var combinedText = textBuilder.ToString(); - if (!string.IsNullOrWhiteSpace(combinedText)) - { - Console.WriteLine("Found paragraph:"); - Console.WriteLine(combinedText); - Console.WriteLine(); - - yield return new TextParagraph - { - Key = Guid.NewGuid().ToString(), - DocumentUri = documentUri, - ParagraphId = paragraph.Attributes?["w14:paraId"]?.Value ?? string.Empty, - Text = combinedText - }; - } - } - } -} -``` +:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="ReadTheParagraphsInTheDocument"::: ## Generate embeddings and upload the data @@ -174,40 +74,7 @@ Next, you add a new class to generate embeddings and upload the paragraphs to Re Add a new file called `DataUploader.cs` with the following contents: -```csharp -using Microsoft.Extensions.AI; -using Microsoft.Extensions.VectorData; - -namespace SKVectorIngest; - -internal class DataUploader(VectorStore vectorStore, IEmbeddingGenerator> embeddingGenerator) -{ - /// - /// Generate an embedding for each text paragraph and upload it to the specified collection. - /// - /// The name of the collection to upload the text paragraphs to. - /// The text paragraphs to upload. - /// An async task. - public async Task GenerateEmbeddingsAndUpload(string collectionName, IEnumerable textParagraphs) - { - var collection = vectorStore.GetCollection(collectionName); - await collection.EnsureCollectionExistsAsync(); - - foreach (var paragraph in textParagraphs) - { - // Generate the text embedding. - Console.WriteLine($"Generating embedding for paragraph: {paragraph.ParagraphId}"); - paragraph.TextEmbedding = (await embeddingGenerator.GenerateEmbeddingAsync(paragraph.Text)).Vector; - - // Upload the text paragraph. - Console.WriteLine($"Upserting paragraph: {paragraph.ParagraphId}"); - await collection.UpsertAsync(paragraph); - - Console.WriteLine(); - } - } -} -``` +:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="GenerateEmbeddingsAndUploadTheData"::: ## Put it all together @@ -216,52 +83,12 @@ In this example, you use standard .NET dependency injection to register the Redi Add the following code to your `Program.cs` file to set up the container, register the Redis vector store, and register the embedding service. Replace the text embedding generation settings with your own values. -```csharp -using Azure; -using Azure.AI.OpenAI; -using Microsoft.Extensions.AI; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.Redis; -using SKVectorIngest; - -// Replace with your values. -var deploymentName = "text-embedding-ada-002"; -var endpoint = "https://sksample.openai.azure.com/"; -var apiKey = "your-api-key"; - -// Register Azure OpenAI embedding generator and Redis vector store. -var services = new ServiceCollection(); -services.AddSingleton>>( - new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey)) - .GetEmbeddingClient(deploymentName) - .AsIEmbeddingGenerator()); - -services.AddRedisVectorStore("localhost:6379"); - -// Register the data uploader. -services.AddSingleton(); - -// Build the service provider and get the data uploader. -var serviceProvider = services.BuildServiceProvider(); -var dataUploader = serviceProvider.GetRequiredService(); -``` +:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="PutItAllTogether1"::: Lastly, add code to read the paragraphs from the Word document and call the data uploader to generate the embeddings and upload the paragraphs. -```csharp -// Load the data. -var textParagraphs = DocumentReader.ReadParagraphs( - new FileStream( - "vector-store-data-ingestion-input.docx", - FileMode.Open), - "file:///c:/vector-store-data-ingestion-input.docx"); - -await dataUploader.GenerateEmbeddingsAndUpload( - "sk-documentation", - textParagraphs); -``` +:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="PutItAllTogether2"::: ## See your data in Redis diff --git a/docs/ai/vector-stores/hybrid-search.md b/docs/ai/vector-stores/hybrid-search.md index 55cefd1e66bb2..23dbf07ca707c 100644 --- a/docs/ai/vector-stores/hybrid-search.md +++ b/docs/ai/vector-stores/hybrid-search.md @@ -29,34 +29,7 @@ Only connectors for databases that currently support vector plus keyword hybrid Assuming you have a collection that already contains data, you can easily do a hybrid search on it. Here is an example using Qdrant. -```csharp -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Microsoft.Extensions.VectorData; -using Qdrant.Client; - -// Placeholder embedding generation method. -async Task> GenerateEmbeddingAsync(string textToVectorize) -{ - // your logic here -} - -// Create a Qdrant VectorStore object and choose an existing collection that already contains records. -VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -IKeywordHybridSearchable collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skhotels"); - -// Generate a vector for your search text, using your chosen embedding generation implementation. -ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); - -// Do the search, passing an options object with a Top value to limit results to the single top match. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 1); - -// Inspect the returned hotel. -await foreach (var record in searchResult) -{ - Console.WriteLine("Found hotel description: " + record.Record.Description); - Console.WriteLine("Found record score: " + record.Score); -} -``` +:::code language="csharp" source="./snippets/hybrid-search.cs" id="HybridSearch"::: > [!TIP] > For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). @@ -86,47 +59,7 @@ If the data model contains no vector or multiple vectors and `VectorProperty` is If no `AdditionalProperty` is provided and the data model contains only one full text search property, that property will be used. If the data model contains no full text search property or multiple full text search properties and `AdditionalProperty` is not provided, the search method will throw. -```csharp -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Microsoft.Extensions.VectorData; -using Qdrant.Client; - -var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -var collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skproducts"); - -// Create the hybrid search options and indicate that you want -// to search the DescriptionEmbedding vector property and the -// Description full text search property. -var hybridSearchOptions = new HybridSearchOptions -{ - VectorProperty = r => r.DescriptionEmbedding, - AdditionalProperty = r => r.Description -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); - -public sealed class Product -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Name { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreData] - public List FeatureList { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DescriptionEmbedding { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory FeatureListEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/hybrid-search.cs" id="VectorPropertyAndAdditionalProperty"::: ### `Top` and `Skip` @@ -134,22 +67,7 @@ The `Top` and `Skip` options allow you to limit the number of results to the Top to skip a number of results from the top of the resultset. Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. -```csharp -// Create the vector search options and indicate that you want to skip the first 40 results and then pass 20 to search to get the next 20. -var hybridSearchOptions = new HybridSearchOptions -{ - Skip = 40 -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 20, hybridSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.Description); -} -``` +:::code language="csharp" source="./snippets/hybrid-search.cs" id="TopAndSkip"::: The default values for `Skip` is 0. @@ -162,22 +80,7 @@ making searches more efficient. The default value for `IncludeVectors` is `false`. -```csharp -// Create the hybrid search options and indicate that you want to include vectors in the search results. -var hybridSearchOptions = new HybridSearchOptions -{ - IncludeVectors = true -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.FeatureList); -} -``` +:::code language="csharp" source="./snippets/hybrid-search.cs" id="IncludeVectors"::: ### Filter @@ -202,42 +105,4 @@ Filters are expressed using LINQ expressions based on the type of the data model The set of LINQ expressions supported will vary depending on the functionality supported by each database, but all databases support a broad base of common expressions, for example, `equals`, `not equals`, `and`, and `or`. -```csharp -// Create the hybrid search options and set the filter on the options. -var hybridSearchOptions = new HybridSearchOptions -{ - Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.Definition); -} - -sealed class Glossary -{ - [VectorStoreKey] - public ulong Key { get; set; } - - // Category is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public string Category { get; set; } - - // Tags is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public List Tags { get; set; } - - [VectorStoreData] - public string Term { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Definition { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DefinitionEmbedding { get; set; } -} -``` \ No newline at end of file +:::code language="csharp" source="./snippets/hybrid-search.cs" id="Filter"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md index 654f43d28d64d..949bc01617432 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md @@ -46,82 +46,23 @@ dotnet add package Microsoft.SemanticKernel.Connectors.AzureAISearch --prereleas You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Azure; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); -``` - -```csharp -using Azure; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted1"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); -``` +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of the Azure AI Search `SearchIndexClient` to be separately registered with the dependency injection container. -```csharp -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret))); -services.AddAzureAISearchVectorStore(); -``` +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted3"::: -```csharp -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret))); -builder.Services.AddAzureAISearchVectorStore(); -``` +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted4"::: You can construct an Azure AI Search Vector Store instance directly. -```csharp -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.SemanticKernel.Connectors.AzureAISearch; - -var vectorStore = new AzureAISearchVectorStore( - new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret))); -``` +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.SemanticKernel.Connectors.AzureAISearch; - -var collection = new AzureAISearchCollection( - new SearchIndexClient(new Uri(azureAISearchUri), new AzureKeyCredential(secret)), - "skhotels"); -``` +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted6"::: ## Data mapping @@ -134,13 +75,4 @@ data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` must be passed to both the `SearchIndexClient` and the `AzureAISearchCollection` on construction. -```csharp -var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; -var collection = new AzureAISearchCollection( - new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret), - new() { Serializer = new JsonObjectSerializer(jsonSerializerOptions) }), - "skhotels", - new() { JsonSerializerOptions = jsonSerializerOptions }); -``` \ No newline at end of file +:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="DataMapping"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md index f3d20baaa305c..7604358924f6e 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md @@ -42,81 +42,23 @@ dotnet add package Microsoft.SemanticKernel.Connectors.CosmosMongoDB --prereleas You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddCosmosMongoVectorStore(connectionString, databaseName); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddCosmosMongoVectorStore(connectionString, databaseName); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of `MongoDB.Driver.IMongoDatabase` to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using MongoDB.Driver; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => - { - var mongoClient = new MongoClient(connectionString); - return mongoClient.GetDatabase(databaseName); - }); -services.AddCosmosMongoVectorStore(); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted3"::: -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using MongoDB.Driver; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => - { - var mongoClient = new MongoClient(connectionString); - return mongoClient.GetDatabase(databaseName); - }); -builder.Services.AddCosmosMongoVectorStore(); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted4"::: You can construct an Azure CosmosDB MongoDB Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var vectorStore = new CosmosMongoVectorStore(database); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var collection = new CosmosMongoCollection( - database, - "skhotels"); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted6"::: ## Data mapping @@ -132,24 +74,4 @@ The property name override is done by setting the `BsonElement` attribute on the Here is an example of a data model with `BsonElement` set. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [BsonElement("hotel_name")] - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [BsonElement("hotel_description")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [BsonElement("hotel_description_embedding")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` \ No newline at end of file +:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="PropertyNameOverride"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md index 30648bd7785d5..7045f8622a420 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md @@ -32,12 +32,7 @@ The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage When initializing `CosmosClient` manually, it is necessary to specify `CosmosClientOptions.UseSystemTextJsonSerializerWithOptions` due to limitations in the default serializer. This option can be set to `JsonSerializerOptions.Default` or customized with other serializer options to meet specific configuration needs. -```csharp -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, -}); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="Limitations"::: ## Get started @@ -49,109 +44,23 @@ dotnet add package Microsoft.SemanticKernel.Connectors.CosmosNoSql --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddCosmosNoSqlVectorStore(connectionString, databaseName); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted1"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddCosmosNoSqlVectorStore(connectionString, databaseName); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of `Microsoft.Azure.Cosmos.Database` to be separately registered with the dependency injection container. -```csharp -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => - { - var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() - { - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, - }); - - return cosmosClient.GetDatabase(databaseName); - }); -services.AddCosmosNoSqlVectorStore(); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted3"::: -```csharp -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => - { - var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() - { - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, - }); - - return cosmosClient.GetDatabase(databaseName); - }); -builder.Services.AddCosmosNoSqlVectorStore(); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted4"::: You can construct an Azure CosmosDB NoSQL Vector Store instance directly. -```csharp -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.SemanticKernel.Connectors.CosmosNoSql; - -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, -}); - -var database = cosmosClient.GetDatabase(databaseName); -var vectorStore = new CosmosNoSqlVectorStore(database); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.SemanticKernel.Connectors.CosmosNoSql; - -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, -}); - -var database = cosmosClient.GetDatabase(databaseName); -var collection = new CosmosNoSqlCollection( - database, - "skhotels"); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted6"::: ## Data mapping @@ -165,49 +74,11 @@ records must use this name for ids. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` must be passed to the `CosmosNoSqlCollection` on construction. -```csharp -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.SemanticKernel.Connectors.CosmosNoSql; - -var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; - -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = jsonSerializerOptions -}); - -var database = cosmosClient.GetDatabase(databaseName); -var collection = new CosmosNoSqlCollection( - database, - "skhotels", - new() { JsonSerializerOptions = jsonSerializerOptions }); -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="DataMapping1"::: Using the above custom `JsonSerializerOptions` which is using `SnakeCaseUpper`, the following data model will be mapped to the below json. -```csharp -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.EuclideanDistance, IndexKind = IndexKind.QuantizedFlat)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="DataMapping2"::: ```json { @@ -228,18 +99,8 @@ If the partition key property is not set (and the default key property is used), Specify partition key property name: -```csharp -var options = new CosmosNoSqlCollectionOptions -{ - PartitionKeyPropertyName = nameof(Hotel.HotelName) -}; - -var collection = new CosmosNoSqlCollection(database, "collection-name", options) - as VectorStoreCollection; -``` +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="UsingPartitionKey1"::: Get with partition key: -```csharp -var record = await collection.GetAsync(new CosmosNoSqlCompositeKey("hotel-id", "hotel-name")); -``` \ No newline at end of file +:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="UsingPartitionKey2"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md index a8461b71b4c61..e520824397d65 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md @@ -40,165 +40,26 @@ dotnet add package CouchbaseConnector.SemanticKernel --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.SemanticKernel; -using Couchbase.SemanticKernel; - -// Using a ServiceCollection. -var kernelBuilder = Kernel - .CreateBuilder() - .AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name"); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="GetStarted1"::: -```csharp -using Couchbase.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name"); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="GetStarted2"::: ## Configure index type The vector store defaults to using Hyperscale indexes. You can specify a different index type by passing `CouchbaseVectorStoreOptions`: -```csharp -using Couchbase.SemanticKernel; - -var builder = WebApplication.CreateBuilder(args); - -// Option 1: Use Hyperscale index -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name", - options: new CouchbaseVectorStoreOptions - { - IndexType = CouchbaseIndexType.Hyperscale - }); - -// Option 2: Use Composite index -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name", - options: new CouchbaseVectorStoreOptions - { - IndexType = CouchbaseIndexType.Composite - }); - -// Option 3: Use Search vector index -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name", - options: new CouchbaseVectorStoreOptions - { - IndexType = CouchbaseIndexType.Search - }); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType1"::: Extension methods that take no parameters are also provided. These require an instance of the `IScope` class to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Couchbase; -using Couchbase.KeyValue; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => -{ - var clusterOptions = new ClusterOptions - { - ConnectionString = "couchbases://your-cluster-address", - UserName = "username", - Password = "password" - }; - - return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); -}); - -services.AddSingleton(sp => -{ - var cluster = sp.GetRequiredService(); - var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); - return bucket.Scope("scope-name"); -}); - -// Add Couchbase Vector Store -services.AddCouchbaseVectorStore(); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Couchbase.KeyValue; -using Couchbase; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); - -builder.Services.AddSingleton(sp => -{ - var clusterOptions = new ClusterOptions - { - ConnectionString = "couchbases://your-cluster-address", - UserName = "username", - Password = "password" - }; - - return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); -}); +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType2"::: -builder.Services.AddSingleton(sp => -{ - var cluster = sp.GetRequiredService(); - var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); - return bucket.Scope("scope-name"); -}); - -// Add Couchbase Vector Store -builder.Services.AddCouchbaseVectorStore(); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType3"::: You can construct a Couchbase Vector Store instance directly. -```csharp -using Couchbase; -using Couchbase.KeyValue; -using Couchbase.SemanticKernel; - -var clusterOptions = new ClusterOptions -{ - ConnectionString = "couchbases://your-cluster-address", - UserName = "username", - Password = "password" -}; - -var cluster = await Cluster.ConnectAsync(clusterOptions); -var bucket = await cluster.BucketAsync("bucket-name"); -var scope = bucket.Scope("scope-name"); - -var vectorStore = new CouchbaseVectorStore(scope); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType4"::: It's possible to construct a direct reference to a named collection. @@ -206,45 +67,13 @@ It's possible to construct a direct reference to a named collection. For high-performance vector search with Hyperscale indexes: -```csharp -using Couchbase.SemanticKernel; -using Couchbase; -using Couchbase.KeyValue; - -var cluster = await Cluster.ConnectAsync(clusterOptions); -var bucket = await cluster.BucketAsync("bucket-name"); -var scope = bucket.Scope("scope-name"); - -// Using Hyperscale index (default) -var collection = new CouchbaseQueryCollection( - scope, - "skhotels", - indexType: CouchbaseIndexType.Hyperscale); - -// Or using Composite index -var collectionComposite = new CouchbaseQueryCollection( - scope, - "skhotels", - indexType: CouchbaseIndexType.Composite); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="UseQueryCollectionHyperscaleOrCompositeI"::: ### Use search collection (seach vector index) For hybrid search scenarios combining full-text search: -```csharp -using Couchbase.SemanticKernel; -using Couchbase; -using Couchbase.KeyValue; - -var cluster = await Cluster.ConnectAsync(clusterOptions); -var bucket = await cluster.BucketAsync("bucket-name"); -var scope = bucket.Scope("scope-name"); - -var collection = new CouchbaseSearchCollection( - scope, - "skhotels"); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="UseSearchCollectionSeachVectorIndex"::: ### Index type comparison @@ -290,46 +119,11 @@ The Couchbase connector will use `System.Text.Json.JsonSerializer` to do mapping Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. -```csharp -using Couchbase.SemanticKernel; -using Couchbase.KeyValue; -using System.Text.Json; - -var jsonSerializerOptions = new JsonSerializerOptions -{ - PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper -}; - -var options = new CouchbaseQueryCollectionOptions -{ - JsonSerializerOptions = jsonSerializerOptions -}; - -var collection = new CouchbaseQueryCollection(scope, "skhotelsjson", options); -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="DataMapping1"::: Since a naming policy of snake case upper was chosen, here is an example of how this data type will be stored in Couchbase. Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. -```csharp -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/couchbase-connector.cs" id="DataMapping2"::: ```json { diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md index a254eed5a0d9c..8eef60cf39da0 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md @@ -45,71 +45,23 @@ dotnet add package Elastic.SemanticKernel.Connectors.Elasticsearch --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; - -// Using a ServiceCollection. -var kernelBuilder = Kernel - .CreateBuilder() - .AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); -``` - -```csharp -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted1"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of the `Elastic.Clients.Elasticsearch.ElasticsearchClient` class to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted3"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); -services.AddElasticsearchVectorStore(); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); -builder.Services.AddElasticsearchVectorStore(); -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted4"::: You can construct an Elasticsearch Vector Store instance directly. -```csharp -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; - -var vectorStore = new ElasticsearchVectorStore( - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; - -var collection = new ElasticsearchVectorStoreRecordCollection( - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))), - "skhotels"); -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted6"::: ## Data mapping @@ -121,63 +73,16 @@ Usage of the `JsonPropertyNameAttribute` is supported if a different storage nam data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, a custom source serializer must be configured. -```csharp -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; -using Elastic.Transport; - -var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); -var settings = new ElasticsearchClientSettings( - nodePool, - sourceSerializer: (defaultSerializer, settings) => - new DefaultSourceSerializer(settings, options => - options.PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper)); -var client = new ElasticsearchClient(settings); - -var collection = new ElasticsearchVectorStoreRecordCollection( - client, - "skhotelsjson"); -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="DataMapping1"::: As an alternative, the `DefaultFieldNameInferrer` lambda function can be configured to achieve the same result or to even further customize property naming based on dynamic conditions. -```csharp -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; - -var settings = new ElasticsearchClientSettings(new Uri("http://localhost:9200")); -settings.DefaultFieldNameInferrer(name => JsonNamingPolicy.SnakeCaseUpper.ConvertName(name)); -var client = new ElasticsearchClient(settings); - -var collection = new ElasticsearchVectorStoreRecordCollection( - client, - "skhotelsjson"); -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="DataMapping2"::: Since a naming policy of snake case upper was chosen, here is an example of how this data type will be set in Elasticsearch. Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. -```csharp -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity, IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="DataMapping3"::: ```json { diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md index 8f9b6616bed2d..9aacc23dd9b21 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md @@ -41,36 +41,14 @@ dotnet add package Microsoft.SemanticKernel.Connectors.InMemory --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddInMemoryVectorStore(); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddInMemoryVectorStore(); -``` +:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted2"::: You can construct an InMemory Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.InMemory; - -var vectorStore = new InMemoryVectorStore(); -``` +:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted3"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.InMemory; - -var collection = new InMemoryCollection("skhotels"); -``` \ No newline at end of file +:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted4"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md index ba56bfb7d0859..0aeb050152a29 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md @@ -40,54 +40,19 @@ dotnet add package Microsoft.SemanticKernel.Connectors.MongoDB --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddMongoVectorStore(connectionString, databaseName); -``` +:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted1"::: Extension methods that take no parameters are also provided. These require an instance of `MongoDB.Driver.IMongoDatabase` to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using MongoDB.Driver; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => - { - var mongoClient = new MongoClient(connectionString); - return mongoClient.GetDatabase(databaseName); - }); -builder.Services.AddMongoVectorStore(); -``` +:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted2"::: You can construct a MongoDB Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.MongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var vectorStore = new MongoVectorStore(database); -``` +:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted3"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.MongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var collection = new MongoCollection( - database, - "skhotels"); -``` +:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted4"::: ## Data mapping @@ -107,25 +72,4 @@ The property name override is done by setting the `BsonElement` attribute on the Here is an example of a data model with `BsonElement` set. -```csharp -using Microsoft.Extensions.VectorData; -using MongoDB.Bson.Serialization.Attributes; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [BsonElement("hotel_name")] - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [BsonElement("hotel_description")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [BsonElement("hotel_description_embedding")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` \ No newline at end of file +:::code language="csharp" source="./snippets/mongodb-connector.cs" id="PropertyNameOverride"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md index f8aedecab0eb6..1ed5224892065 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md @@ -42,95 +42,27 @@ dotnet add package Oracle.VectorData --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. In this case, an instance of the `Oracle.VectorData.OracleVectorStore` class also gets registered with the container. -```csharp -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddOracleVectorStore(""); -``` - -```csharp -using Microsoft.AspNetCore.Builder; -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddOracleVectorStore(""); -``` +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also available. These require an instance of the `Oracle.ManagedDataAccess.Client.OracleDataSource` class to be separately registered with the dependency injection container. -```csharp -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; -using Oracle.ManagedDataAccess.Client; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => -{ - OracleDataSourceBuilder dataSourceBuilder = new(""); - return dataSourceBuilder.Build(); -}); - -services.AddOracleVectorStore(); -``` +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted3"::: -```csharp -using Microsoft.AspNetCore.Builder; -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; -using Oracle.ManagedDataAccess.Client; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => -{ - OracleDataSourceBuilder dataSourceBuilder = new(""); - return dataSourceBuilder.Build(); -}); - -builder.Services.AddOracleVectorStore(); -``` +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted4"::: You can construct an Oracle Database Vector Store instance directly with a custom data source or with a connection string. -```csharp -using Oracle.VectorData; -using Oracle.ManagedDataAccess.Client; - -OracleDataSourceBuilder dataSourceBuilder = new(""); -var dataSource = dataSourceBuilder.Build(); +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted5"::: -var connection = new OracleVectorStore(dataSource); -``` - -```csharp -using Oracle.VectorData; - -var connection = new OracleVectorStore(""); -``` +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted6"::: It's possible to construct a direct reference to a named collection with a custom data source or with a connection string. -```csharp -using Oracle.VectorData; -using Oracle.ManagedDataAccess.Client; - -OracleDataSourceBuilder dataSourceBuilder = new(""); -var dataSource = dataSourceBuilder.Build(); - -var collection = new OracleCollection(dataSource, "skhotels"); -``` - -```csharp -using Oracle.VectorData; +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted7"::: -var collection = new OracleCollection("", "skhotels"); -``` +:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted8"::: ## Data mapping @@ -184,24 +116,7 @@ For data properties and vector properties, you can override names to use in stor Here is a data model with set code sample and how that will be represented in an Oracle SQL command. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public long HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string? HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 384, DistanceFunction = DistanceFunction.CosineDistance)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/oracle-connector.cs" id="PropertyNameOverride"::: ```SQL CREATE TABLE "MYSCHEMA"."Hotels" diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md index 7c62105cc0b51..bcd71209f591d 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md @@ -39,69 +39,23 @@ dotnet add package Microsoft.SemanticKernel.Connectors.Pinecone --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddPineconeVectorStore(pineconeApiKey); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddPineconeVectorStore(pineconeApiKey); -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of the `PineconeClient` to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using PineconeClient = Pinecone.PineconeClient; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => new PineconeClient(pineconeApiKey)); -services.AddPineconeVectorStore(); -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted3"::: -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using PineconeClient = Pinecone.PineconeClient; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => new PineconeClient(pineconeApiKey)); -builder.Services.AddPineconeVectorStore(); -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted4"::: You can construct a Pinecone Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.Pinecone; -using PineconeClient = Pinecone.PineconeClient; - -var vectorStore = new PineconeVectorStore( - new PineconeClient(pineconeApiKey)); -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.Pinecone; -using PineconeClient = Pinecone.PineconeClient; - -var collection = new PineconeCollection( - new PineconeClient(pineconeApiKey), - "skhotels"); -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted6"::: ## Index namespace @@ -111,15 +65,7 @@ and no second level exists in the abstraction. Pinecone does support a second le By default the Pinecone connector will pass null as the namespace for all operations. However it is possible to pass a single namespace to the Pinecone collection when constructing it and use this instead for all operations. -```csharp -using Microsoft.SemanticKernel.Connectors.Pinecone; -using PineconeClient = Pinecone.PineconeClient; - -var collection = new PineconeCollection( - new PineconeClient(pineconeApiKey), - "skhotels", - new() { IndexNamespace = "seasidehotels" }); -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="IndexNamespace"::: ## Data mapping @@ -140,24 +86,7 @@ The property name override is done by setting the set on its attributes and how that will be represented in Pinecone. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/pinecone-connector.cs" id="PropertyNameOverride"::: ```json { diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md index 4170a219471a0..52a3e6c94e2b2 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md @@ -37,11 +37,7 @@ The connector has the following characteristics. Here is an example of how to call `UseVector`. -```csharp -NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); -dataSourceBuilder.UseVector(); -NpgsqlDataSource dataSource = dataSourceBuilder.Build(); -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="Limitations"::: When using the `AddPostgresVectorStore` dependency injection registration method with a connection string, the datasource will be constructed by this method and will automatically have `UseVector` applied. @@ -55,78 +51,27 @@ dotnet add package Microsoft.SemanticKernel.Connectors.PgVector --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; - -var services = new ServiceCollection(); -services.AddPostgresVectorStore(""); -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted1"::: Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects, for example `Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres`. Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that `UseVector` must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted2"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddPostgresVectorStore(""); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Npgsql; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => -{ - NpgsqlDataSourceBuilder dataSourceBuilder = new(""); - dataSourceBuilder.UseVector(); - return dataSourceBuilder.Build(); -}); -builder.Services.AddPostgresVectorStore(); -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted3"::: You can construct a Postgres Vector Store instance directly with a custom data source or with a connection string. -```csharp -using Microsoft.SemanticKernel.Connectors.PgVector; -using Npgsql; +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted4"::: -NpgsqlDataSourceBuilder dataSourceBuilder = new(""); -dataSourceBuilder.UseVector(); -NpgsqlDataSource dataSource = dataSourceBuilder.Build(); -var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); -``` - -```csharp -using Microsoft.SemanticKernel.Connectors.PgVector; - -var connection = new PostgresVectorStore(""); -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection with a custom data source or with a connection string. -```csharp -using Microsoft.SemanticKernel.Connectors.PgVector; -using Npgsql; - -NpgsqlDataSourceBuilder dataSourceBuilder = new(""); -dataSourceBuilder.UseVector(); -var dataSource = dataSourceBuilder.Build(); - -var collection = new PostgresCollection(dataSource, "skhotels", ownsDataSource: true); -``` - -```csharp -using Microsoft.SemanticKernel.Connectors.PgVector; +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted6"::: -var collection = new PostgresCollection("", "skhotels"); -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted7"::: ## Data mapping @@ -148,25 +93,7 @@ The property name override is done by setting the set on its attributes and how it will be represented in Postgres as a table, assuming the Collection name is `Hotels`. -```csharp -using System; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey(StorageName = "hotel_id")] - public int HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="PropertyNameOverride"::: ```sql CREATE TABLE IF NOT EXISTS public."Hotels" ( @@ -192,136 +119,10 @@ Azure Database for PostgreSQL provides the ability to connect to your database u This removes the need to store a username and password in your connection string. To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following Npgsql extension method and set a connection string that does not have a username or password: -```csharp -using System.Text; -using System.Text.Json; -using Azure.Core; -using Azure.Identity; -using Npgsql; - -namespace Program; - -public static class NpgsqlDataSourceBuilderExtensions -{ - private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new(["https://ossrdbms-aad.database.windows.net/.default"]); - - public static NpgsqlDataSourceBuilder UseEntraAuthentication(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default) - { - credential ??= new DefaultAzureCredential(); - - if (dataSourceBuilder.ConnectionStringBuilder.Username == null) - { - var token = credential.GetToken(s_azureDBForPostgresTokenRequestContext, default); - SetUsernameFromToken(dataSourceBuilder, token.Token); - } - - SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); - - return dataSourceBuilder; - } - - public static async Task UseEntraAuthenticationAsync(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default, CancellationToken cancellationToken = default) - { - credential ??= new DefaultAzureCredential(); - - if (dataSourceBuilder.ConnectionStringBuilder.Username == null) - { - var token = await credential.GetTokenAsync(s_azureDBForPostgresTokenRequestContext, cancellationToken).ConfigureAwait(false); - SetUsernameFromToken(dataSourceBuilder, token.Token); - } - - SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); - - return dataSourceBuilder; - } - - private static void SetPasswordProvider(NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential credential, TokenRequestContext tokenRequestContext) - { - dataSourceBuilder.UsePasswordProvider(_ => - { - var token = credential.GetToken(tokenRequestContext, default); - return token.Token; - }, async (_, ct) => - { - var token = await credential.GetTokenAsync(tokenRequestContext, ct).ConfigureAwait(false); - return token.Token; - }); - } - - private static void SetUsernameFromToken(NpgsqlDataSourceBuilder dataSourceBuilder, string token) - { - var username = TryGetUsernameFromToken(token); - - if (username != null) - { - dataSourceBuilder.ConnectionStringBuilder.Username = username; - } - else - { - throw new Exception("Could not determine username from token claims"); - } - } - - private static string? TryGetUsernameFromToken(string jwtToken) - { - // Split the token into its parts (Header, Payload, Signature) - var tokenParts = jwtToken.Split('.'); - if (tokenParts.Length != 3) - { - return null; - } - - // The payload is the second part, Base64Url encoded - var payload = tokenParts[1]; - - // Add padding if necessary - payload = AddBase64Padding(payload); - - // Decode the payload from Base64Url - var decodedBytes = Convert.FromBase64String(payload); - var decodedPayload = Encoding.UTF8.GetString(decodedBytes); - - // Parse the decoded payload as JSON - var payloadJson = JsonSerializer.Deserialize(decodedPayload); - - // Try to get the username from 'upn', 'preferred_username', or 'unique_name' claims - if (payloadJson.TryGetProperty("upn", out var upn)) - { - return upn.GetString(); - } - else if (payloadJson.TryGetProperty("preferred_username", out var preferredUsername)) - { - return preferredUsername.GetString(); - } - else if (payloadJson.TryGetProperty("unique_name", out var uniqueName)) - { - return uniqueName.GetString(); - } - - return null; - } - - private static string AddBase64Padding(string base64) => (base64.Length % 4) switch - { - 2 => base64 + "==", - 3 => base64 + "=", - _ => base64, - }; -} -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="UseWithEntraAuthentication1"::: Now you can use the `UseEntraAuthentication` method to set up the connection string for the Postgres connector: -```csharp -using Microsoft.SemanticKernel.Connectors.Postgres; - -var connectionString = "Host=mydb.postgres.database.azure.com;Port=5432;Database=postgres;SSL Mode=Require;"; // No Username or Password -var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); -dataSourceBuilder.UseEntraAuthentication(); -dataSourceBuilder.UseVector(); -var dataSource = dataSourceBuilder.Build(); - -var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); -``` +:::code language="csharp" source="./snippets/postgres-connector.cs" id="UseWithEntraAuthentication2"::: By default, the `UseEntraAuthentication` method uses the [DefaultAzureCredential](/dotnet/api/azure.identity.defaultazurecredential) to authenticate with Azure AD. You can also provide a custom `TokenCredential` implementation if needed. \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md index f523600f15dae..32b80d0857e60 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md @@ -38,67 +38,23 @@ dotnet add package Microsoft.SemanticKernel.Connectors.Qdrant --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddQdrantVectorStore("localhost"); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddQdrantVectorStore("localhost"); -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of the `Qdrant.Client.QdrantClient` class to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Qdrant.Client; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => new QdrantClient("localhost")); -services.AddQdrantVectorStore(); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Qdrant.Client; +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted3"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => new QdrantClient("localhost")); -builder.Services.AddQdrantVectorStore(); -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted4"::: You can construct a Qdrant Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Qdrant.Client; - -var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Qdrant.Client; - -var collection = new QdrantCollection( - new QdrantClient("localhost"), - "skhotels", - ownsClient: true); -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted6"::: ## Data mapping @@ -120,24 +76,7 @@ The property name override is done by setting the set on its attributes and how that will be represented in Qdrant. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="PropertyNameOverride"::: ```json { @@ -159,15 +98,7 @@ The default mode is *single unnamed vector*. With this option a collection might only contain a single vector and it will be unnamed in the storage model in Qdrant. Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: -```csharp -new Hotel -{ - HotelId = 1, - HotelName = "Hotel Happy", - Description = "A place where everyone can be happy.", - DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } -}; -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="SingleUnnamedVector"::: ```json { @@ -182,16 +113,7 @@ new Hotel If using the named vectors mode, it means that each point in a collection might contain more than one vector, and each will be named. Here is an example of how an object is represented in Qdrant when using *named vectors* mode: -```csharp -new Hotel -{ - HotelId = 1, - HotelName = "Hotel Happy", - Description = "A place where everyone can be happy.", - HotelNameEmbedding = new float[4] { 0.9f, 0.5f, 0.5f, 0.5f } - DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } -}; -``` +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="NamedVectors1"::: ```json { @@ -207,18 +129,4 @@ new Hotel To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. The same options can also be passed to any of the provided dependency injection container extension methods. -```csharp -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Qdrant.Client; - -var vectorStore = new QdrantVectorStore( - new QdrantClient("localhost"), - ownsClient: true, - new() { HasNamedVectors = true }); - -var collection = new QdrantCollection( - new QdrantClient("localhost"), - "skhotels", - ownsClient: true, - new() { HasNamedVectors = true }); -``` \ No newline at end of file +:::code language="csharp" source="./snippets/qdrant-connector.cs" id="NamedVectors2"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md index 5b8bbac4ebe58..4b9abf2529cef 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md @@ -40,90 +40,31 @@ dotnet add package Microsoft.SemanticKernel.Connectors.Redis --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddRedisVectorStore("localhost:6379"); -``` - -```csharp -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddRedisVectorStore("localhost:6379"); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted2"::: Extension methods that take no parameters are also provided. These require an instance of the Redis `IDatabase` to be separately registered with the dependency injection container. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using StackExchange.Redis; +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted3"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); -services.AddRedisVectorStore(); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using StackExchange.Redis; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); -builder.Services.AddRedisVectorStore(); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted4"::: You can construct a Redis Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -var vectorStore = new RedisVectorStore(ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. When doing so, you have to choose between the JSON or Hashes instance depending on how you wish to store data in Redis. -```csharp -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -// Using Hashes. -var hashesCollection = new RedisHashSetCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelshashes"); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted6"::: -```csharp -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -// Using JSON. -var jsonCollection = new RedisJsonCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelsjson"); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted7"::: When constructing a `RedisVectorStore` or registering it with the dependency injection container, it's possible to pass a `RedisVectorStoreOptions` instance that configures the preferred storage type / mode used: Hashes or JSON. If not specified, the default is JSON. -```csharp -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -var vectorStore = new RedisVectorStore( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - new() { StorageType = RedisStorageType.HashSet }); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted8"::: ## Index prefixes @@ -141,17 +82,7 @@ By default, the connector will also prefix all keys with the this prefix when do If you didn't want to use a prefix consisting of the collection name and a colon, it is possible to switch off the prefixing behavior and pass in the fully prefixed key to the record operations. -```csharp -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -var collection = new RedisJsonCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelsjson", - new() { PrefixCollectionNameToKeyNames = false }); - -await collection.GetAsync("myprefix_h1"); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="IndexPrefixes"::: ## Data mapping @@ -167,37 +98,12 @@ Usage of the `JsonPropertyNameAttribute` is supported if a different storage nam data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` must be passed to the `RedisJsonCollection` on construction. -```csharp -var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; -var collection = new RedisJsonCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelsjson", - new() { JsonSerializerOptions = jsonSerializerOptions }); -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="DataMappingWhenUsingTheJSONStorageType1"::: Since a naming policy of snake case upper was chosen, here is an example of how this data type will be set in Redis. Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. -```csharp -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="DataMappingWhenUsingTheJSONStorageType2"::: ```redis JSON.SET skhotelsjson:h1 $ '{ "HOTEL_NAME": "Hotel Happy", "HOTEL_DESCRIPTION": "A place where everyone can be happy.", "DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1] }' @@ -215,24 +121,7 @@ Property name overriding is done by setting the set on its attributes and how these are set in Redis. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/redis-connector.cs" id="DataMappingWhenUsingTheHashesStorageType"::: ```redis HSET skhotelshashes:h1 hotel_name "Hotel Happy" hotel_description 'A place where everyone can be happy.' hotel_description_embedding diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj new file mode 100644 index 0000000000000..a2840a4891709 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj @@ -0,0 +1,46 @@ + + + + net10.0 + enable + enable + $(NoWarn);CS8019;CS0219;CS1591;CS8602 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs new file mode 100644 index 0000000000000..5569636cbb9bd --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs @@ -0,0 +1,81 @@ +// +using Azure; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); +// + +// +using Azure; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); +// + +// +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret))); +services.AddAzureAISearchVectorStore(); +// + +// +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret))); +builder.Services.AddAzureAISearchVectorStore(); +// + +// +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.SemanticKernel.Connectors.AzureAISearch; + +var vectorStore = new AzureAISearchVectorStore( + new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret))); +// + +// +using Azure; +using Azure.Search.Documents.Indexes; +using Microsoft.SemanticKernel.Connectors.AzureAISearch; + +var collection = new AzureAISearchCollection( + new SearchIndexClient(new Uri(azureAISearchUri), new AzureKeyCredential(secret)), + "skhotels"); +// + +// +var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; +var collection = new AzureAISearchCollection( + new SearchIndexClient( + new Uri(azureAISearchUri), + new AzureKeyCredential(secret), + new() { Serializer = new JsonObjectSerializer(jsonSerializerOptions) }), + "skhotels", + new() { JsonSerializerOptions = jsonSerializerOptions }); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs new file mode 100644 index 0000000000000..2cf6f2c12067c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs @@ -0,0 +1,91 @@ +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddCosmosMongoVectorStore(connectionString, databaseName); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddCosmosMongoVectorStore(connectionString, databaseName); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using MongoDB.Driver; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => + { + var mongoClient = new MongoClient(connectionString); + return mongoClient.GetDatabase(databaseName); + }); +services.AddCosmosMongoVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using MongoDB.Driver; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => + { + var mongoClient = new MongoClient(connectionString); + return mongoClient.GetDatabase(databaseName); + }); +builder.Services.AddCosmosMongoVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var vectorStore = new CosmosMongoVectorStore(database); +// + +// +using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var collection = new CosmosMongoCollection( + database, + "skhotels"); +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [BsonElement("hotel_name")] + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [BsonElement("hotel_description")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [BsonElement("hotel_description_embedding")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs new file mode 100644 index 0000000000000..13a08e0327df6 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs @@ -0,0 +1,160 @@ +// +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, +}); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddCosmosNoSqlVectorStore(connectionString, databaseName); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddCosmosNoSqlVectorStore(connectionString, databaseName); +// + +// +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => + { + var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() + { + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, + }); + + return cosmosClient.GetDatabase(databaseName); + }); +services.AddCosmosNoSqlVectorStore(); +// + +// +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => + { + var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() + { + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, + }); + + return cosmosClient.GetDatabase(databaseName); + }); +builder.Services.AddCosmosNoSqlVectorStore(); +// + +// +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.SemanticKernel.Connectors.CosmosNoSql; + +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, +}); + +var database = cosmosClient.GetDatabase(databaseName); +var vectorStore = new CosmosNoSqlVectorStore(database); +// + +// +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.SemanticKernel.Connectors.CosmosNoSql; + +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, +}); + +var database = cosmosClient.GetDatabase(databaseName); +var collection = new CosmosNoSqlCollection( + database, + "skhotels"); +// + +// +using System.Text.Json; +using Microsoft.Azure.Cosmos; +using Microsoft.SemanticKernel.Connectors.CosmosNoSql; + +var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; + +var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() +{ + // When initializing CosmosClient manually, setting this property is required + // due to limitations in default serializer. + UseSystemTextJsonSerializerWithOptions = jsonSerializerOptions +}); + +var database = cosmosClient.GetDatabase(databaseName); +var collection = new CosmosNoSqlCollection( + database, + "skhotels", + new() { JsonSerializerOptions = jsonSerializerOptions }); +// + +// +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.EuclideanDistance, IndexKind = IndexKind.QuantizedFlat)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// + +// +var options = new CosmosNoSqlCollectionOptions +{ + PartitionKeyPropertyName = nameof(Hotel.HotelName) +}; + +var collection = new CosmosNoSqlCollection(database, "collection-name", options) + as VectorStoreCollection; +// + +// +var record = await collection.GetAsync(new CosmosNoSqlCompositeKey("hotel-id", "hotel-name")); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs new file mode 100644 index 0000000000000..0289629e677dc --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs @@ -0,0 +1,225 @@ +// +using Microsoft.SemanticKernel; +using Couchbase.SemanticKernel; + +// Using a ServiceCollection. +var kernelBuilder = Kernel + .CreateBuilder() + .AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name"); +// + +// +using Couchbase.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name"); +// + +// +using Couchbase.SemanticKernel; + +var builder = WebApplication.CreateBuilder(args); + +// Option 1: Use Hyperscale index +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name", + options: new CouchbaseVectorStoreOptions + { + IndexType = CouchbaseIndexType.Hyperscale + }); + +// Option 2: Use Composite index +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name", + options: new CouchbaseVectorStoreOptions + { + IndexType = CouchbaseIndexType.Composite + }); + +// Option 3: Use Search vector index +builder.Services.AddCouchbaseVectorStore( + connectionString: "couchbases://your-cluster-address", + username: "username", + password: "password", + bucketName: "bucket-name", + scopeName: "scope-name", + options: new CouchbaseVectorStoreOptions + { + IndexType = CouchbaseIndexType.Search + }); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Couchbase; +using Couchbase.KeyValue; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => +{ + var clusterOptions = new ClusterOptions + { + ConnectionString = "couchbases://your-cluster-address", + UserName = "username", + Password = "password" + }; + + return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); +}); + +services.AddSingleton(sp => +{ + var cluster = sp.GetRequiredService(); + var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); + return bucket.Scope("scope-name"); +}); + +// Add Couchbase Vector Store +services.AddCouchbaseVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Couchbase.KeyValue; +using Couchbase; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); + +builder.Services.AddSingleton(sp => +{ + var clusterOptions = new ClusterOptions + { + ConnectionString = "couchbases://your-cluster-address", + UserName = "username", + Password = "password" + }; + + return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); +}); + +builder.Services.AddSingleton(sp => +{ + var cluster = sp.GetRequiredService(); + var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); + return bucket.Scope("scope-name"); +}); + +// Add Couchbase Vector Store +builder.Services.AddCouchbaseVectorStore(); +// + +// +using Couchbase; +using Couchbase.KeyValue; +using Couchbase.SemanticKernel; + +var clusterOptions = new ClusterOptions +{ + ConnectionString = "couchbases://your-cluster-address", + UserName = "username", + Password = "password" +}; + +var cluster = await Cluster.ConnectAsync(clusterOptions); +var bucket = await cluster.BucketAsync("bucket-name"); +var scope = bucket.Scope("scope-name"); + +var vectorStore = new CouchbaseVectorStore(scope); +// + +// +using Couchbase.SemanticKernel; +using Couchbase; +using Couchbase.KeyValue; + +var cluster = await Cluster.ConnectAsync(clusterOptions); +var bucket = await cluster.BucketAsync("bucket-name"); +var scope = bucket.Scope("scope-name"); + +// Using Hyperscale index (default) +var collection = new CouchbaseQueryCollection( + scope, + "skhotels", + indexType: CouchbaseIndexType.Hyperscale); + +// Or using Composite index +var collectionComposite = new CouchbaseQueryCollection( + scope, + "skhotels", + indexType: CouchbaseIndexType.Composite); +// + +// +using Couchbase.SemanticKernel; +using Couchbase; +using Couchbase.KeyValue; + +var cluster = await Cluster.ConnectAsync(clusterOptions); +var bucket = await cluster.BucketAsync("bucket-name"); +var scope = bucket.Scope("scope-name"); + +var collection = new CouchbaseSearchCollection( + scope, + "skhotels"); +// + +// +using Couchbase.SemanticKernel; +using Couchbase.KeyValue; +using System.Text.Json; + +var jsonSerializerOptions = new JsonSerializerOptions +{ + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper +}; + +var options = new CouchbaseQueryCollectionOptions +{ + JsonSerializerOptions = jsonSerializerOptions +}; + +var collection = new CouchbaseQueryCollection(scope, "skhotelsjson", options); +// + +// +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs new file mode 100644 index 0000000000000..89ad2bb28d2a1 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs @@ -0,0 +1,112 @@ +// +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using a ServiceCollection. +var kernelBuilder = Kernel + .CreateBuilder() + .AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); +// + +// +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); +services.AddElasticsearchVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Elastic.Clients.Elasticsearch; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); +builder.Services.AddElasticsearchVectorStore(); +// + +// +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; + +var vectorStore = new ElasticsearchVectorStore( + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); +// + +// +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; + +var collection = new ElasticsearchVectorStoreRecordCollection( + new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))), + "skhotels"); +// + +// +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; +using Elastic.Clients.Elasticsearch.Serialization; +using Elastic.Transport; + +var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); +var settings = new ElasticsearchClientSettings( + nodePool, + sourceSerializer: (defaultSerializer, settings) => + new DefaultSourceSerializer(settings, options => + options.PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper)); +var client = new ElasticsearchClient(settings); + +var collection = new ElasticsearchVectorStoreRecordCollection( + client, + "skhotelsjson"); +// + +// +using Elastic.SemanticKernel.Connectors.Elasticsearch; +using Elastic.Clients.Elasticsearch; + +var settings = new ElasticsearchClientSettings(new Uri("http://localhost:9200")); +settings.DefaultFieldNameInferrer(name => JsonNamingPolicy.SnakeCaseUpper.ConvertName(name)); +var client = new ElasticsearchClient(settings); + +var collection = new ElasticsearchVectorStoreRecordCollection( + client, + "skhotelsjson"); +// + +// +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity, IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs new file mode 100644 index 0000000000000..ec4af5a1939f7 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs @@ -0,0 +1,29 @@ +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddInMemoryVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddInMemoryVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.InMemory; + +var vectorStore = new InMemoryVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.InMemory; + +var collection = new InMemoryCollection("skhotels"); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs new file mode 100644 index 0000000000000..ce0cc8b7b9feb --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs @@ -0,0 +1,65 @@ +// +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMongoVectorStore(connectionString, databaseName); +// + +// +using Microsoft.Extensions.DependencyInjection; +using MongoDB.Driver; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => + { + var mongoClient = new MongoClient(connectionString); + return mongoClient.GetDatabase(databaseName); + }); +builder.Services.AddMongoVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.MongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var vectorStore = new MongoVectorStore(database); +// + +// +using Microsoft.SemanticKernel.Connectors.MongoDB; +using MongoDB.Driver; + +var mongoClient = new MongoClient(connectionString); +var database = mongoClient.GetDatabase(databaseName); +var collection = new MongoCollection( + database, + "skhotels"); +// + +// +using Microsoft.Extensions.VectorData; +using MongoDB.Bson.Serialization.Attributes; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [BsonElement("hotel_name")] + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [BsonElement("hotel_description")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [BsonElement("hotel_description_embedding")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs new file mode 100644 index 0000000000000..beec21e035efa --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs @@ -0,0 +1,102 @@ +// +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddOracleVectorStore(""); +// + +// +using Microsoft.AspNetCore.Builder; +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddOracleVectorStore(""); +// + +// +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; +using Oracle.ManagedDataAccess.Client; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => +{ + OracleDataSourceBuilder dataSourceBuilder = new(""); + return dataSourceBuilder.Build(); +}); + +services.AddOracleVectorStore(); +// + +// +using Microsoft.AspNetCore.Builder; +using Oracle.VectorData; +using Microsoft.Extensions.DependencyInjection; +using Oracle.ManagedDataAccess.Client; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => +{ + OracleDataSourceBuilder dataSourceBuilder = new(""); + return dataSourceBuilder.Build(); +}); + +builder.Services.AddOracleVectorStore(); +// + +// +using Oracle.VectorData; +using Oracle.ManagedDataAccess.Client; + +OracleDataSourceBuilder dataSourceBuilder = new(""); +var dataSource = dataSourceBuilder.Build(); + +var connection = new OracleVectorStore(dataSource); +// + +// +using Oracle.VectorData; + +var connection = new OracleVectorStore(""); +// + +// +using Oracle.VectorData; +using Oracle.ManagedDataAccess.Client; + +OracleDataSourceBuilder dataSourceBuilder = new(""); +var dataSource = dataSourceBuilder.Build(); + +var collection = new OracleCollection(dataSource, "skhotels"); +// + +// +using Oracle.VectorData; + +var collection = new OracleCollection("", "skhotels"); +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public long HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string? HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 384, DistanceFunction = DistanceFunction.CosineDistance)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs new file mode 100644 index 0000000000000..ad5e18f7632bb --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs @@ -0,0 +1,86 @@ +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddPineconeVectorStore(pineconeApiKey); +// + +// +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddPineconeVectorStore(pineconeApiKey); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using PineconeClient = Pinecone.PineconeClient; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton( + sp => new PineconeClient(pineconeApiKey)); +services.AddPineconeVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using PineconeClient = Pinecone.PineconeClient; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton( + sp => new PineconeClient(pineconeApiKey)); +builder.Services.AddPineconeVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.Pinecone; +using PineconeClient = Pinecone.PineconeClient; + +var vectorStore = new PineconeVectorStore( + new PineconeClient(pineconeApiKey)); +// + +// +using Microsoft.SemanticKernel.Connectors.Pinecone; +using PineconeClient = Pinecone.PineconeClient; + +var collection = new PineconeCollection( + new PineconeClient(pineconeApiKey), + "skhotels"); +// + +// +using Microsoft.SemanticKernel.Connectors.Pinecone; +using PineconeClient = Pinecone.PineconeClient; + +var collection = new PineconeCollection( + new PineconeClient(pineconeApiKey), + "skhotels", + new() { IndexNamespace = "seasidehotels" }); +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs new file mode 100644 index 0000000000000..d344d58eac67e --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs @@ -0,0 +1,220 @@ +// +NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +dataSourceBuilder.UseVector(); +NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +// + +// +using Microsoft.Extensions.DependencyInjection; + +var services = new ServiceCollection(); +services.AddPostgresVectorStore(""); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddPostgresVectorStore(""); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Npgsql; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => +{ + NpgsqlDataSourceBuilder dataSourceBuilder = new(""); + dataSourceBuilder.UseVector(); + return dataSourceBuilder.Build(); +}); +builder.Services.AddPostgresVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.PgVector; +using Npgsql; + +NpgsqlDataSourceBuilder dataSourceBuilder = new(""); +dataSourceBuilder.UseVector(); +NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); +// + +// +using Microsoft.SemanticKernel.Connectors.PgVector; + +var connection = new PostgresVectorStore(""); +// + +// +using Microsoft.SemanticKernel.Connectors.PgVector; +using Npgsql; + +NpgsqlDataSourceBuilder dataSourceBuilder = new(""); +dataSourceBuilder.UseVector(); +var dataSource = dataSourceBuilder.Build(); + +var collection = new PostgresCollection(dataSource, "skhotels", ownsDataSource: true); +// + +// +using Microsoft.SemanticKernel.Connectors.PgVector; + +var collection = new PostgresCollection("", "skhotels"); +// + +// +using System; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey(StorageName = "hotel_id")] + public int HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// + +// +using System.Text; +using System.Text.Json; +using Azure.Core; +using Azure.Identity; +using Npgsql; + +namespace Program; + +public static class NpgsqlDataSourceBuilderExtensions +{ + private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new(["https://ossrdbms-aad.database.windows.net/.default"]); + + public static NpgsqlDataSourceBuilder UseEntraAuthentication(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default) + { + credential ??= new DefaultAzureCredential(); + + if (dataSourceBuilder.ConnectionStringBuilder.Username == null) + { + var token = credential.GetToken(s_azureDBForPostgresTokenRequestContext, default); + SetUsernameFromToken(dataSourceBuilder, token.Token); + } + + SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); + + return dataSourceBuilder; + } + + public static async Task UseEntraAuthenticationAsync(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default, CancellationToken cancellationToken = default) + { + credential ??= new DefaultAzureCredential(); + + if (dataSourceBuilder.ConnectionStringBuilder.Username == null) + { + var token = await credential.GetTokenAsync(s_azureDBForPostgresTokenRequestContext, cancellationToken).ConfigureAwait(false); + SetUsernameFromToken(dataSourceBuilder, token.Token); + } + + SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); + + return dataSourceBuilder; + } + + private static void SetPasswordProvider(NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential credential, TokenRequestContext tokenRequestContext) + { + dataSourceBuilder.UsePasswordProvider(_ => + { + var token = credential.GetToken(tokenRequestContext, default); + return token.Token; + }, async (_, ct) => + { + var token = await credential.GetTokenAsync(tokenRequestContext, ct).ConfigureAwait(false); + return token.Token; + }); + } + + private static void SetUsernameFromToken(NpgsqlDataSourceBuilder dataSourceBuilder, string token) + { + var username = TryGetUsernameFromToken(token); + + if (username != null) + { + dataSourceBuilder.ConnectionStringBuilder.Username = username; + } + else + { + throw new Exception("Could not determine username from token claims"); + } + } + + private static string? TryGetUsernameFromToken(string jwtToken) + { + // Split the token into its parts (Header, Payload, Signature) + var tokenParts = jwtToken.Split('.'); + if (tokenParts.Length != 3) + { + return null; + } + + // The payload is the second part, Base64Url encoded + var payload = tokenParts[1]; + + // Add padding if necessary + payload = AddBase64Padding(payload); + + // Decode the payload from Base64Url + var decodedBytes = Convert.FromBase64String(payload); + var decodedPayload = Encoding.UTF8.GetString(decodedBytes); + + // Parse the decoded payload as JSON + var payloadJson = JsonSerializer.Deserialize(decodedPayload); + + // Try to get the username from 'upn', 'preferred_username', or 'unique_name' claims + if (payloadJson.TryGetProperty("upn", out var upn)) + { + return upn.GetString(); + } + else if (payloadJson.TryGetProperty("preferred_username", out var preferredUsername)) + { + return preferredUsername.GetString(); + } + else if (payloadJson.TryGetProperty("unique_name", out var uniqueName)) + { + return uniqueName.GetString(); + } + + return null; + } + + private static string AddBase64Padding(string base64) => (base64.Length % 4) switch + { + 2 => base64 + "==", + 3 => base64 + "=", + _ => base64, + }; +} +// + +// +using Microsoft.SemanticKernel.Connectors.Postgres; + +var connectionString = "Host=mydb.postgres.database.azure.com;Port=5432;Database=postgres;SSL Mode=Require;"; // No Username or Password +var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); +dataSourceBuilder.UseEntraAuthentication(); +dataSourceBuilder.UseVector(); +var dataSource = dataSourceBuilder.Build(); + +var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs new file mode 100644 index 0000000000000..153a96107df6c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs @@ -0,0 +1,111 @@ +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddQdrantVectorStore("localhost"); +// + +// +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddQdrantVectorStore("localhost"); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Qdrant.Client; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => new QdrantClient("localhost")); +services.AddQdrantVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Qdrant.Client; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => new QdrantClient("localhost")); +builder.Services.AddQdrantVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +// + +// +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +var collection = new QdrantCollection( + new QdrantClient("localhost"), + "skhotels", + ownsClient: true); +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// + +// +new Hotel +{ + HotelId = 1, + HotelName = "Hotel Happy", + Description = "A place where everyone can be happy.", + DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } +}; +// + +// +new Hotel +{ + HotelId = 1, + HotelName = "Hotel Happy", + Description = "A place where everyone can be happy.", + HotelNameEmbedding = new float[4] { 0.9f, 0.5f, 0.5f, 0.5f } + DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } +}; +// + +// +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + ownsClient: true, + new() { HasNamedVectors = true }); + +var collection = new QdrantCollection( + new QdrantClient("localhost"), + "skhotels", + ownsClient: true, + new() { HasNamedVectors = true }); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs new file mode 100644 index 0000000000000..857042773e92c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs @@ -0,0 +1,134 @@ +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddRedisVectorStore("localhost:6379"); +// + +// +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddRedisVectorStore("localhost:6379"); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using StackExchange.Redis; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); +services.AddRedisVectorStore(); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using StackExchange.Redis; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); +builder.Services.AddRedisVectorStore(); +// + +// +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +var vectorStore = new RedisVectorStore(ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); +// + +// +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +// Using Hashes. +var hashesCollection = new RedisHashSetCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelshashes"); +// + +// +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +// Using JSON. +var jsonCollection = new RedisJsonCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelsjson"); +// + +// +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +var vectorStore = new RedisVectorStore( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + new() { StorageType = RedisStorageType.HashSet }); +// + +// +using Microsoft.SemanticKernel.Connectors.Redis; +using StackExchange.Redis; + +var collection = new RedisJsonCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelsjson", + new() { PrefixCollectionNameToKeyNames = false }); + +await collection.GetAsync("myprefix_h1"); +// + +// +var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; +var collection = new RedisJsonCollection( + ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), + "skhotelsjson", + new() { JsonSerializerOptions = jsonSerializerOptions }); +// + +// +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION")] + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public string HotelId { get; set; } + + [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs new file mode 100644 index 0000000000000..9ab75e7613f69 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs @@ -0,0 +1,47 @@ +// +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqlServerVectorStore(_ => ""); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqlServerVectorStore(_ => "") +// + +// +using Microsoft.SemanticKernel.Connectors.SqlServer; + +var vectorStore = new SqlServerVectorStore(""); +// + +// +using Microsoft.SemanticKernel.Connectors.SqlServer; + +var collection = new SqlServerCollection("", "skhotels"); +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string? HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs new file mode 100644 index 0000000000000..a0197f7e52a8d --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs @@ -0,0 +1,47 @@ +// +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:"); +// + +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:") +// + +// +using Microsoft.SemanticKernel.Connectors.SqliteVec; + +var vectorStore = new SqliteVectorStore("Data Source=:memory:"); +// + +// +using Microsoft.SemanticKernel.Connectors.SqliteVec; + +var collection = new SqliteCollection("Data Source=:memory:", "skhotels"); +// + +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(StorageName = "hotel_name")] + public string? HotelName { get; set; } + + [VectorStoreData(StorageName = "hotel_description")] + public string? Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs new file mode 100644 index 0000000000000..2c9612b5f166c --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs @@ -0,0 +1,28 @@ +// +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var kernelBuilder = Kernel + .CreateBuilder() + .AddVolatileVectorStore(); +// + +// +using Microsoft.SemanticKernel; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddVolatileVectorStore(); +// + +// +using Microsoft.SemanticKernel.Data; + +var vectorStore = new VolatileVectorStore(); +// + +// +using Microsoft.SemanticKernel.Data; + +var collection = new VolatileVectorStoreRecordCollection("skhotels"); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs new file mode 100644 index 0000000000000..54670662bd596 --- /dev/null +++ b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs @@ -0,0 +1,82 @@ +// +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); +// + +// +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); +// + +// +using System.Net.Http; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; + +// Using a ServiceCollection. +var services = new ServiceCollection(); +using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; +services.AddWeaviateVectorStore(_ => client); +// + +// +using System.Net.Http; +using Microsoft.Extensions.DependencyInjection; + +// Using IServiceCollection with ASP.NET Core. +var builder = WebApplication.CreateBuilder(args); +using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; +builder.Services.AddWeaviateVectorStore(_ => client); +// + +// +using System.Net.Http; +using Microsoft.SemanticKernel.Connectors.Weaviate; + +var vectorStore = new WeaviateVectorStore( + new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }); +// + +// +using System.Net.Http; +using Microsoft.SemanticKernel.Connectors.Weaviate; + +var collection = new WeaviateCollection( + new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }, + "Skhotels"); +// + +// +using Microsoft.SemanticKernel; + +var services = new ServiceCollection(); +services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), secretVar); +// + +// +using System.Text.Json.Serialization; +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public Guid HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] + [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.QuantizedFlat)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md index 2dd3ac1ac504c..fa1e2fef5c1c7 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md @@ -39,38 +39,17 @@ dotnet add package Microsoft.SemanticKernel.Connectors.SqlServer --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; +:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted1"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqlServerVectorStore(_ => ""); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqlServerVectorStore(_ => "") -``` +:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted2"::: You can construct a Sql Server Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.SqlServer; - -var vectorStore = new SqlServerVectorStore(""); -``` +:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted3"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.SqlServer; - -var collection = new SqlServerCollection("", "skhotels"); -``` +:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted4"::: ## Data mapping @@ -84,24 +63,7 @@ The property name override is done by setting the set on its attributes and how that will be represented in a SQL Server command. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string? HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/sql-connector.cs" id="PropertyNameOverride"::: ```sql CREATE TABLE Hotel ( diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md index e13c5a5461bbd..01aa8bfb1e34c 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md @@ -40,38 +40,17 @@ dotnet add package Microsoft.SemanticKernel.Connectors.SqliteVec --prerelease You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. -```csharp -using Microsoft.Extensions.DependencyInjection; +:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted1"::: -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:"); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:") -``` +:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted2"::: You can construct a SQLite Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Connectors.SqliteVec; - -var vectorStore = new SqliteVectorStore("Data Source=:memory:"); -``` +:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted3"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Connectors.SqliteVec; - -var collection = new SqliteCollection("Data Source=:memory:", "skhotels"); -``` +:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted4"::: ## Data mapping @@ -88,24 +67,7 @@ The property name override is done by setting the set on its attributes and how that will be represented in a SQLite command. -```csharp -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string? HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/sqlite-connector.cs" id="PropertyNameOverride"::: ```tsql CREATE TABLE Hotels ( diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md index 01d87816bbd1d..d7ff65676f076 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md @@ -39,35 +39,14 @@ dotnet add package Microsoft.SemanticKernel.Core You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -```csharp -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var kernelBuilder = Kernel - .CreateBuilder() - .AddVolatileVectorStore(); -``` - -```csharp -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddVolatileVectorStore(); -``` +:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted2"::: You can construct a Volatile Vector Store instance directly. -```csharp -using Microsoft.SemanticKernel.Data; - -var vectorStore = new VolatileVectorStore(); -``` +:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted3"::: It's possible to construct a direct reference to a named collection. -```csharp -using Microsoft.SemanticKernel.Data; - -var collection = new VolatileVectorStoreRecordCollection("skhotels"); -``` \ No newline at end of file +:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted4"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md index d1acf98dbcfe6..7c73490632bd0 100644 --- a/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md +++ b/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md @@ -54,76 +54,28 @@ It can be provided via options or by setting the base address of the `HttpClient This first example shows how to set the service URL via options. Also note that these methods will retrieve an `HttpClient` instance for making calls to the Weaviate service from the dependency injection service provider. -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted1"::: -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted2"::: Overloads where you can specify your own `HttpClient` are also provided. In this case it's possible to set the service url via the `HttpClient` `BaseAddress` option. -```csharp -using System.Net.Http; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; -services.AddWeaviateVectorStore(_ => client); -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted3"::: -```csharp -using System.Net.Http; -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; -builder.Services.AddWeaviateVectorStore(_ => client); -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted4"::: You can construct a Weaviate Vector Store instance directly as well. -```csharp -using System.Net.Http; -using Microsoft.SemanticKernel.Connectors.Weaviate; - -var vectorStore = new WeaviateVectorStore( - new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }); -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted5"::: It's possible to construct a direct reference to a named collection. -```csharp -using System.Net.Http; -using Microsoft.SemanticKernel.Connectors.Weaviate; - -var collection = new WeaviateCollection( - new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }, - "Skhotels"); -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted6"::: If needed, it is possible to pass an API key, as an option, when using any of the previously mentioned mechanisms. For example: -```csharp -using Microsoft.SemanticKernel; - -var services = new ServiceCollection(); -services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), secretVar); -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted7"::: ## Data mapping @@ -141,26 +93,7 @@ data model property name is required. Here is an example of a data model with `JsonPropertyNameAttribute` set and how that will be represented in Weaviate. -```csharp -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public Guid HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.QuantizedFlat)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/weaviate-connector.cs" id="DataMapping"::: ```json { diff --git a/docs/ai/vector-stores/schema-with-record-definition.md b/docs/ai/vector-stores/schema-with-record-definition.md index 28b108ff4cf58..40815e36de72b 100644 --- a/docs/ai/vector-stores/schema-with-record-definition.md +++ b/docs/ai/vector-stores/schema-with-record-definition.md @@ -17,28 +17,13 @@ This can be useful in multiple scenarios: Here is an example of how to create a record definition. -```csharp -using Microsoft.Extensions.VectorData; - -var hotelDefinition = new VectorStoreCollectionDefinition -{ - Properties = new List - { - new VectorStoreKeyProperty("HotelId", typeof(ulong)), - new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, - new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, - new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, - } -}; -``` +:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="DefineYourStorageSchemaUsingARecordDefin1"::: When creating a definition you always have to provide a name and type for each property in your schema, since this is required for index creation and data mapping. To use the definition, pass it to the GetCollection method. -```csharp -var collection = vectorStore.GetCollection("skhotels", hotelDefinition); -``` +:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="DefineYourStorageSchemaUsingARecordDefin2"::: ## Record property configuration classes @@ -46,9 +31,7 @@ var collection = vectorStore.GetCollection("skhotels", hotelDefini Use this class to indicate that your property is the key of the record. -```csharp -new VectorStoreKeyProperty("HotelId", typeof(ulong)), -``` +:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreKeyProperty"::: #### VectorStoreKeyProperty configuration settings @@ -65,9 +48,7 @@ new VectorStoreKeyProperty("HotelId", typeof(ulong)), Use this class to indicate that your property contains general data that is not a key or a vector. -```csharp -new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, -``` +:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreDataProperty"::: #### VectorStoreDataProperty configuration settings @@ -86,9 +67,7 @@ new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, Use this class to indicate that your property contains a vector. -```csharp -new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, -``` +:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreVectorProperty"::: #### VectorStoreVectorProperty configuration settings diff --git a/docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj b/docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj new file mode 100644 index 0000000000000..760fc2801b1e6 --- /dev/null +++ b/docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj @@ -0,0 +1,21 @@ + + + + net10.0 + enable + enable + $(NoWarn);CS8019;CS0219;CS1591;CS8602 + + + + + + + + + + + + + + diff --git a/docs/ai/vector-stores/snippets/defining-your-data-model.cs b/docs/ai/vector-stores/snippets/defining-your-data-model.cs new file mode 100644 index 0000000000000..2cde90a27efc5 --- /dev/null +++ b/docs/ai/vector-stores/snippets/defining-your-data-model.cs @@ -0,0 +1,41 @@ +// +using Microsoft.Extensions.VectorData; + +public class Hotel +{ + [VectorStoreKey] + public ulong HotelId { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string HotelName { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + + [VectorStoreData(IsIndexed = true)] + public string[] Tags { get; set; } +} +// + +// +[VectorStoreKey] +public ulong HotelId { get; set; } +// + +// +[VectorStoreData(IsIndexed = true)] +public string HotelName { get; set; } +// + +// +[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] +public ReadOnlyMemory? DescriptionEmbedding { get; set; } +// + +// +[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] +public string DescriptionEmbedding { get; set; } +// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/dynamic-data-model.cs b/docs/ai/vector-stores/snippets/dynamic-data-model.cs new file mode 100644 index 0000000000000..b2226d3a3ede8 --- /dev/null +++ b/docs/ai/vector-stores/snippets/dynamic-data-model.cs @@ -0,0 +1,39 @@ +// +// Create the definition to define the schema. +VectorStoreCollectionDefinition definition = new() +{ + Properties = new List + { + new VectorStoreKeyProperty("Key", typeof(string)), + new VectorStoreDataProperty("Term", typeof(string)), + new VectorStoreDataProperty("Definition", typeof(string)), + new VectorStoreVectorProperty("DefinitionEmbedding", typeof(ReadOnlyMemory), dimensions: 1536) + } +}; + +// When getting your collection instance from a vector store instance +// specify the Dictionary, using object as the key type for your database +// and also pass your record definition. +// Note that you have to use GetDynamicCollection instead of the regular GetCollection method +// to get an instance of a collection using Dictionary. +var dynamicDataModelCollection = vectorStore.GetDynamicCollection( + "glossary", + definition); + +// Since schema information is available from the record definition +// it's possible to create a collection with the right vectors, +// dimensions, indexes, and distance functions. +await dynamicDataModelCollection.EnsureCollectionExistsAsync(); + +// When retrieving a record from the collection, key, data, and vector values can +// now be accessed via the dictionary entries. +var record = await dynamicDataModelCollection.GetAsync("SK"); +Console.WriteLine(record["Definition"]); +// + +// +new AzureAISearchDynamicCollection( + searchIndexClient, + "glossary", + new() { Definition = definition }); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/embedding-generation.cs b/docs/ai/vector-stores/snippets/embedding-generation.cs new file mode 100644 index 0000000000000..652619fc98e34 --- /dev/null +++ b/docs/ai/vector-stores/snippets/embedding-generation.cs @@ -0,0 +1,202 @@ +// +[VectorStoreVector(1536)] +public string Embedding { get; set; } +// + +// +using Microsoft.Extensions.AI; +using Microsoft.SemanticKernel.Connectors.Qdrant; +using OpenAI; +using Qdrant.Client; + +var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + +var vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + ownsClient: true, + new QdrantVectorStoreOptions + { + EmbeddingGenerator = embeddingGenerator + }); +// + +// +using Microsoft.Extensions.AI; +using Microsoft.SemanticKernel.Connectors.Qdrant; +using OpenAI; +using Qdrant.Client; + +var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + +var collectionOptions = new QdrantCollectionOptions +{ + EmbeddingGenerator = embeddingGenerator +}; +var collection = new QdrantCollection( + new QdrantClient("localhost"), + "myCollection", + ownsClient: true, + collectionOptions); +// + +// +using Microsoft.Extensions.AI; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.Qdrant; +using OpenAI; +using Qdrant.Client; + +var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + +var definition = new VectorStoreCollectionDefinition +{ + EmbeddingGenerator = embeddingGenerator, + Properties = new List + { + new VectorStoreKeyProperty("Key", typeof(ulong)), + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) + } +}; + +var collectionOptions = new QdrantCollectionOptions +{ + Definition = definition +}; +var collection = new QdrantCollection( + new QdrantClient("localhost"), + "myCollection", + ownsClient: true, + collectionOptions); +// + +// +using Microsoft.Extensions.AI; +using Microsoft.Extensions.VectorData; +using OpenAI; + +var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + +var vectorProperty = new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) +{ + EmbeddingGenerator = embeddingGenerator +}; +// + +// + +// The data model +internal class FinanceInfo +{ + [VectorStoreKey] + public string Key { get; set; } = string.Empty; + + [VectorStoreData] + public string Text { get; set; } = string.Empty; + + // Note that the vector property is typed as a string, and + // its value is derived from the Text property. The string + // value will however be converted to a vector on upsert and + // stored in the database as a vector. + [VectorStoreVector(1536)] + public string Embedding => this.Text; +} + +// Create an OpenAI embedding generator. +var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + +// Use the embedding generator with the vector store. +var vectorStore = new InMemoryVectorStore(new() { EmbeddingGenerator = embeddingGenerator }); +var collection = vectorStore.GetCollection("finances"); +await collection.EnsureCollectionExistsAsync(); + +// Create some test data. +string[] budgetInfo = +{ + "The budget for 2020 is EUR 100 000", + "The budget for 2021 is EUR 120 000", + "The budget for 2022 is EUR 150 000", + "The budget for 2023 is EUR 200 000", + "The budget for 2024 is EUR 364 000" +}; + +// Embeddings are generated automatically on upsert. +var records = budgetInfo.Select((input, index) => new FinanceInfo { Key = index.ToString(), Text = input }); +await collection.UpsertAsync(records); + +// Embeddings for the search is automatically generated on search. +var searchResult = collection.SearchAsync( + "What is my budget for 2024?", + top: 1); + +// Output the matching result. +await foreach (var result in searchResult) +{ + Console.WriteLine($"Key: {result.Record.Key}, Text: {result.Record.Text}"); +} +// + +// +public async Task GenerateEmbeddingsAndUpsertAsync( + IEmbeddingGenerator> embeddingGenerator, + VectorStoreCollection collection) +{ + // Upsert a record. + string descriptionText = "A place where everyone can be happy."; + ulong hotelId = 1; + + // Generate the embedding. + ReadOnlyMemory embedding = + (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; + + // Create a record and upsert with the already generated embedding. + await collection.UpsertAsync(new Hotel + { + HotelId = hotelId, + HotelName = "Hotel Happy", + Description = descriptionText, + DescriptionEmbedding = embedding, + Tags = new[] { "luxury", "pool" } + }); +} +// + +// +public async Task GenerateEmbeddingsAndSearchAsync( + IEmbeddingGenerator> embeddingGenerator, + VectorStoreCollection collection) +{ + // Upsert a record. + string descriptionText = "Find me a hotel with happiness in mind."; + + // Generate the embedding. + ReadOnlyMemory searchEmbedding = + (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; + + // Search using the already generated embedding. + IAsyncEnumerable> searchResult = collection.SearchAsync(searchEmbedding, top: 1); + List> resultItems = await searchResult.ToListAsync(); + + // Print the first search result. + Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); + Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); +} +// + +// +[VectorStoreVector(Dimensions: 1536)] +public ReadOnlyMemory? DescriptionEmbedding { get; set; } +// + +// +new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 1536); +// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/hybrid-search.cs b/docs/ai/vector-stores/snippets/hybrid-search.cs new file mode 100644 index 0000000000000..686d03eadad08 --- /dev/null +++ b/docs/ai/vector-stores/snippets/hybrid-search.cs @@ -0,0 +1,144 @@ +// +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +// Placeholder embedding generation method. +async Task> GenerateEmbeddingAsync(string textToVectorize) +{ + // your logic here +} + +// Create a Qdrant VectorStore object and choose an existing collection that already contains records. +VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +IKeywordHybridSearchable collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skhotels"); + +// Generate a vector for your search text, using your chosen embedding generation implementation. +ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); + +// Do the search, passing an options object with a Top value to limit results to the single top match. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 1); + +// Inspect the returned hotel. +await foreach (var record in searchResult) +{ + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); +} +// + +// +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +var collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skproducts"); + +// Create the hybrid search options and indicate that you want +// to search the DescriptionEmbedding vector property and the +// Description full text search property. +var hybridSearchOptions = new HybridSearchOptions +{ + VectorProperty = r => r.DescriptionEmbedding, + AdditionalProperty = r => r.Description +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + +public sealed class Product +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Name { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Description { get; set; } + + [VectorStoreData] + public List FeatureList { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } +} +// + +// +// Create the vector search options and indicate that you want to skip the first 40 results and then pass 20 to search to get the next 20. +var hybridSearchOptions = new HybridSearchOptions +{ + Skip = 40 +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 20, hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.Description); +} +// + +// +// Create the hybrid search options and indicate that you want to include vectors in the search results. +var hybridSearchOptions = new HybridSearchOptions +{ + IncludeVectors = true +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.FeatureList); +} +// + +// +// Create the hybrid search options and set the filter on the options. +var hybridSearchOptions = new HybridSearchOptions +{ + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.Definition); +} + +sealed class Glossary +{ + [VectorStoreKey] + public ulong Key { get; set; } + + // Category is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public string Category { get; set; } + + // Tags is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public List Tags { get; set; } + + [VectorStoreData] + public string Term { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public string Definition { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/schema-with-record-definition.cs b/docs/ai/vector-stores/snippets/schema-with-record-definition.cs new file mode 100644 index 0000000000000..f4af9be040c88 --- /dev/null +++ b/docs/ai/vector-stores/snippets/schema-with-record-definition.cs @@ -0,0 +1,30 @@ +// +using Microsoft.Extensions.VectorData; + +var hotelDefinition = new VectorStoreCollectionDefinition +{ + Properties = new List + { + new VectorStoreKeyProperty("HotelId", typeof(ulong)), + new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, + new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, + } +}; +// + +// +var collection = vectorStore.GetCollection("skhotels", hotelDefinition); +// + +// +new VectorStoreKeyProperty("HotelId", typeof(ulong)), +// + +// +new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, +// + +// +new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, +// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/vector-search.cs b/docs/ai/vector-stores/snippets/vector-search.cs new file mode 100644 index 0000000000000..24e5cb0412110 --- /dev/null +++ b/docs/ai/vector-stores/snippets/vector-search.cs @@ -0,0 +1,139 @@ +// +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +// Placeholder embedding generation method. +async Task> GenerateEmbeddingAsync(string textToVectorize) +{ + // your logic here +} + +// Create a Qdrant VectorStore object and choose an existing collection that already contains records. +VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); +VectorStoreCollection collection = vectorStore.GetCollection("skhotels"); + +// Generate a vector for your search text, using your chosen embedding generation implementation. +ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); + +// Do the search, passing an options object with a Top value to limit results to the single top match. +var searchResult = collection.SearchAsync(searchVector, top: 1); + +// Inspect the returned hotel. +await foreach (var record in searchResult) +{ + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); +} +// + +// +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; + +var vectorStore = new InMemoryVectorStore(); +var collection = vectorStore.GetCollection("skproducts"); + +// Create the vector search options and indicate that you want to search the FeatureListEmbedding property. +var vectorSearchOptions = new VectorSearchOptions +{ + VectorProperty = r => r.FeatureListEmbedding +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + +public sealed class Product +{ + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public string Description { get; set; } + + [VectorStoreData] + public List FeatureList { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } +} +// + +// +// Create the vector search options and indicate that you want to skip the first 40 results. +var vectorSearchOptions = new VectorSearchOptions +{ + Skip = 40 +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +// Pass 'top: 20' to indicate that you want to retrieve the next 20 results after skipping +// the first 40 +var searchResult = collection.SearchAsync(searchVector, top: 20, vectorSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.FeatureList); +} +// + +// +// Create the vector search options and indicate that you want to include vectors in the search results. +var vectorSearchOptions = new VectorSearchOptions +{ + IncludeVectors = true +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.FeatureList); +} +// + +// +// Create the vector search options and set the filter on the options. +var vectorSearchOptions = new VectorSearchOptions +{ + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult) +{ + Console.WriteLine(result.Record.Definition); +} + +sealed class Glossary +{ + [VectorStoreKey] + public ulong Key { get; set; } + + // Category is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public string Category { get; set; } + + // Tags is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public List Tags { get; set; } + + [VectorStoreData] + public string Term { get; set; } + + [VectorStoreData] + public string Definition { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } +} +// \ No newline at end of file diff --git a/docs/ai/vector-stores/vector-search.md b/docs/ai/vector-stores/vector-search.md index 2ad12b696fba4..3ce25df726482 100644 --- a/docs/ai/vector-stores/vector-search.md +++ b/docs/ai/vector-stores/vector-search.md @@ -22,34 +22,7 @@ Note that imple Assuming you have a collection that already contains data, you can easily search it. Here is an example using Qdrant. -```csharp -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Microsoft.Extensions.VectorData; -using Qdrant.Client; - -// Placeholder embedding generation method. -async Task> GenerateEmbeddingAsync(string textToVectorize) -{ - // your logic here -} - -// Create a Qdrant VectorStore object and choose an existing collection that already contains records. -VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -VectorStoreCollection collection = vectorStore.GetCollection("skhotels"); - -// Generate a vector for your search text, using your chosen embedding generation implementation. -ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); - -// Do the search, passing an options object with a Top value to limit results to the single top match. -var searchResult = collection.SearchAsync(searchVector, top: 1); - -// Inspect the returned hotel. -await foreach (var record in searchResult) -{ - Console.WriteLine("Found hotel description: " + record.Record.Description); - Console.WriteLine("Found record score: " + record.Score); -} -``` +:::code language="csharp" source="./snippets/vector-search.cs" id="VectorSearch"::: > [!TIP] > For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). @@ -75,40 +48,7 @@ The `VectorProperty` option can be used to specify the vector property to target If none is provided and the data model contains only one vector, that vector will be used. If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. -```csharp -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.InMemory; - -var vectorStore = new InMemoryVectorStore(); -var collection = vectorStore.GetCollection("skproducts"); - -// Create the vector search options and indicate that you want to search the FeatureListEmbedding property. -var vectorSearchOptions = new VectorSearchOptions -{ - VectorProperty = r => r.FeatureListEmbedding -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); - -public sealed class Product -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData] - public string Description { get; set; } - - [VectorStoreData] - public List FeatureList { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DescriptionEmbedding { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory FeatureListEmbedding { get; set; } -} -``` +:::code language="csharp" source="./snippets/vector-search.cs" id="VectorProperty"::: ### `Top` and `Skip` @@ -116,24 +56,7 @@ The `Top` and `Skip` options allow you to limit the number of results to the Top to skip a number of results from the top of the resultset. Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. -```csharp -// Create the vector search options and indicate that you want to skip the first 40 results. -var vectorSearchOptions = new VectorSearchOptions -{ - Skip = 40 -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -// Pass 'top: 20' to indicate that you want to retrieve the next 20 results after skipping -// the first 40 -var searchResult = collection.SearchAsync(searchVector, top: 20, vectorSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.FeatureList); -} -``` +:::code language="csharp" source="./snippets/vector-search.cs" id="TopAndSkip"::: The default value `Skip` is 0. @@ -146,22 +69,7 @@ making searches more efficient. The default value for `IncludeVectors` is `false`. -```csharp -// Create the vector search options and indicate that you want to include vectors in the search results. -var vectorSearchOptions = new VectorSearchOptions -{ - IncludeVectors = true -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.FeatureList); -} -``` +:::code language="csharp" source="./snippets/vector-search.cs" id="IncludeVectors"::: ### Filter @@ -187,42 +95,4 @@ The set of LINQ expressions supported will vary depending on the functionality s by each database, but all databases support a broad base of common expressions, for example, equals, not equals, and, or, etc. -```csharp -// Create the vector search options and set the filter on the options. -var vectorSearchOptions = new VectorSearchOptions -{ - Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.Definition); -} - -sealed class Glossary -{ - [VectorStoreKey] - public ulong Key { get; set; } - - // Category is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public string Category { get; set; } - - // Tags is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public List Tags { get; set; } - - [VectorStoreData] - public string Term { get; set; } - - [VectorStoreData] - public string Definition { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DefinitionEmbedding { get; set; } -} -``` \ No newline at end of file +:::code language="csharp" source="./snippets/vector-search.cs" id="Filter"::: \ No newline at end of file From 65e3905a9544636a156e68ace6693365f5e6321e Mon Sep 17 00:00:00 2001 From: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Date: Sat, 28 Feb 2026 10:29:05 -0800 Subject: [PATCH 09/11] delete out of the box connectors folder --- .../azure-ai-search-connector.md | 78 ------ .../azure-cosmosdb-mongodb-connector.md | 77 ------ .../azure-cosmosdb-nosql-connector.md | 106 --------- .../couchbase-connector.md | 141 ----------- .../elasticsearch-connector.md | 102 -------- .../inmemory-connector.md | 54 ----- .../mongodb-connector.md | 75 ------ .../oracle-connector.md | 140 ----------- .../pinecone-connector.md | 97 -------- .../postgres-connector.md | 128 ---------- .../qdrant-connector.md | 132 ---------- .../redis-connector.md | 128 ---------- .../snippets/ConnectorSnippets.csproj | 46 ---- .../snippets/azure-ai-search-connector.cs | 81 ------- .../azure-cosmosdb-mongodb-connector.cs | 91 ------- .../azure-cosmosdb-nosql-connector.cs | 160 ------------- .../snippets/couchbase-connector.cs | 225 ------------------ .../snippets/elasticsearch-connector.cs | 112 --------- .../snippets/inmemory-connector.cs | 29 --- .../snippets/mongodb-connector.cs | 65 ----- .../snippets/oracle-connector.cs | 102 -------- .../snippets/pinecone-connector.cs | 86 ------- .../snippets/postgres-connector.cs | 220 ----------------- .../snippets/qdrant-connector.cs | 111 --------- .../snippets/redis-connector.cs | 134 ----------- .../snippets/sql-connector.cs | 47 ---- .../snippets/sqlite-connector.cs | 47 ---- .../snippets/volatile-connector.cs | 28 --- .../snippets/weaviate-connector.cs | 82 ------- .../sql-connector.md | 76 ------ .../sqlite-connector.md | 83 ------- .../volatile-connector.md | 52 ---- .../weaviate-connector.md | 106 --------- 33 files changed, 3241 deletions(-) delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md delete mode 100644 docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md deleted file mode 100644 index 949bc01617432..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/azure-ai-search-connector.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: Using the Azure AI Search Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Azure AI Search. -ms.topic: concept-article -ms.date: 07/08/2024 ---- -# Using the Azure AI Search Vector Store connector (Preview) - -> [!WARNING] -> The Azure AI Search Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Azure AI Search Vector Store connector can be used to access and manage data in Azure AI Search. The connector has the following characteristics. - -| Feature Area | Support | -|-------------------------------|-----------------------| -| Collection maps to | Azure AI Search Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types |
  • Hnsw
  • Flat
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StorageName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | | - -## Limitations - -Notable Azure AI Search connector functionality limitations. - -| Feature Area | Workaround | -|--------------|------------| -| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | - -## Get started - -Add the Azure AI Search Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.AzureAISearch --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of the Azure AI Search `SearchIndexClient` to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted4"::: - -You can construct an Azure AI Search Vector Store instance directly. - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="GetStarted6"::: - -## Data mapping - -The default mapper used by the Azure AI Search connector when mapping data from the data model to storage is the one provided by the Azure AI Search SDK. - -This mapper does a direct conversion of the list of properties on the data model to the fields in Azure AI Search and uses `System.Text.Json.JsonSerializer` -to convert to the storage schema. This means that usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the -data model property name is required. - -It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` -must be passed to both the `SearchIndexClient` and the `AzureAISearchCollection` on construction. - -:::code language="csharp" source="./snippets/azure-ai-search-connector.cs" id="DataMapping"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md deleted file mode 100644 index 7604358924f6e..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: Using the Azure CosmosDB MongoDB (vCore) Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Azure CosmosDB MongoDB (vCore). -ms.topic: concept-article -ms.date: 09/23/2024 ---- -# Using the Azure CosmosDB MongoDB (vCore) Vector Store connector (Preview) - -> [!WARNING] -> The Azure CosmosDB MongoDB (vCore) Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Azure CosmosDB MongoDB Vector Store connector can be used to access and manage data in Azure CosmosDB MongoDB (vCore). The connector has the following characteristics. - -| Feature Area | Support | -|------------------------------|----------------------------------------------------| -| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types |
  • Hnsw
  • IvfFlat
| -| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | No | -| StorageName supported? | No, use BsonElementAttribute instead. For more information, see [Data mapping](#data-mapping). | -| HybridSearch supported? | No | - -## Limitations - -This connector is compatible with Azure Cosmos DB MongoDB (vCore) and is *not* designed to be compatible with Azure Cosmos DB MongoDB (RU). - -## Get started - -Add the Azure CosmosDB MongoDB Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.CosmosMongoDB --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of `MongoDB.Driver.IMongoDatabase` to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted4"::: - -You can construct an Azure CosmosDB MongoDB Vector Store instance directly. - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="GetStarted6"::: - -## Data mapping - -The Azure CosmosDB MongoDB Vector Store connector provides a default mapper when mapping data from the data model to storage. - -This mapper does a direct conversion of the list of properties on the data model to the fields in Azure CosmosDB MongoDB and uses `MongoDB.Bson.Serialization` to convert to the storage schema. This means that usage of the `MongoDB.Bson.Serialization.Attributes.BsonElement` is supported if a different storage name to the data model property name is required. The only exception is the key of the record which is mapped to a database field named `_id`, since all CosmosDB MongoDB records must use this name for ids. - -### Property name override - -For data properties and vector properties, you can provide override field names to use in storage that is different to the property names on the data model. This is not supported for keys, since a key has a fixed name in MongoDB. - -The property name override is done by setting the `BsonElement` attribute on the data model properties. - -Here is an example of a data model with `BsonElement` set. - -:::code language="csharp" source="./snippets/azure-cosmosdb-mongodb-connector.cs" id="PropertyNameOverride"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md deleted file mode 100644 index 7045f8622a420..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Using the Azure CosmosDB NoSQL Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Azure CosmosDB NoSQL. -ms.topic: concept-article -ms.date: 09/23/2024 ---- -# Using the Azure CosmosDB NoSQL Vector Store connector (Preview) - -> [!WARNING] -> The Azure CosmosDB NoSQL Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage data in Azure CosmosDB NoSQL. The connector has the following characteristics. - -| Feature Area | Support | -|--------------------|---------------------------------| -| Collection maps to | Azure Cosmos DB NoSQL Container | -| Supported key property types |
  • string
  • CosmosNoSqlCompositeKey
| -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • byte[]
  • ReadOnlyMemory\
  • Embedding\
  • sbyte[]
| -| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StorageName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | Yes | - -## Limitations - -When initializing `CosmosClient` manually, it is necessary to specify `CosmosClientOptions.UseSystemTextJsonSerializerWithOptions` due to limitations in the default serializer. This option can be set to `JsonSerializerOptions.Default` or customized with other serializer options to meet specific configuration needs. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="Limitations"::: - -## Get started - -Add the Azure CosmosDB NoSQL Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.CosmosNoSql --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of `Microsoft.Azure.Cosmos.Database` to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted4"::: - -You can construct an Azure CosmosDB NoSQL Vector Store instance directly. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="GetStarted6"::: - -## Data mapping - -The Azure CosmosDB NoSQL Vector Store connector provides a default mapper when mapping from the data model to storage. - -This mapper does a direct conversion of the list of properties on the data model to the fields in Azure CosmosDB NoSQL and uses `System.Text.Json.JsonSerializer` -to convert to the storage schema. This means that usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the -data model property name is required. The only exception is the key of the record which is mapped to a database field named `id`, since all CosmosDB NoSQL -records must use this name for ids. - -It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` -must be passed to the `CosmosNoSqlCollection` on construction. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="DataMapping1"::: - -Using the above custom `JsonSerializerOptions` which is using `SnakeCaseUpper`, the following data model will be mapped to the below json. - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="DataMapping2"::: - -```json -{ - "id": "1", - "HOTEL_NAME": "Hotel Happy", - "DESCRIPTION": "A place where everyone can be happy.", - "HOTEL_DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1], -} -``` - -## Using partition key - -In the Azure Cosmos DB for NoSQL connector, the partition key property defaults to the key property - `id`. The `PartitionKeyPropertyName` property in `CosmosNoSqlCollectionOptions` class allows specifying a different property as the partition key. - -The `CosmosNoSqlCollection` class supports two key types: `string` and `CosmosNoSqlCompositeKey`. The `CosmosNoSqlCompositeKey` consists of `RecordKey` and `PartitionKey`. - -If the partition key property is not set (and the default key property is used), `string` keys can be used for operations with database records. However, if a partition key property is specified, it is recommended to use `CosmosNoSqlCompositeKey` to provide both the key and partition key values. - -Specify partition key property name: - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="UsingPartitionKey1"::: - -Get with partition key: - -:::code language="csharp" source="./snippets/azure-cosmosdb-nosql-connector.cs" id="UsingPartitionKey2"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md deleted file mode 100644 index e520824397d65..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/couchbase-connector.md +++ /dev/null @@ -1,141 +0,0 @@ ---- -title: Using the Couchbase Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Couchbase. -author: azaddhirajkumar -ms.topic: concept-article -ms.date: 11/03/2025 ---- - -# Using the Couchbase connector (Preview) - -> [!WARNING] -> The Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Couchbase Vector Store connector can be used to access and manage data in Couchbase. The connector has the -following characteristics. - -| Feature Area | Support | -|------------------------------|------------------------------| -| Collection maps to | Couchbase collection + index | -| Supported key property types |
  • `string`
| -| Supported data property types | All types that are supported by `System.Text.Json` (either built-in or by using a custom converter) | -| Supported vector property types |
  • `ReadOnlyMemory`
  • `Embedding`
  • `float[]`
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • `AnyTagEqualTo`
  • `EqualTo`
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | Yes | - -## Get started - -Add the Couchbase Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package CouchbaseConnector.SemanticKernel --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="GetStarted2"::: - -## Configure index type - -The vector store defaults to using Hyperscale indexes. You can specify a different index type by passing `CouchbaseVectorStoreOptions`: - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType1"::: - -Extension methods that take no parameters are also provided. These require an instance of the `IScope` class to be -separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType2"::: - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType3"::: - -You can construct a Couchbase Vector Store instance directly. - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="ConfigureIndexType4"::: - -It's possible to construct a direct reference to a named collection. - -## Use query collection (Hyperscale or composite index) - -For high-performance vector search with Hyperscale indexes: - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="UseQueryCollectionHyperscaleOrCompositeI"::: - -### Use search collection (seach vector index) - -For hybrid search scenarios combining full-text search: - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="UseSearchCollectionSeachVectorIndex"::: - -### Index type comparison - -Couchbase offers three types of indexes for vector search: - -**Hyperscale vector indexes** - -- Best for pure vector searches - content discovery, recommendations, semantic search -- High performance with low memory footprint - designed to scale to billions of vectors -- Optimized for concurrent operations - supports simultaneous searches and inserts -- **Use when:** You primarily perform vector-only queries without complex scalar filtering -- **Ideal for:** Large-scale semantic search, recommendation systems, content discovery -- **Requires:** Couchbase Server 8.0+ or Capella - -**Composite vector indexes** - -- Best for filtered vector searches - combines vector search with scalar value filtering -- Efficient pre-filtering - scalar attributes reduce the vector comparison scope -- **Use when:** Your queries combine vector similarity with scalar filters that eliminate large portions of data -- **Ideal for:** Compliance-based filtering, user-specific searches, time-bounded queries -- **Requires:** Couchbase Server 8.0+ or Capella - -**Search vector indexes** - -- Best for hybrid searches combining full-text search with vector similarity -- Allows semantic search alongside traditional keyword matching -- Supports geospatial searches in addition to vector and text -- **Use when:** You need to combine traditional keyword search with vector similarity search in the same query -- **Ideal for:** E-commerce product search, travel recommendations, content discovery with multiple search criteria -- **Requires:** Couchbase Server 7.6+ or Capella - -#### Choose the right index type - -- Start with **Hyperscale Index** for pure vector searches and large datasets (scales to billions) -- Choose **Composite Index** when scalar filters significantly reduce your search space (works well for tens of millions to billions of vectors) -- Use **Search Vector Index** for hybrid search combining text and vectors - -[Detailed comparison of vector index types](https://docs.couchbase.com/server/current/vector-index/use-vector-indexes.html) - -## Data mapping - -The Couchbase connector will use `System.Text.Json.JsonSerializer` to do mapping. Properties in the data model are serialized into a JSON object and stored as the document value in Couchbase. - -Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="DataMapping1"::: - -Since a naming policy of snake case upper was chosen, here is an example of how this data type will be stored in Couchbase. Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. - -:::code language="csharp" source="./snippets/couchbase-connector.cs" id="DataMapping2"::: - -```json -{ - "_id" : "h1", - "HOTEL_ID" : "h1", - "HOTEL_NAME" : "Hotel Happy", - "HOTEL_DESCRIPTION" : "A place where everyone can be happy.", - "DESCRIPTION_EMBEDDING" : [ - 0.9, - 0.1, - 0.1, - 0.1 - ] -} -``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md deleted file mode 100644 index 8eef60cf39da0..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/elasticsearch-connector.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -title: Using the Elasticsearch Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Elasticsearch. -author: flobernd -ms.topic: concept-article -ms.date: 11/04/2024 ---- -# Using the Elasticsearch connector (Preview) - -> [!WARNING] -> The Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Elasticsearch Vector Store connector can be used to access and manage data in Elasticsearch. The connector has the following characteristics. - -| Feature area | Support | -|--------------------|---------------------| -| Collection maps to | Elasticsearch index | -| Supported key property types |
  • `string`
  • `long`
  • `Guid`
| -| Supported data property types | All types that are supported by `System.Text.Json` (etiher built-in or by using a custom converter) | -| Supported vector property types |
  • `ReadOnlyMemory`
  • `IEnumerable`
  • `IReadOnlyCollection`
  • `ICollection`
  • `IReadOnlyList`
  • `IList`
  • `float[]`
| -| Supported index types |
  • HNSW (32, 8, or 4 bit or BBQ)
  • FLAT (32, 8, or 4 bit or BBQ)
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • MaxInnerProduct
| -| Supported filter clauses |
  • `AnyTagEqualTo`
  • `EqualTo`
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | Yes | - -## Get started - -To [run Elasticsearch locally](https://www.elastic.co/guide/en/elasticsearch/reference/current/run-elasticsearch-locally.html) for local development or testing run the `start-local` script with one command: - -```bash -curl -fsSL https://elastic.co/start-local | sh -``` - -Add the Elasticsearch Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Elastic.SemanticKernel.Connectors.Elasticsearch --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of the `Elastic.Clients.Elasticsearch.ElasticsearchClient` class to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted4"::: - -You can construct an Elasticsearch Vector Store instance directly. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="GetStarted6"::: - -## Data mapping - -The Elasticsearch connector will use `System.Text.Json.JsonSerializer` to do mapping. -Since Elasticsearch stores documents with a separate key/id and value, the mapper will serialize all properties except for the key to a JSON object -and use that as the value. - -Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the -data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, -a custom source serializer must be configured. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="DataMapping1"::: - -As an alternative, the `DefaultFieldNameInferrer` lambda function can be configured to achieve the same result or to even further customize property naming based on dynamic conditions. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="DataMapping2"::: - -Since a naming policy of snake case upper was chosen, here is an example of how this data type will be set in Elasticsearch. -Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. - -:::code language="csharp" source="./snippets/elasticsearch-connector.cs" id="DataMapping3"::: - -```json -{ - "_index" : "skhotelsjson", - "_id" : "h1", - "_source" : { - "HOTEL_NAME" : "Hotel Happy", - "HOTEL_DESCRIPTION" : "A place where everyone can be happy.", - "DESCRIPTION_EMBEDDING" : [ - 0.9, - 0.1, - 0.1, - 0.1 - ] - } -} -``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md deleted file mode 100644 index 9aacc23dd9b21..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/inmemory-connector.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: Using the In-Memory Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in an in-memory vector store. -ms.topic: concept-article -ms.date: 11/10/2024 ---- -# Using the In-Memory connector (Preview) - -> [!WARNING] -> The In-Memory Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The In-Memory Vector Store connector is a Vector Store implementation that uses no external database and stores data in memory. -This Vector Store is useful for prototyping scenarios or where high-speed in-memory operations are required. - -The connector has the following characteristics. - -| Feature Area | Support | -|-------------------------------|-------------------------------| -| Collection maps to | In-memory dictionary | -| Supported key property types | Any type that can be compared | -| Supported data property types | Any type | -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types | Flat | -| Supported distance functions |
  • CosineSimilarity
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StorageName supported? | No, since storage is in-memory and data reuse is therefore not possible, custom naming is not applicable. | -| HybridSearch supported? | No | - -## Get started - -Add the Semantic Kernel Core NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.InMemory --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted2"::: - -You can construct an InMemory Vector Store instance directly. - -:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted3"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/inmemory-connector.cs" id="GetStarted4"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md deleted file mode 100644 index 0aeb050152a29..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/mongodb-connector.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: Using the MongoDB Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in MongoDB. -author: dmytrostruk -ms.topic: concept-article -ms.author: dmytrostruk -ms.date: 10/25/2024 ---- -# Using the MongoDB Vector Store connector (Preview) - -> [!WARNING] -> The MongoDB Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The MongoDB Vector Store connector can be used to access and manage data in MongoDB. The connector has the following characteristics. - -| Feature Area | Support | -|--|--| -| Collection maps to | MongoDB Collection + Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types | N/A | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | No | -| StorageName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | Yes | - -## Get started - -Add the MongoDB Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.MongoDB --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted1"::: - -Extension methods that take no parameters are also provided. These require an instance of `MongoDB.Driver.IMongoDatabase` to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted2"::: - -You can construct a MongoDB Vector Store instance directly. - -:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted3"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/mongodb-connector.cs" id="GetStarted4"::: - -## Data mapping - -The MongoDB Vector Store connector provides a default mapper when mapping data from the data model to storage. - -This mapper does a direct conversion of the list of properties on the data model to the fields in MongoDB and uses `MongoDB.Bson.Serialization` -to convert to the storage schema. This means that usage of the `MongoDB.Bson.Serialization.Attributes.BsonElement` is supported if a different storage name to the -data model property name is required. The only exception is the key of the record which is mapped to a database field named `_id`, since all MongoDB -records must use this name for ids. - -### Property name override - -For data properties and vector properties, you can provide override field names to use in storage that is different to the -property names on the data model. This is not supported for keys, since a key has a fixed name in MongoDB. - -The property name override is done by setting the `BsonElement` attribute on the data model properties. - -Here is an example of a data model with `BsonElement` set. - -:::code language="csharp" source="./snippets/mongodb-connector.cs" id="PropertyNameOverride"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md deleted file mode 100644 index 1ed5224892065..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/oracle-connector.md +++ /dev/null @@ -1,140 +0,0 @@ ---- -title: Using the Oracle Database Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Oracle Database. -author: minal-agashe-oracle -ms.topic: article -ms.date: 08/14/2025 ---- -# Using the Oracle Database Vector Store connector (Preview) - -> [!WARNING] -> The Oracle Database Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Oracle Database Vector Store Connector can be used to access and manage data in Oracle Database. The connector has the following characteristics. - -| Feature Area | Support | -| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Collection maps to | Oracle database table | -| Supported key property types |
  • short
  • int
  • long
  • string
  • Guid
| -| Supported data property types |
  • bool
  • byte
  • short
  • int
  • decimal
  • long
  • float
  • double
  • DateTime
  • DateTimeOffset
  • TimeSpan
  • char
  • char[]
  • byte[]
  • String
  • Guid
  • *and nullable type of the above types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • double[]
  • ReadOnlyMemory\
  • Embedding\
  • short[]
  • ReadOnlyMemory\
  • Embedding\
  • byte[]
  • BitArray
  • BinaryEmbedding
| -| Supported index types |
  • Flat (default)
  • HNSW
  • IVF
| -| Supported distance functions |
  • CosineDistance
    • FLOAT32, FLOAT64, and INT8 vector default
  • CosineSimilarity
  • DotProductSimilarity
  • NegativeDotProductSimilarity
  • EuclideanDistance
  • EuclideanSquaredDistance
  • HammingDistance
    • BINARY vector default
  • ManhattanDistance
  • JaccardSimilarity
    To use Jaccard similarity, set the DistanceFunction string to "JACCARD" or "JACCARDSIMILARITY" (for example, DistanceFunction = "JACCARDSIMILARITY"). This value is case sensitive. Jaccard similarity requires BINARY numeric format vectors.
| -| Supported filter clauses |
  • ==
  • !=
  • <
  • <=
  • >
  • >=
  • List.Contains()
    • Only when checking if the model property is in the list
| -| Supports zero, one, or multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | No | -| StorageName supported? | Yes | -| HybridSearch supported? | No | - -> [!IMPORTANT] -> Vector data searches require Oracle Database 23ai or higher. All other Oracle connector features are available using Oracle Database 19c or higher. - -## Get started - -Add the Oracle Database Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Oracle.VectorData --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. In this case, an instance of the `Oracle.VectorData.OracleVectorStore` class also gets registered with the container. - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also available. These require an instance of the `Oracle.ManagedDataAccess.Client.OracleDataSource` class to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted4"::: - -You can construct an Oracle Database Vector Store instance directly with a custom data source or with a connection string. - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted5"::: - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted6"::: - -It's possible to construct a direct reference to a named collection with a custom data source or with a connection string. - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted7"::: - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="GetStarted8"::: - -## Data mapping - -The Oracle Database Vector Store connector provides a default mapper when mapping data from the data model to storage. This mapper does a direct conversion of the data model properties list to the Oracle database columns to convert to the storage schema. - -The Oracle Database Vector Store connector supports data model annotations and record definitions.Using annotations, the information can be provided to the data model for creating indexes and database column mapping. Using [record definitions](../schema-with-record-definition.md), the information can be defined and supplied separately from the data model. - -The following table shows the default primary key data type mapping between Oracle Database and C#: - -| C# Data Type | Database Type | -|--------------|-----------------| -| short/int16 | NUMBER(5) | -| int/int32 | NUMBER(10) | -| long/int64 | NUMBER(19) | -| string | NVARCHAR2(2000) | -| Guid | RAW(16) | - -The following table shows the default data property type mapping, including nullable types: - -| C# data type | Database type | -|------------------|--------------------------------| -| `bool` | `BOOLEAN` for Oracle Database 23ai and higher
`NUMBER(1)` for earlier versions | -| `byte` | `NUMBER(3)` | -| `short`/`int16` | `NUMBER(5)` | -| `int`/`int32` | `NUMBER(10)` | -| `decimal` | `NUMBER(18,2)` | -| `long`/`int64` | `NUMBER(19)` | -| `float` | `BINARY_FLOAT` | -| `double` | `BINARY_DOUBLE` | -| `DateTime` | `TIMESTAMP(7)` | -| `DateTimeOffset` | `TIMESTAMP(7) WITH TIME ZONE` | -| `TimeSpan` | `INTERVAL DAY(8) TO SECOND(7)` | -| `char` | `NVARCHAR2(1)` | -| `char[]` | `NVARCHAR2(2000)` | -| `byte[]` | `RAW(2000)` | -| `string` | `NVARCHAR2(2000)` | -| `Guid` | `RAW(16)` | - -Starting with Oracle Database 23ai, database vectors can be mapped to .NET data types. Multiple vector columns are supported. The following table shows the default vector property type mapping, including nullable types: - -| C# data type | Database type | -|--|--| -|
  • ReadOnlyMemory\
  • Embedding\
  • BinaryEmbedding
  • Embedding\
  • byte[]
  • System.Byte[]
  • BitArray
| VECTOR(dimensions, BINARY) | -|
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • Embedding\
  • Embedding\
  • short[]
  • System.Int16[]
| VECTOR(dimensions, INT8) | -|
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • Embedding\
  • Embedding\
  • double[]
  • System.Double[]
| VECTOR(dimensions, FLOAT64) | -|
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • Embedding\
  • Embedding\
  • float[]
  • System.Float[]
| VECTOR(dimensions, FLOAT32) | - -### Property name override - -For data properties and vector properties, you can override names to use in storage that are different from the data model property names. The property name override occurs when setting the option either in the data model properties or record definition. - -Here is a data model with set code sample and how that will be represented in an Oracle SQL command. - -:::code language="csharp" source="./snippets/oracle-connector.cs" id="PropertyNameOverride"::: - -```SQL -CREATE TABLE "MYSCHEMA"."Hotels" - ("HotelId" NUMBER(10), - "hotel_name" NVARCHAR2(2000), - "hotel_description" NVARCHAR2(2000), - "DescriptionEmbedding" VECTOR(384, FLOAT32), - PRIMARY KEY ( "HotelId" ) -); -``` - -## See also - -To learn more, see the following Oracle Database Vector Store connector resources: - -- [Introducing the Oracle Database Vector Store Connector for Semantic Kernel](https://medium.com/oracledevs/announcing-the-oracle-database-vector-store-connector-for-semantic-kernel-adb83e806d4e) -Describes key connector features, classes, and guides the reader through a sample AI vector search application using the connector. -- [Documentation: Oracle Database Vector Store Connector Classes for Semantic Kernel (.NET) APIs](https://docs.oracle.com/en/database/oracle/oracle-database/23/odpnt/VSConnector4SKClasses.html) -Contains information on Oracle Database Vector Store connector classes for adding data, retrieving data, and performing vector search in the Oracle vector database. -- [Documentation: Oracle Data Provider for .NET](https://docs.oracle.com/en/database/oracle/oracle-database/23/odpnt/intro.html) -Contains information on Oracle Data Provider for .NET (ODP.NET), the ADO.NET data provider for Oracle Database Vector Store connector. \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md deleted file mode 100644 index bcd71209f591d..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/pinecone-connector.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: Using the Pinecone Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Pinecone. -ms.topic: concept-article -ms.date: 07/08/2024 ---- -# Using the Pinecone connector (Preview) - -> [!WARNING] -> The Pinecone Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Pinecone Vector Store connector can be used to access and manage data in Pinecone. The connector has the following characteristics. - -| Feature Area | Support | -|------------------------------|---------------------------| -| Collection maps to | Pinecone serverless Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • decimal
  • *enumerables of type* string
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types | PGA (Pinecone Graph Algorithm) | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | No | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | No | -| StorageName supported? | Yes | -| HybridSearch supported? | No | -| Integrated Embeddings supported? | No | - -## Get started - -Add the Pinecone Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.Pinecone --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of the `PineconeClient` to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted4"::: - -You can construct a Pinecone Vector Store instance directly. - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="GetStarted6"::: - -## Index namespace - -The Vector Store abstraction does not support a multi tiered record grouping mechanism. Collections in the abstraction map to a Pinecone serverless index -and no second level exists in the abstraction. Pinecone does support a second level of grouping called namespaces. - -By default the Pinecone connector will pass null as the namespace for all operations. However it is possible to pass a single namespace to the -Pinecone collection when constructing it and use this instead for all operations. - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="IndexNamespace"::: - -## Data mapping - -The Pinecone connector provides a default mapper when mapping data from the data model to storage. -Pinecone requires properties to be mapped into id, metadata and values groupings. -The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping. - -- The data model property annotated as a key will be mapped to the Pinecone id property. -- The data model properties annotated as data will be mapped to the Pinecone metadata object. -- The data model property annotated as a vector will be mapped to the Pinecone vector property. - -### Property name override - -For data properties, you can provide override field names to use in storage that is different to the -property names on the data model. This is not supported for keys, since a key has a fixed name in Pinecone. -It's also not supported for vectors, since the vector is stored under a fixed name `values`. -The property name override is done by setting the option via the data model attributes or record definition. - -Here is an example of a data model with set on its attributes and how that will be represented in Pinecone. - -:::code language="csharp" source="./snippets/pinecone-connector.cs" id="PropertyNameOverride"::: - -```json -{ - "id": "h1", - "values": [0.9, 0.1, 0.1, 0.1], - "metadata": { "hotel_name": "Hotel Happy", "hotel_description": "A place where everyone can be happy." } -} -``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md deleted file mode 100644 index 52a3e6c94e2b2..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/postgres-connector.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Using the Postgres Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Postgres. -ms.topic: concept-article -ms.date: 10/24/2024 ---- -# Using the Postgres Vector Store connector (Preview) - -> [!WARNING] -> The Postgres Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Postgres Vector Store connector can be used to access and manage data in Postgres and also supports [Neon Serverless Postgres](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/neon1722366567200.neon_serverless_postgres_azure_prod). - -The connector has the following characteristics. - -| Feature Area | Support | -|--|--| -| Collection maps to | Postgres table | -| Supported key property types |
  • short
  • int
  • long
  • string
  • Guid
| -| Supported data property types |
  • bool
  • short
  • int
  • long
  • float
  • double
  • decimal
  • string
  • DateTime
  • DateTimeOffset
  • Guid
  • byte[]
  • bool Enumerables
  • short Enumerables
  • int Enumerables
  • long Enumerables
  • float Enumerables
  • double Enumerables
  • decimal Enumerables
  • string Enumerables
  • DateTime Enumerables
  • DateTimeOffset Enumerables
  • Guid Enumerables
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • Half[]
  • BitArray
  • Pgvector.SparseVector
| -| Supported index types | Hnsw | -| Supported distance functions |
  • CosineDistance
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • ManhattanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | No | -| IsFullTextIndexed supported? | No | -| StorageName supported? | Yes | -| HybridSearch supported? | No | - -## Limitations - -> [!IMPORTANT] -> When initializing `NpgsqlDataSource` manually, it is necessary to call `UseVector` on the `NpgsqlDataSourceBuilder`. This enables vector support. Without this, usage of the VectorStore implementation will fail. - -Here is an example of how to call `UseVector`. - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="Limitations"::: - -When using the `AddPostgresVectorStore` dependency injection registration method with a connection string, the datasource will be constructed by this method and will automatically have `UseVector` applied. - -## Get started - -Add the Postgres Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.PgVector --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted1"::: - -Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects, for example `Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres`. - -Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that `UseVector` must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted2"::: - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted3"::: - -You can construct a Postgres Vector Store instance directly with a custom data source or with a connection string. - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted4"::: - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection with a custom data source or with a connection string. - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted6"::: - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="GetStarted7"::: - -## Data mapping - -The Postgres connector provides a default mapper when mapping data from the data model to storage. -The default mapper uses the model annotations or record definition to determine the type of each property and to map the model -into a Dictionary that can be serialized to Postgres. - -- The data model property annotated as a key will be mapped to the PRIMARY KEY in the Postgres table. -- The data model properties annotated as data will be mapped to a table column in Postgres. -- The data model properties annotated as vectors will be mapped to a table column that has the pgvector `VECTOR` type in Postgres. - -### Property name override - -You can provide override field names to use in storage that is different from the -property names on the data model. This allows you to match table column names even -if they don't match the property names on the data model. - -The property name override is done by setting the option via the data model attributes or record definition. - -Here is an example of a data model with set on its attributes and how it will be represented in Postgres as a table, assuming the Collection name is `Hotels`. - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="PropertyNameOverride"::: - -```sql -CREATE TABLE IF NOT EXISTS public."Hotels" ( - "hotel_id" INTEGER PRIMARY KEY NOT NULL, - hotel_name TEXT, - hotel_description TEXT, - hotel_description_embedding VECTOR(4) -); -``` - -## Vector indexing - -The `hotel_description_embedding` in the above `Hotel` model is a vector property with `IndexKind.HNSW` -indexing. This index will be created automatically when the collection is created. -HNSW is the only index type supported for index creation. IVFFlat index building requires that data already -exist in the table at index creation time, and so it is not appropriate for the creation of an empty table. -You are free to create and modify indexes on tables outside of the connector, which will -be used by the connector when performing queries. - -## Use with Entra authentication - -Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](/azure/postgresql/flexible-server/concepts-azure-ad-authentication). -This removes the need to store a username and password in your connection string. -To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following Npgsql extension method and set a connection string that does not have a username or password: - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="UseWithEntraAuthentication1"::: - -Now you can use the `UseEntraAuthentication` method to set up the connection string for the Postgres connector: - -:::code language="csharp" source="./snippets/postgres-connector.cs" id="UseWithEntraAuthentication2"::: - -By default, the `UseEntraAuthentication` method uses the [DefaultAzureCredential](/dotnet/api/azure.identity.defaultazurecredential) to authenticate with Azure AD. You can also provide a custom `TokenCredential` implementation if needed. \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md deleted file mode 100644 index 32b80d0857e60..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/qdrant-connector.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: Using the Qdrant Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Qdrant. -ms.topic: concept-article -ms.date: 07/08/2024 ---- -# Using the Qdrant connector (Preview) - -> [!WARNING] -> The Qdrant Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Qdrant Vector Store connector can be used to access and manage data in Qdrant. The connector has the following characteristics. - -| Feature Area | Support | -|------------------------------|-------------------------------------------------------------------| -| Collection maps to | Qdrant collection with payload indices for filterable data fields | -| Supported key property types |
  • ulong
  • Guid
| -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types | Hnsw | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • ManhattanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes (configurable) | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StorageName supported? | Yes | -| HybridSearch supported? | Yes | - -## Get started - -Add the Qdrant Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.Qdrant --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of the `Qdrant.Client.QdrantClient` class to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted4"::: - -You can construct a Qdrant Vector Store instance directly. - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="GetStarted6"::: - -## Data mapping - -The Qdrant connector provides a default mapper when mapping data from the data model to storage. -Qdrant requires properties to be mapped into id, payload and vector(s) groupings. -The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping. - -- The data model property annotated as a key will be mapped to the Qdrant point id. -- The data model properties annotated as data will be mapped to the Qdrant point payload object. -- The data model properties annotated as vectors will be mapped to the Qdrant point vector object. - -### Property name override - -For data properties and vector properties (if using named vectors mode), you can provide override field names to use in storage that is different to the -property names on the data model. This is not supported for keys, since a key has a fixed name in Qdrant. It's also not supported for vectors in *single -unnamed vector* mode, since the vector is stored under a fixed name. - -The property name override is done by setting the option via the data model attributes or record definition. - -Here is an example of a data model with set on its attributes and how that will be represented in Qdrant. - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="PropertyNameOverride"::: - -```json -{ - "id": 1, - "payload": { "hotel_name": "Hotel Happy", "hotel_description": "A place where everyone can be happy." }, - "vector": { - "hotel_description_embedding": [0.9, 0.1, 0.1, 0.1], - } -} -``` - -### Qdrant vector modes - -Qdrant supports two modes for vector storage and the Qdrant Connector with default mapper supports both modes. -The default mode is *single unnamed vector*. - -#### Single unnamed vector - -With this option a collection might only contain a single vector and it will be unnamed in the storage model in Qdrant. -Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="SingleUnnamedVector"::: - -```json -{ - "id": 1, - "payload": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." }, - "vector": [0.9, 0.1, 0.1, 0.1] -} -``` - -#### Named vectors - -If using the named vectors mode, it means that each point in a collection might contain more than one vector, and each will be named. -Here is an example of how an object is represented in Qdrant when using *named vectors* mode: - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="NamedVectors1"::: - -```json -{ - "id": 1, - "payload": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." }, - "vector": { - "HotelNameEmbedding": [0.9, 0.5, 0.5, 0.5], - "DescriptionEmbedding": [0.9, 0.1, 0.1, 0.1], - } -} -``` - -To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. -The same options can also be passed to any of the provided dependency injection container extension methods. - -:::code language="csharp" source="./snippets/qdrant-connector.cs" id="NamedVectors2"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md deleted file mode 100644 index 4b9abf2529cef..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/redis-connector.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: Using the Redis Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Redis. -ms.topic: concept-article -ms.date: 07/08/2024 ---- -# Using the Redis connector (Preview) - -> [!WARNING] -> The Redis Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Redis Vector Store connector can be used to access and manage data in Redis. The connector supports both Hashes and JSON modes and which mode you pick will determine what other features are supported. - -The connector has the following characteristics. - -| Feature Area | Support | -|--|--| -| Collection maps to | Redis index with prefix set to `:` | -| Supported key property types | string | -| Supported data property types | **When using Hashes:**
  • string
  • int
  • uint
  • long
  • ulong
  • double
  • float
  • bool
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
  • ReadOnlyMemory\
  • Embedding\
  • double[]
| -| Supported index types |
  • Hnsw
  • Flat
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StorageName supported? | **When using Hashes:** Yes
**When using JSON:** No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | No | - -## Get started - -Add the Redis Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.Redis --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted2"::: - -Extension methods that take no parameters are also provided. These require an instance of the Redis `IDatabase` to be separately registered with the dependency injection container. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted4"::: - -You can construct a Redis Vector Store instance directly. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. -When doing so, you have to choose between the JSON or Hashes instance depending on how you wish to store data in Redis. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted6"::: - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted7"::: - -When constructing a `RedisVectorStore` or registering it with the dependency injection container, it's possible to pass a `RedisVectorStoreOptions` instance -that configures the preferred storage type / mode used: Hashes or JSON. If not specified, the default is JSON. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="GetStarted8"::: - -## Index prefixes - -Redis uses a system of key prefixing to associate a record with an index. -When creating an index you can specify one or more prefixes to use with that index. -If you want to associate a record with that index, you have to add the prefix to the key of that record. - -For example, If you create a index called `skhotelsjson` with a prefix of `skhotelsjson:`, when setting a record -with key `h1`, the record key will need to be prefixed like this `skhotelsjson:h1` to be added to the index. - -When creating a new collection using the Redis connector, the connector will create an index in Redis with a -prefix consisting of the collection name and a colon, like this `:`. -By default, the connector will also prefix all keys with the this prefix when doing record operations like Get, Upsert, and Delete. - -If you didn't want to use a prefix consisting of the collection name and a colon, it is possible to switch -off the prefixing behavior and pass in the fully prefixed key to the record operations. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="IndexPrefixes"::: - -## Data mapping - -Redis supports two modes for storing data: JSON and Hashes. The Redis connector supports both storage types, and mapping differs depending on the chosen storage type. - -### Data mapping when using the JSON storage type - -When using the JSON storage type, the Redis connector will use `System.Text.Json.JsonSerializer` to do mapping. -Since Redis stores records with a separate key and value, the mapper will serialize all properties except for the key to a JSON object -and use that as the value. - -Usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the -data model property name is required. It's also possible to use a custom `JsonSerializerOptions` instance with a customized property naming policy. To enable this, the `JsonSerializerOptions` -must be passed to the `RedisJsonCollection` on construction. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="DataMappingWhenUsingTheJSONStorageType1"::: - -Since a naming policy of snake case upper was chosen, here is an example of how this data type will be set in Redis. -Also note the use of `JsonPropertyNameAttribute` on the `Description` property to further customize the storage naming. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="DataMappingWhenUsingTheJSONStorageType2"::: - -```redis -JSON.SET skhotelsjson:h1 $ '{ "HOTEL_NAME": "Hotel Happy", "HOTEL_DESCRIPTION": "A place where everyone can be happy.", "DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1] }' -``` - -### Data mapping when using the Hashes storage type - -When using the Hashes storage type, the Redis connector provides its own mapper to do mapping. -This mapper will map each property to a field-value pair as supported by the Redis `HSET` command. - -For data properties and vector properties, you can provide override field names to use in storage that is different to the -property names on the data model. This is not supported for keys, since keys cannot be named in Redis. - -Property name overriding is done by setting the option via the data model attributes or record definition. - -Here is an example of a data model with set on its attributes and how these are set in Redis. - -:::code language="csharp" source="./snippets/redis-connector.cs" id="DataMappingWhenUsingTheHashesStorageType"::: - -```redis -HSET skhotelshashes:h1 hotel_name "Hotel Happy" hotel_description 'A place where everyone can be happy.' hotel_description_embedding -``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj deleted file mode 100644 index a2840a4891709..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/ConnectorSnippets.csproj +++ /dev/null @@ -1,46 +0,0 @@ - - - - net10.0 - enable - enable - $(NoWarn);CS8019;CS0219;CS1591;CS8602 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs deleted file mode 100644 index 5569636cbb9bd..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-ai-search-connector.cs +++ /dev/null @@ -1,81 +0,0 @@ -// -using Azure; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); -// - -// -using Azure; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddAzureAISearchVectorStore(new Uri(azureAISearchUri), new AzureKeyCredential(secret)); -// - -// -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret))); -services.AddAzureAISearchVectorStore(); -// - -// -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret))); -builder.Services.AddAzureAISearchVectorStore(); -// - -// -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.SemanticKernel.Connectors.AzureAISearch; - -var vectorStore = new AzureAISearchVectorStore( - new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret))); -// - -// -using Azure; -using Azure.Search.Documents.Indexes; -using Microsoft.SemanticKernel.Connectors.AzureAISearch; - -var collection = new AzureAISearchCollection( - new SearchIndexClient(new Uri(azureAISearchUri), new AzureKeyCredential(secret)), - "skhotels"); -// - -// -var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; -var collection = new AzureAISearchCollection( - new SearchIndexClient( - new Uri(azureAISearchUri), - new AzureKeyCredential(secret), - new() { Serializer = new JsonObjectSerializer(jsonSerializerOptions) }), - "skhotels", - new() { JsonSerializerOptions = jsonSerializerOptions }); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs deleted file mode 100644 index 2cf6f2c12067c..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-mongodb-connector.cs +++ /dev/null @@ -1,91 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddCosmosMongoVectorStore(connectionString, databaseName); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddCosmosMongoVectorStore(connectionString, databaseName); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using MongoDB.Driver; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => - { - var mongoClient = new MongoClient(connectionString); - return mongoClient.GetDatabase(databaseName); - }); -services.AddCosmosMongoVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using MongoDB.Driver; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => - { - var mongoClient = new MongoClient(connectionString); - return mongoClient.GetDatabase(databaseName); - }); -builder.Services.AddCosmosMongoVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var vectorStore = new CosmosMongoVectorStore(database); -// - -// -using Microsoft.SemanticKernel.Connectors.CosmosMongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var collection = new CosmosMongoCollection( - database, - "skhotels"); -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [BsonElement("hotel_name")] - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [BsonElement("hotel_description")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [BsonElement("hotel_description_embedding")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs deleted file mode 100644 index 13a08e0327df6..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/azure-cosmosdb-nosql-connector.cs +++ /dev/null @@ -1,160 +0,0 @@ -// -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, -}); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddCosmosNoSqlVectorStore(connectionString, databaseName); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddCosmosNoSqlVectorStore(connectionString, databaseName); -// - -// -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => - { - var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() - { - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, - }); - - return cosmosClient.GetDatabase(databaseName); - }); -services.AddCosmosNoSqlVectorStore(); -// - -// -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => - { - var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() - { - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, - }); - - return cosmosClient.GetDatabase(databaseName); - }); -builder.Services.AddCosmosNoSqlVectorStore(); -// - -// -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.SemanticKernel.Connectors.CosmosNoSql; - -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, -}); - -var database = cosmosClient.GetDatabase(databaseName); -var vectorStore = new CosmosNoSqlVectorStore(database); -// - -// -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.SemanticKernel.Connectors.CosmosNoSql; - -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = JsonSerializerOptions.Default, -}); - -var database = cosmosClient.GetDatabase(databaseName); -var collection = new CosmosNoSqlCollection( - database, - "skhotels"); -// - -// -using System.Text.Json; -using Microsoft.Azure.Cosmos; -using Microsoft.SemanticKernel.Connectors.CosmosNoSql; - -var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; - -var cosmosClient = new CosmosClient(connectionString, new CosmosClientOptions() -{ - // When initializing CosmosClient manually, setting this property is required - // due to limitations in default serializer. - UseSystemTextJsonSerializerWithOptions = jsonSerializerOptions -}); - -var database = cosmosClient.GetDatabase(databaseName); -var collection = new CosmosNoSqlCollection( - database, - "skhotels", - new() { JsonSerializerOptions = jsonSerializerOptions }); -// - -// -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.EuclideanDistance, IndexKind = IndexKind.QuantizedFlat)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// - -// -var options = new CosmosNoSqlCollectionOptions -{ - PartitionKeyPropertyName = nameof(Hotel.HotelName) -}; - -var collection = new CosmosNoSqlCollection(database, "collection-name", options) - as VectorStoreCollection; -// - -// -var record = await collection.GetAsync(new CosmosNoSqlCompositeKey("hotel-id", "hotel-name")); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs deleted file mode 100644 index 0289629e677dc..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/couchbase-connector.cs +++ /dev/null @@ -1,225 +0,0 @@ -// -using Microsoft.SemanticKernel; -using Couchbase.SemanticKernel; - -// Using a ServiceCollection. -var kernelBuilder = Kernel - .CreateBuilder() - .AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name"); -// - -// -using Couchbase.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name"); -// - -// -using Couchbase.SemanticKernel; - -var builder = WebApplication.CreateBuilder(args); - -// Option 1: Use Hyperscale index -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name", - options: new CouchbaseVectorStoreOptions - { - IndexType = CouchbaseIndexType.Hyperscale - }); - -// Option 2: Use Composite index -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name", - options: new CouchbaseVectorStoreOptions - { - IndexType = CouchbaseIndexType.Composite - }); - -// Option 3: Use Search vector index -builder.Services.AddCouchbaseVectorStore( - connectionString: "couchbases://your-cluster-address", - username: "username", - password: "password", - bucketName: "bucket-name", - scopeName: "scope-name", - options: new CouchbaseVectorStoreOptions - { - IndexType = CouchbaseIndexType.Search - }); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Couchbase; -using Couchbase.KeyValue; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => -{ - var clusterOptions = new ClusterOptions - { - ConnectionString = "couchbases://your-cluster-address", - UserName = "username", - Password = "password" - }; - - return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); -}); - -services.AddSingleton(sp => -{ - var cluster = sp.GetRequiredService(); - var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); - return bucket.Scope("scope-name"); -}); - -// Add Couchbase Vector Store -services.AddCouchbaseVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Couchbase.KeyValue; -using Couchbase; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); - -builder.Services.AddSingleton(sp => -{ - var clusterOptions = new ClusterOptions - { - ConnectionString = "couchbases://your-cluster-address", - UserName = "username", - Password = "password" - }; - - return Cluster.ConnectAsync(clusterOptions).GetAwaiter().GetResult(); -}); - -builder.Services.AddSingleton(sp => -{ - var cluster = sp.GetRequiredService(); - var bucket = cluster.BucketAsync("bucket-name").GetAwaiter().GetResult(); - return bucket.Scope("scope-name"); -}); - -// Add Couchbase Vector Store -builder.Services.AddCouchbaseVectorStore(); -// - -// -using Couchbase; -using Couchbase.KeyValue; -using Couchbase.SemanticKernel; - -var clusterOptions = new ClusterOptions -{ - ConnectionString = "couchbases://your-cluster-address", - UserName = "username", - Password = "password" -}; - -var cluster = await Cluster.ConnectAsync(clusterOptions); -var bucket = await cluster.BucketAsync("bucket-name"); -var scope = bucket.Scope("scope-name"); - -var vectorStore = new CouchbaseVectorStore(scope); -// - -// -using Couchbase.SemanticKernel; -using Couchbase; -using Couchbase.KeyValue; - -var cluster = await Cluster.ConnectAsync(clusterOptions); -var bucket = await cluster.BucketAsync("bucket-name"); -var scope = bucket.Scope("scope-name"); - -// Using Hyperscale index (default) -var collection = new CouchbaseQueryCollection( - scope, - "skhotels", - indexType: CouchbaseIndexType.Hyperscale); - -// Or using Composite index -var collectionComposite = new CouchbaseQueryCollection( - scope, - "skhotels", - indexType: CouchbaseIndexType.Composite); -// - -// -using Couchbase.SemanticKernel; -using Couchbase; -using Couchbase.KeyValue; - -var cluster = await Cluster.ConnectAsync(clusterOptions); -var bucket = await cluster.BucketAsync("bucket-name"); -var scope = bucket.Scope("scope-name"); - -var collection = new CouchbaseSearchCollection( - scope, - "skhotels"); -// - -// -using Couchbase.SemanticKernel; -using Couchbase.KeyValue; -using System.Text.Json; - -var jsonSerializerOptions = new JsonSerializerOptions -{ - PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper -}; - -var options = new CouchbaseQueryCollectionOptions -{ - JsonSerializerOptions = jsonSerializerOptions -}; - -var collection = new CouchbaseQueryCollection(scope, "skhotelsjson", options); -// - -// -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs deleted file mode 100644 index 89ad2bb28d2a1..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/elasticsearch-connector.cs +++ /dev/null @@ -1,112 +0,0 @@ -// -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; - -// Using a ServiceCollection. -var kernelBuilder = Kernel - .CreateBuilder() - .AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); -// - -// -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddElasticsearchVectorStore(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); -services.AddElasticsearchVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Elastic.Clients.Elasticsearch; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); -builder.Services.AddElasticsearchVectorStore(); -// - -// -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; - -var vectorStore = new ElasticsearchVectorStore( - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200")))); -// - -// -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; - -var collection = new ElasticsearchVectorStoreRecordCollection( - new ElasticsearchClient(new ElasticsearchClientSettings(new Uri("http://localhost:9200"))), - "skhotels"); -// - -// -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; -using Elastic.Clients.Elasticsearch.Serialization; -using Elastic.Transport; - -var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); -var settings = new ElasticsearchClientSettings( - nodePool, - sourceSerializer: (defaultSerializer, settings) => - new DefaultSourceSerializer(settings, options => - options.PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper)); -var client = new ElasticsearchClient(settings); - -var collection = new ElasticsearchVectorStoreRecordCollection( - client, - "skhotelsjson"); -// - -// -using Elastic.SemanticKernel.Connectors.Elasticsearch; -using Elastic.Clients.Elasticsearch; - -var settings = new ElasticsearchClientSettings(new Uri("http://localhost:9200")); -settings.DefaultFieldNameInferrer(name => JsonNamingPolicy.SnakeCaseUpper.ConvertName(name)); -var client = new ElasticsearchClient(settings); - -var collection = new ElasticsearchVectorStoreRecordCollection( - client, - "skhotelsjson"); -// - -// -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction.CosineSimilarity, IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs deleted file mode 100644 index ec4af5a1939f7..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/inmemory-connector.cs +++ /dev/null @@ -1,29 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddInMemoryVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddInMemoryVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.InMemory; - -var vectorStore = new InMemoryVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.InMemory; - -var collection = new InMemoryCollection("skhotels"); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs deleted file mode 100644 index ce0cc8b7b9feb..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/mongodb-connector.cs +++ /dev/null @@ -1,65 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddMongoVectorStore(connectionString, databaseName); -// - -// -using Microsoft.Extensions.DependencyInjection; -using MongoDB.Driver; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => - { - var mongoClient = new MongoClient(connectionString); - return mongoClient.GetDatabase(databaseName); - }); -builder.Services.AddMongoVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.MongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var vectorStore = new MongoVectorStore(database); -// - -// -using Microsoft.SemanticKernel.Connectors.MongoDB; -using MongoDB.Driver; - -var mongoClient = new MongoClient(connectionString); -var database = mongoClient.GetDatabase(databaseName); -var collection = new MongoCollection( - database, - "skhotels"); -// - -// -using Microsoft.Extensions.VectorData; -using MongoDB.Bson.Serialization.Attributes; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [BsonElement("hotel_name")] - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [BsonElement("hotel_description")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [BsonElement("hotel_description_embedding")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs deleted file mode 100644 index beec21e035efa..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/oracle-connector.cs +++ /dev/null @@ -1,102 +0,0 @@ -// -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddOracleVectorStore(""); -// - -// -using Microsoft.AspNetCore.Builder; -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddOracleVectorStore(""); -// - -// -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; -using Oracle.ManagedDataAccess.Client; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => -{ - OracleDataSourceBuilder dataSourceBuilder = new(""); - return dataSourceBuilder.Build(); -}); - -services.AddOracleVectorStore(); -// - -// -using Microsoft.AspNetCore.Builder; -using Oracle.VectorData; -using Microsoft.Extensions.DependencyInjection; -using Oracle.ManagedDataAccess.Client; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => -{ - OracleDataSourceBuilder dataSourceBuilder = new(""); - return dataSourceBuilder.Build(); -}); - -builder.Services.AddOracleVectorStore(); -// - -// -using Oracle.VectorData; -using Oracle.ManagedDataAccess.Client; - -OracleDataSourceBuilder dataSourceBuilder = new(""); -var dataSource = dataSourceBuilder.Build(); - -var connection = new OracleVectorStore(dataSource); -// - -// -using Oracle.VectorData; - -var connection = new OracleVectorStore(""); -// - -// -using Oracle.VectorData; -using Oracle.ManagedDataAccess.Client; - -OracleDataSourceBuilder dataSourceBuilder = new(""); -var dataSource = dataSourceBuilder.Build(); - -var collection = new OracleCollection(dataSource, "skhotels"); -// - -// -using Oracle.VectorData; - -var collection = new OracleCollection("", "skhotels"); -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public long HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string? HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 384, DistanceFunction = DistanceFunction.CosineDistance)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs deleted file mode 100644 index ad5e18f7632bb..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/pinecone-connector.cs +++ /dev/null @@ -1,86 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddPineconeVectorStore(pineconeApiKey); -// - -// -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddPineconeVectorStore(pineconeApiKey); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using PineconeClient = Pinecone.PineconeClient; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton( - sp => new PineconeClient(pineconeApiKey)); -services.AddPineconeVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using PineconeClient = Pinecone.PineconeClient; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton( - sp => new PineconeClient(pineconeApiKey)); -builder.Services.AddPineconeVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.Pinecone; -using PineconeClient = Pinecone.PineconeClient; - -var vectorStore = new PineconeVectorStore( - new PineconeClient(pineconeApiKey)); -// - -// -using Microsoft.SemanticKernel.Connectors.Pinecone; -using PineconeClient = Pinecone.PineconeClient; - -var collection = new PineconeCollection( - new PineconeClient(pineconeApiKey), - "skhotels"); -// - -// -using Microsoft.SemanticKernel.Connectors.Pinecone; -using PineconeClient = Pinecone.PineconeClient; - -var collection = new PineconeCollection( - new PineconeClient(pineconeApiKey), - "skhotels", - new() { IndexNamespace = "seasidehotels" }); -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs deleted file mode 100644 index d344d58eac67e..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/postgres-connector.cs +++ /dev/null @@ -1,220 +0,0 @@ -// -NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); -dataSourceBuilder.UseVector(); -NpgsqlDataSource dataSource = dataSourceBuilder.Build(); -// - -// -using Microsoft.Extensions.DependencyInjection; - -var services = new ServiceCollection(); -services.AddPostgresVectorStore(""); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddPostgresVectorStore(""); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Npgsql; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => -{ - NpgsqlDataSourceBuilder dataSourceBuilder = new(""); - dataSourceBuilder.UseVector(); - return dataSourceBuilder.Build(); -}); -builder.Services.AddPostgresVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.PgVector; -using Npgsql; - -NpgsqlDataSourceBuilder dataSourceBuilder = new(""); -dataSourceBuilder.UseVector(); -NpgsqlDataSource dataSource = dataSourceBuilder.Build(); -var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); -// - -// -using Microsoft.SemanticKernel.Connectors.PgVector; - -var connection = new PostgresVectorStore(""); -// - -// -using Microsoft.SemanticKernel.Connectors.PgVector; -using Npgsql; - -NpgsqlDataSourceBuilder dataSourceBuilder = new(""); -dataSourceBuilder.UseVector(); -var dataSource = dataSourceBuilder.Build(); - -var collection = new PostgresCollection(dataSource, "skhotels", ownsDataSource: true); -// - -// -using Microsoft.SemanticKernel.Connectors.PgVector; - -var collection = new PostgresCollection("", "skhotels"); -// - -// -using System; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey(StorageName = "hotel_id")] - public int HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// - -// -using System.Text; -using System.Text.Json; -using Azure.Core; -using Azure.Identity; -using Npgsql; - -namespace Program; - -public static class NpgsqlDataSourceBuilderExtensions -{ - private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new(["https://ossrdbms-aad.database.windows.net/.default"]); - - public static NpgsqlDataSourceBuilder UseEntraAuthentication(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default) - { - credential ??= new DefaultAzureCredential(); - - if (dataSourceBuilder.ConnectionStringBuilder.Username == null) - { - var token = credential.GetToken(s_azureDBForPostgresTokenRequestContext, default); - SetUsernameFromToken(dataSourceBuilder, token.Token); - } - - SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); - - return dataSourceBuilder; - } - - public static async Task UseEntraAuthenticationAsync(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default, CancellationToken cancellationToken = default) - { - credential ??= new DefaultAzureCredential(); - - if (dataSourceBuilder.ConnectionStringBuilder.Username == null) - { - var token = await credential.GetTokenAsync(s_azureDBForPostgresTokenRequestContext, cancellationToken).ConfigureAwait(false); - SetUsernameFromToken(dataSourceBuilder, token.Token); - } - - SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); - - return dataSourceBuilder; - } - - private static void SetPasswordProvider(NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential credential, TokenRequestContext tokenRequestContext) - { - dataSourceBuilder.UsePasswordProvider(_ => - { - var token = credential.GetToken(tokenRequestContext, default); - return token.Token; - }, async (_, ct) => - { - var token = await credential.GetTokenAsync(tokenRequestContext, ct).ConfigureAwait(false); - return token.Token; - }); - } - - private static void SetUsernameFromToken(NpgsqlDataSourceBuilder dataSourceBuilder, string token) - { - var username = TryGetUsernameFromToken(token); - - if (username != null) - { - dataSourceBuilder.ConnectionStringBuilder.Username = username; - } - else - { - throw new Exception("Could not determine username from token claims"); - } - } - - private static string? TryGetUsernameFromToken(string jwtToken) - { - // Split the token into its parts (Header, Payload, Signature) - var tokenParts = jwtToken.Split('.'); - if (tokenParts.Length != 3) - { - return null; - } - - // The payload is the second part, Base64Url encoded - var payload = tokenParts[1]; - - // Add padding if necessary - payload = AddBase64Padding(payload); - - // Decode the payload from Base64Url - var decodedBytes = Convert.FromBase64String(payload); - var decodedPayload = Encoding.UTF8.GetString(decodedBytes); - - // Parse the decoded payload as JSON - var payloadJson = JsonSerializer.Deserialize(decodedPayload); - - // Try to get the username from 'upn', 'preferred_username', or 'unique_name' claims - if (payloadJson.TryGetProperty("upn", out var upn)) - { - return upn.GetString(); - } - else if (payloadJson.TryGetProperty("preferred_username", out var preferredUsername)) - { - return preferredUsername.GetString(); - } - else if (payloadJson.TryGetProperty("unique_name", out var uniqueName)) - { - return uniqueName.GetString(); - } - - return null; - } - - private static string AddBase64Padding(string base64) => (base64.Length % 4) switch - { - 2 => base64 + "==", - 3 => base64 + "=", - _ => base64, - }; -} -// - -// -using Microsoft.SemanticKernel.Connectors.Postgres; - -var connectionString = "Host=mydb.postgres.database.azure.com;Port=5432;Database=postgres;SSL Mode=Require;"; // No Username or Password -var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); -dataSourceBuilder.UseEntraAuthentication(); -dataSourceBuilder.UseVector(); -var dataSource = dataSourceBuilder.Build(); - -var vectorStore = new PostgresVectorStore(dataSource, ownsDataSource: true); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs deleted file mode 100644 index 153a96107df6c..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/qdrant-connector.cs +++ /dev/null @@ -1,111 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddQdrantVectorStore("localhost"); -// - -// -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddQdrantVectorStore("localhost"); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Qdrant.Client; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => new QdrantClient("localhost")); -services.AddQdrantVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Qdrant.Client; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => new QdrantClient("localhost")); -builder.Services.AddQdrantVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Qdrant.Client; - -var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -// - -// -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Qdrant.Client; - -var collection = new QdrantCollection( - new QdrantClient("localhost"), - "skhotels", - ownsClient: true); -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// - -// -new Hotel -{ - HotelId = 1, - HotelName = "Hotel Happy", - Description = "A place where everyone can be happy.", - DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } -}; -// - -// -new Hotel -{ - HotelId = 1, - HotelName = "Hotel Happy", - Description = "A place where everyone can be happy.", - HotelNameEmbedding = new float[4] { 0.9f, 0.5f, 0.5f, 0.5f } - DescriptionEmbedding = new float[4] { 0.9f, 0.1f, 0.1f, 0.1f } -}; -// - -// -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Qdrant.Client; - -var vectorStore = new QdrantVectorStore( - new QdrantClient("localhost"), - ownsClient: true, - new() { HasNamedVectors = true }); - -var collection = new QdrantCollection( - new QdrantClient("localhost"), - "skhotels", - ownsClient: true, - new() { HasNamedVectors = true }); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs deleted file mode 100644 index 857042773e92c..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/redis-connector.cs +++ /dev/null @@ -1,134 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddRedisVectorStore("localhost:6379"); -// - -// -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddRedisVectorStore("localhost:6379"); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using StackExchange.Redis; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); -services.AddRedisVectorStore(); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using StackExchange.Redis; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); -builder.Services.AddRedisVectorStore(); -// - -// -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -var vectorStore = new RedisVectorStore(ConnectionMultiplexer.Connect("localhost:6379").GetDatabase()); -// - -// -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -// Using Hashes. -var hashesCollection = new RedisHashSetCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelshashes"); -// - -// -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -// Using JSON. -var jsonCollection = new RedisJsonCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelsjson"); -// - -// -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -var vectorStore = new RedisVectorStore( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - new() { StorageType = RedisStorageType.HashSet }); -// - -// -using Microsoft.SemanticKernel.Connectors.Redis; -using StackExchange.Redis; - -var collection = new RedisJsonCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelsjson", - new() { PrefixCollectionNameToKeyNames = false }); - -await collection.GetAsync("myprefix_h1"); -// - -// -var jsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseUpper }; -var collection = new RedisJsonCollection( - ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), - "skhotelsjson", - new() { JsonSerializerOptions = jsonSerializerOptions }); -// - -// -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION")] - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public string HotelId { get; set; } - - [VectorStoreData(IsIndexed = true, StorageName = "hotel_name")] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true, StorageName = "hotel_description")] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw, StorageName = "hotel_description_embedding")] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs deleted file mode 100644 index 9ab75e7613f69..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sql-connector.cs +++ /dev/null @@ -1,47 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqlServerVectorStore(_ => ""); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqlServerVectorStore(_ => "") -// - -// -using Microsoft.SemanticKernel.Connectors.SqlServer; - -var vectorStore = new SqlServerVectorStore(""); -// - -// -using Microsoft.SemanticKernel.Connectors.SqlServer; - -var collection = new SqlServerCollection("", "skhotels"); -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string? HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs deleted file mode 100644 index a0197f7e52a8d..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/sqlite-connector.cs +++ /dev/null @@ -1,47 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:"); -// - -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSqliteVectorStore(_ => "Data Source=:memory:") -// - -// -using Microsoft.SemanticKernel.Connectors.SqliteVec; - -var vectorStore = new SqliteVectorStore("Data Source=:memory:"); -// - -// -using Microsoft.SemanticKernel.Connectors.SqliteVec; - -var collection = new SqliteCollection("Data Source=:memory:", "skhotels"); -// - -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(StorageName = "hotel_name")] - public string? HotelName { get; set; } - - [VectorStoreData(StorageName = "hotel_description")] - public string? Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineDistance)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs deleted file mode 100644 index 2c9612b5f166c..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/volatile-connector.cs +++ /dev/null @@ -1,28 +0,0 @@ -// -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var kernelBuilder = Kernel - .CreateBuilder() - .AddVolatileVectorStore(); -// - -// -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddVolatileVectorStore(); -// - -// -using Microsoft.SemanticKernel.Data; - -var vectorStore = new VolatileVectorStore(); -// - -// -using Microsoft.SemanticKernel.Data; - -var collection = new VolatileVectorStoreRecordCollection("skhotels"); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs b/docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs deleted file mode 100644 index 54670662bd596..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/snippets/weaviate-connector.cs +++ /dev/null @@ -1,82 +0,0 @@ -// -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); -// - -// -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), apiKey: null); -// - -// -using System.Net.Http; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; - -// Using a ServiceCollection. -var services = new ServiceCollection(); -using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; -services.AddWeaviateVectorStore(_ => client); -// - -// -using System.Net.Http; -using Microsoft.Extensions.DependencyInjection; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }; -builder.Services.AddWeaviateVectorStore(_ => client); -// - -// -using System.Net.Http; -using Microsoft.SemanticKernel.Connectors.Weaviate; - -var vectorStore = new WeaviateVectorStore( - new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }); -// - -// -using System.Net.Http; -using Microsoft.SemanticKernel.Connectors.Weaviate; - -var collection = new WeaviateCollection( - new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") }, - "Skhotels"); -// - -// -using Microsoft.SemanticKernel; - -var services = new ServiceCollection(); -services.AddWeaviateVectorStore(new Uri("http://localhost:8080/v1/"), secretVar); -// - -// -using System.Text.Json.Serialization; -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public Guid HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")] - [VectorStoreVector(4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.QuantizedFlat)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md deleted file mode 100644 index fa1e2fef5c1c7..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/sql-connector.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -title: Using the SQL Server Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in SQL Server. -author: eavanvalkenburg -ms.topic: concept-article -ms.author: edvan -ms.date: 03/21/2024 ---- -# Using the SQL Server Vector Store connector (Preview) - -> [!WARNING] -> The Sql Server Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The SQL Server Vector Store connector can be used to access and manage data in SQL Server. The connector has the following characteristics. - -| Feature Area | Support | -|--|--| -| Collection maps to | SQL Server table | -| Supported key property types |
  • int
  • long
  • string
  • Guid
  • DateTime
  • byte[]
| -| Supported data property types |
  • int
  • short
  • byte
  • long
  • Guid
  • string
  • bool
  • float
  • double
  • decimal
  • byte[]
  • DateTime
  • TimeOnly
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types |
  • Flat
| -| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanDistance
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | No | -| StorageName supported? | Yes | -| HybridSearch supported? | No | - -## Get started - -Add the SQL Sever Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.SqlServer --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted2"::: - -You can construct a Sql Server Vector Store instance directly. - -:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted3"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/sql-connector.cs" id="GetStarted4"::: - -## Data mapping - -The SQL Server Vector Store connector provides a default mapper when mapping from the data model to storage. -This mapper does a direct conversion of the list of properties on the data model to the columns in SQL Server. - -### Property name override - -You can provide override property names to use in storage that is different to the property names on the data model. -The property name override is done by setting the option via the data model property attributes or record definition. - -Here is an example of a data model with set on its attributes and how that will be represented in a SQL Server command. - -:::code language="csharp" source="./snippets/sql-connector.cs" id="PropertyNameOverride"::: - -```sql -CREATE TABLE Hotel ( -[HotelId] BIGINT NOT NULL, -[hotel_name] NVARCHAR(MAX), -[hotel_description] NVARCHAR(MAX), -[DescriptionEmbedding] VECTOR(4), -PRIMARY KEY ([HotelId]) -); -``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md deleted file mode 100644 index 01aa8bfb1e34c..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/sqlite-connector.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: Using the SQLite Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in SQLite. -author: dmytrostruk -ms.topic: concept-article -ms.author: dmytrostruk -ms.date: 10/24/2024 ---- -# Using the SQLite Vector Store connector (Preview) - -> [!WARNING] -> The Sqlite Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The SQLite Vector Store connector can be used to access and manage data in SQLite. The connector has the following characteristics. - -| Feature Area | Support | -|--|--| -| Collection maps to | SQLite table | -| Supported key property types |
  • int
  • long
  • string
| -| Supported data property types |
  • int
  • long
  • short
  • string
  • bool
  • float
  • double
  • byte[]
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types | N/A | -| Supported distance functions |
  • CosineDistance
  • ManhattanDistance
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | No | -| IsFullTextIndexed supported? | No | -| StorageName supported? | Yes | -| HybridSearch supported? | No | - -## Get started - -Add the SQLite Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.SqliteVec --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods from the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted2"::: - -You can construct a SQLite Vector Store instance directly. - -:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted3"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/sqlite-connector.cs" id="GetStarted4"::: - -## Data mapping - -The SQLite Vector Store connector provides a default mapper when mapping from the data model to storage. -This mapper does a direct conversion of the list of properties on the data model to the columns in SQLite. - -With the vector search extension, vectors are stored in virtual tables, separately from key and data properties. -By default, the virtual table with vectors will use the same name as the table with key and data properties, but with a `vec_` prefix. For example, if the collection name in `SqliteCollection` is `skhotels`, the name of the virtual table with vectors will be `vec_skhotels`. It's possible to override the virtual table name by using the `SqliteVectorStoreOptions.VectorVirtualTableName` or `SqliteCollectionOptions.VectorVirtualTableName` properties. - -### Property name override - -You can provide override property names to use in storage that is different to the property names on the data model. -The property name override is done by setting the option via the data model property attributes or record definition. - -Here is an example of a data model with set on its attributes and how that will be represented in a SQLite command. - -:::code language="csharp" source="./snippets/sqlite-connector.cs" id="PropertyNameOverride"::: - -```tsql -CREATE TABLE Hotels ( - HotelId INTEGER PRIMARY KEY, - hotel_name TEXT, - hotel_description TEXT -); - -CREATE VIRTUAL TABLE vec_Hotels ( - HotelId INTEGER PRIMARY KEY, - DescriptionEmbedding FLOAT[4] distance_metric=cosine -); -``` \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md deleted file mode 100644 index d7ff65676f076..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/volatile-connector.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: Using the Volatile (In-Memory) Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data using the Volatile (in-memory) vector store. -ms.topic: concept-article -ms.date: 07/08/2024 ---- -# Using the Volatile (In-Memory) connector (Preview) - -> [!WARNING] -> The C# VolatileVectorStore is obsolete and has been replaced with a new package. See [InMemory Connector](./inmemory-connector.md) - -## Overview - -The Volatile Vector Store connector is a Vector Store implementation that uses no external database and stores data in memory. -This Vector Store is useful for prototyping scenarios or where high-speed in-memory operations are required. - -The connector has the following characteristics. - -| Feature Area | Support | -|---------------------------------------|-------------------------------| -| Collection maps to | In-memory dictionary | -| Supported key property types | Any type that can be compared | -| Supported data property types | Any type | -| Supported vector property types | ReadOnlyMemory\ | -| Supported index types | N/A | -| Supported distance functions | N/A | -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StoragePropertyName supported? | No, since storage is volatile and data reuse is therefore not possible, custom naming is not useful and not supported. | - -## Get started - -Add the Semantic Kernel Core NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Core -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. - -:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted2"::: - -You can construct a Volatile Vector Store instance directly. - -:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted3"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/volatile-connector.cs" id="GetStarted4"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md b/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md deleted file mode 100644 index 7c73490632bd0..0000000000000 --- a/docs/ai/vector-stores/out-of-the-box-connectors/weaviate-connector.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Using the Weaviate Vector Store connector (Preview) -description: Contains information on how to use a Vector Store connector to access and manipulate data in Weaviate. -ms.topic: concept-article -ms.date: 09/23/2024 ---- -# Using the Weaviate Vector Store connector (Preview) - -> [!WARNING] -> The Weaviate Vector Store functionality is in preview, and improvements that require breaking changes might still occur in limited circumstances before release. - -## Overview - -The Weaviate Vector Store connector can be used to access and manage data in Weaviate. The connector has the following characteristics. - -| Feature Area | Support | -|------------------------------|---------------------| -| Collection maps to | Weaviate Collection | -| Supported key property types | Guid | -| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • DateTimeOffset
  • Guid
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • Embedding\
  • float[]
| -| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| -| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • HammingDistance
  • ManhattanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsIndexed supported? | Yes | -| IsFullTextIndexed supported? | Yes | -| StorageName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | -| HybridSearch supported? | Yes | - -## Limitations - -Notable Weaviate connector functionality limitations. - -| Feature Area | Workaround | -|--|--| -| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | - -> [!WARNING] -> Weaviate requires collection names to start with an upper case letter. If you do not provide a collection name with an upper case letter, Weaviate will return an error when you try and create your collection. The error that you will see is `Cannot query field "mycollection" on type "GetObjectsObj". Did you mean "Mycollection"?` where `mycollection` is your collection name. In this example, if you change your collection name to `Mycollection` instead, this will fix the error. - -## Get started - -Add the Weaviate Vector Store connector NuGet package to your project. - -```dotnetcli -dotnet add package Microsoft.SemanticKernel.Connectors.Weaviate --prerelease -``` - -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by the Semantic Kernel connector packages. -The Weaviate vector store uses an `HttpClient` to communicate with the Weaviate service. There are two options for providing the URL/endpoint for the Weaviate service. -It can be provided via options or by setting the base address of the `HttpClient`. - -This first example shows how to set the service URL via options. -Also note that these methods will retrieve an `HttpClient` instance for making calls to the Weaviate service from the dependency injection service provider. - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted1"::: - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted2"::: - -Overloads where you can specify your own `HttpClient` are also provided. -In this case it's possible to set the service url via the `HttpClient` `BaseAddress` option. - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted3"::: - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted4"::: - -You can construct a Weaviate Vector Store instance directly as well. - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted5"::: - -It's possible to construct a direct reference to a named collection. - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted6"::: - -If needed, it is possible to pass an API key, as an option, when using any of the previously mentioned mechanisms. For example: - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="GetStarted7"::: - -## Data mapping - -The Weaviate Vector Store connector provides a default mapper when mapping from the data model to storage. -Weaviate requires properties to be mapped into id, payload and vectors groupings. -The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping. - -- The data model property annotated as a key will be mapped to the Weaviate `id` property. -- The data model properties annotated as data will be mapped to the Weaviate `properties` object. -- The data model properties annotated as vectors will be mapped to the Weaviate `vectors` object. - -The default mapper uses `System.Text.Json.JsonSerializer` to convert to the storage schema. -This means that usage of the `JsonPropertyNameAttribute` is supported if a different storage name to the -data model property name is required. - -Here is an example of a data model with `JsonPropertyNameAttribute` set and how that will be represented in Weaviate. - -:::code language="csharp" source="./snippets/weaviate-connector.cs" id="DataMapping"::: - -```json -{ - "id": "11111111-1111-1111-1111-111111111111", - "properties": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." }, - "vectors": { - "HOTEL_DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1], - } -} -``` \ No newline at end of file From 1dba7443cf8b9e9d85806b61a24f5c062703b946 Mon Sep 17 00:00:00 2001 From: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Date: Sat, 28 Feb 2026 18:40:14 -0800 Subject: [PATCH 10/11] move code to code files with projects --- .../vector-stores/defining-your-data-model.md | 35 +- docs/ai/vector-stores/dynamic-data-model.md | 29 +- docs/ai/vector-stores/embedding-generation.md | 67 ++- .../how-to/build-vector-search-app.md | 14 +- .../how-to/build-your-own-connector.md | 63 ++- .../AdditionalExamples.cs | 70 --- .../VectorStoresExamples/AutoEmbedding.cs | 56 --- .../VectorStoresExamples/RecordDefinition.cs | 35 -- .../VectorStoresExamples.csproj | 15 - .../snippets/build-your-own-connector.cs | 19 - .../vector-stores/how-to/use-vector-stores.md | 65 +-- .../how-to/vector-store-data-ingestion.md | 23 +- docs/ai/vector-stores/hybrid-search.md | 82 ++- docs/ai/vector-stores/overview.md | 39 +- .../schema-with-record-definition.md | 75 ++- .../azure-openai/CloudService.cs | 0 .../chat-with-data/azure-openai/Program.cs | 0 .../azure-openai/VectorDataAI.csproj | 0 .../chat-with-data/openai/CloudService.cs | 0 .../snippets/chat-with-data/openai/Program.cs | 0 .../chat-with-data/openai/VectorDataAI.csproj | 0 .../VectorStoreSnippets.csproj | 9 +- .../conceptual/defining-your-data-model.cs | 27 + .../snippets/conceptual/dynamic-data-model.cs | 39 ++ .../conceptual/embedding-generation.cs | 191 +++++++ .../snippets/conceptual/hybrid-search.cs | 188 +++++++ .../schema-with-record-definition.cs | 20 + .../snippets/conceptual/vector-search.cs | 177 +++++++ .../snippets/defining-your-data-model.cs | 41 -- .../snippets/dynamic-data-model.cs | 39 -- .../snippets/embedding-generation.cs | 202 -------- .../how-to/DataIngestion.cs} | 83 +--- .../how-to}/Hotel.cs | 2 +- .../how-to}/HowToSnippets.csproj | 7 +- .../how-to}/Program.cs | 10 +- .../snippets/how-to/PutItAllTogether.cs | 50 ++ .../vector-stores/snippets/hybrid-search.cs | 144 ------ .../snippets/schema-with-record-definition.cs | 30 -- .../vector-stores/snippets/vector-search.cs | 139 ------ .../vector-stores/tutorial-vector-search.md | 470 +++++++++--------- docs/ai/vector-stores/vector-search.md | 58 +-- 41 files changed, 1184 insertions(+), 1429 deletions(-) delete mode 100644 docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs delete mode 100644 docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AutoEmbedding.cs delete mode 100644 docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/RecordDefinition.cs delete mode 100644 docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/VectorStoresExamples.csproj delete mode 100644 docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs rename docs/ai/{quickstarts => vector-stores}/snippets/chat-with-data/azure-openai/CloudService.cs (100%) rename docs/ai/{quickstarts => vector-stores}/snippets/chat-with-data/azure-openai/Program.cs (100%) rename docs/ai/{quickstarts => vector-stores}/snippets/chat-with-data/azure-openai/VectorDataAI.csproj (100%) rename docs/ai/{quickstarts => vector-stores}/snippets/chat-with-data/openai/CloudService.cs (100%) rename docs/ai/{quickstarts => vector-stores}/snippets/chat-with-data/openai/Program.cs (100%) rename docs/ai/{quickstarts => vector-stores}/snippets/chat-with-data/openai/VectorDataAI.csproj (100%) rename docs/ai/vector-stores/snippets/{ => conceptual}/VectorStoreSnippets.csproj (82%) create mode 100644 docs/ai/vector-stores/snippets/conceptual/defining-your-data-model.cs create mode 100644 docs/ai/vector-stores/snippets/conceptual/dynamic-data-model.cs create mode 100644 docs/ai/vector-stores/snippets/conceptual/embedding-generation.cs create mode 100644 docs/ai/vector-stores/snippets/conceptual/hybrid-search.cs create mode 100644 docs/ai/vector-stores/snippets/conceptual/schema-with-record-definition.cs create mode 100644 docs/ai/vector-stores/snippets/conceptual/vector-search.cs delete mode 100644 docs/ai/vector-stores/snippets/defining-your-data-model.cs delete mode 100644 docs/ai/vector-stores/snippets/dynamic-data-model.cs delete mode 100644 docs/ai/vector-stores/snippets/embedding-generation.cs rename docs/ai/vector-stores/{how-to/snippets/vector-store-data-ingestion.cs => snippets/how-to/DataIngestion.cs} (66%) rename docs/ai/vector-stores/{how-to/snippets/VectorStoresExamples => snippets/how-to}/Hotel.cs (95%) rename docs/ai/vector-stores/{how-to/snippets => snippets/how-to}/HowToSnippets.csproj (71%) rename docs/ai/vector-stores/{how-to/snippets/VectorStoresExamples => snippets/how-to}/Program.cs (95%) create mode 100644 docs/ai/vector-stores/snippets/how-to/PutItAllTogether.cs delete mode 100644 docs/ai/vector-stores/snippets/hybrid-search.cs delete mode 100644 docs/ai/vector-stores/snippets/schema-with-record-definition.cs delete mode 100644 docs/ai/vector-stores/snippets/vector-search.cs diff --git a/docs/ai/vector-stores/defining-your-data-model.md b/docs/ai/vector-stores/defining-your-data-model.md index 5fb16481c84b7..c22417fc49c9c 100644 --- a/docs/ai/vector-stores/defining-your-data-model.md +++ b/docs/ai/vector-stores/defining-your-data-model.md @@ -2,23 +2,20 @@ title: Define your Vector Store data model description: Describes how to create a data model with Microsoft.Extensions.VectorData to use when writing to or reading from a Vector Store. ms.topic: reference -ms.date: 07/08/2024 +ms.date: 02/28/2026 --- # Define your data model -## Overview - The Vector Store connectors use a model-first approach to interacting with databases. -All methods to upsert or get records use strongly typed model classes. -The properties on these classes are decorated with attributes that indicate the purpose of each property. +All methods to upsert or get records use strongly typed model classes. The properties on these classes are decorated with attributes that indicate the purpose of each property. > [!TIP] -> For an alternative to using attributes, see [defining your schema with a record definition](./schema-with-record-definition.md). -> [!TIP] -> For an alternative to defining your own data model, see [using Vector Store abstractions without defining your own data model](./generic-data-model.md). +> +> - For an alternative to using attributes, see [defining your schema with a record definition](./schema-with-record-definition.md). +> - For an alternative to defining your own data model, see [using Vector Store abstractions without defining your own data model](./generic-data-model.md). -Here is an example of a model that is decorated with these attributes. +Here's an example of a model that's decorated with these attributes. :::code language="csharp" source="./snippets/defining-your-data-model.cs" id="Overview"::: @@ -50,29 +47,26 @@ Use the attribut | | No | Indicates whether the property should be indexed for full text search for databases that support full text search. The default is `false`. | | | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` are supported. | -> [!TIP] -> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). - ### VectorStoreVectorAttribute Use the attribute to indicate that your property contains a vector. :::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreVectorAttribute1"::: -It's also possible to use on properties that dont' have a vector type, for example, a property of type `string`. -When a property is decorated in this way, you need to provide an instance to the vector store. -When upserting the record, the text that is in the `string` property is automatically converted and stored as a vector in the database. -It's not possible to retrieve a vector using this mechanism. +It's also possible to use on properties that don't have a vector type, for example, a property of type `string`. When a property is decorated in this way, you need to provide an instance to the vector store. When upserting the record, the text that's in the `string` property is automatically converted and stored as a vector in the database. (It's not possible to retrieve a vector using this mechanism.) -:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreVectorAttribute2"::: +```csharp +[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] +public string DescriptionEmbedding { get; set; } +``` > [!TIP] > For more information on how to use built-in embedding generation, see [Let the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). #### VectorStoreVectorAttribute parameters -| Parameter | Required | Description | -|------------|:--------:|-------------| +| Parameter | Required | Description | +|--------------|:--------:|-------------| | `Dimensions` | Yes | The number of dimensions that the vector has. This is required when creating a vector index for a collection. | | | No | The type of index to index the vector with. Default varies by vector store type. | | | No | The type of function to use when doing vector comparison during vector search over this vector. Default varies by vector store type. | @@ -80,6 +74,3 @@ It's not possible to retrieve a vector using this mechanism. Common index kinds and distance function types are supplied as static values on the and classes. Individual Vector Store implementations might also use their own index kinds and distance functions, where the database supports unusual types. - -> [!TIP] -> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). \ No newline at end of file diff --git a/docs/ai/vector-stores/dynamic-data-model.md b/docs/ai/vector-stores/dynamic-data-model.md index 8594e38093373..7c7b5e68754bc 100644 --- a/docs/ai/vector-stores/dynamic-data-model.md +++ b/docs/ai/vector-stores/dynamic-data-model.md @@ -2,40 +2,25 @@ title: Using Vector Store abstractions without defining your own data model description: Describes how to use Vector Store abstractions without defining your own data model. ms.topic: reference -ms.date: 10/16/2024 +ms.date: 02/28/2026 --- # Use Vector Store abstractions without defining your own data model -The Vector Store connectors use a model-first approach to interact with databases. This makes using the connectors easy and simple, since -your data model reflects the schema of your database records and to add any additional schema information required, you can simply add attributes to your data model properties. +The Vector Store connectors use a model-first approach to interact with databases. This makes using the connectors easy and simple, since your data model reflects the schema of your database records. To add any additional schema information, you can simply add attributes to your data model properties. -However, there are cases where it isn't desirable or possible to define your own data model. For example, imagine that you don't know at compile time what your -database schema looks like, and the schema is only provided via configuration. Creating a data model that reflects the schema would be impossible in this case. - -In this case, you can use a `Dictionary` for the record type. Properties are added to the `Dictionary` with key as the property name and the value as the property value. +However, there are cases where it isn't desirable or possible to define your own data model. For example, imagine that you don't know at compile time what your database schema looks like, and the schema is only provided via configuration. Creating a data model that reflects the schema would be impossible in this case. Instead, you can use a `Dictionary` for the record type. Properties are added to the `Dictionary` with key as the property name and the value as the property value. ## Supply schema information when using `Dictionary` -When using a `Dictionary`, connectors still need to know what the database schema looks like. Without the schema information -the connector would not be able to create a collection, or know how to map to and from the storage representation that each database uses. +When using a `Dictionary`, connectors still need to know what the database schema looks like. Without the schema information, the connector would not be able to create a collection or know how to map to and from the storage representation that each database uses. -A record definition can be used to provide the schema information. Unlike a data model, a record definition can be created from configuration at runtime, providing a solution for when schema information is not known at compile time. +A record definition can be used to provide the schema information. Unlike a data model, a record definition can be created from configuration at *runtime* when schema information isn't known at compile time. > [!TIP] -> To see how to create a record definition, see [defining your schema with a record definition](./schema-with-record-definition.md). +> For information about creating a record definition, see [Defining your schema with a record definition](./schema-with-record-definition.md). ## Example -To use the `Dictionary` with a connector, simply specify it as your data model when creating a collection, and simultaneously provide a record definition. +To use the `Dictionary` with a connector, specify it as your data model when you create a collection. Also provide a record definition. :::code language="csharp" source="./snippets/dynamic-data-model.cs" id="Example1"::: - -When constructing a collection instance directly, the record definition -is passed as an option. For example, here is an example of constructing -an Azure AI Search collection instance with `Dictionary`. - -Each vector store collection implementation has a separate `*DynamicCollection` -class that can be used with `Dictionary`. -This is because these implementations might support NativeAOT/trimming. - -:::code language="csharp" source="./snippets/dynamic-data-model.cs" id="Example2"::: \ No newline at end of file diff --git a/docs/ai/vector-stores/embedding-generation.md b/docs/ai/vector-stores/embedding-generation.md index 3cc3236657837..ea3704a73136a 100644 --- a/docs/ai/vector-stores/embedding-generation.md +++ b/docs/ai/vector-stores/embedding-generation.md @@ -1,14 +1,13 @@ --- -title: Generating embeddings for Vector Store connectors +title: Generate embeddings for Vector Store connectors description: Describes how you can generate embeddings to use with Vector Store connectors. ms.topic: concept-article -ms.date: 09/23/2024 +ms.date: 02/28/2026 --- -# Generating embeddings for Vector Store connectors +# Generate embeddings for Vector Store connectors -Vector Store connectors support multiple ways of generating embeddings. -Embeddings can be generated by the developer and passed as part of a record when using a or can be generated internally to the . +Vector Store connectors support multiple ways of generating embeddings. Embeddings can be generated by the developer and passed as part of a record when using a . Or they can be generated internally to the . ## Let the Vector Store generate embeddings @@ -21,32 +20,33 @@ To enable generating vectors automatically on upsert, the vector property on you Before upsert, the `Embedding` property should contain the string from which a vector should be generated. The type of the vector stored in the database (for example, float32, float16, etc.) will be derived from the configured embedding generator. > [!IMPORTANT] -> These vector properties do not support retrieving either the generated vector or the original text that the vector was generated from. They also do not store the original text. -> If the original text needs to be stored, a separate Data property should be added to store it. +> These vector properties do not support retrieving either the generated vector or the original text that the vector was generated from. They also do not store the original text. If the original text needs to be stored, add a separate Data property to store it. -Embedding generators implementing the `Microsoft.Extensions.AI` abstractions are supported and can be configured at various levels: +Embedding generators that implement the `Microsoft.Extensions.AI` abstractions are supported and can be configured at various levels: -1. **On the Vector Store**: +- **On the Vector Store**: - You can set a default embedding generator for the entire vector store. This generator will be used for all collections and properties unless overridden. + You can set a default embedding generator for the entire vector store. This generator will be used for all collections and properties unless overridden. - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnTheVectorStore"::: + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnTheVectorStore"::: -2. **On a Collection**: +- **On a Collection**: - You can configure an embedding generator for a specific collection, overriding the store-level generator. + You can configure an embedding generator for a specific collection, overriding the store-level generator. - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnACollection"::: + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnACollection"::: -3. **On a Record Definition**: - When defining properties programmatically using , you can specify an embedding generator for all properties. +- **On a Record Definition**: - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnARecordDefinition"::: + When defining properties programmatically using , you can specify an embedding generator for all properties. -4. **On a Vector Property Definition**: - When defining properties programmatically, you can set an embedding generator directly on the property. + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnARecordDefinition"::: - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnAVectorPropertyDefinition"::: +- **On a Vector Property Definition**: + + When defining properties programmatically, you can set an embedding generator directly on the property. + + :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnAVectorPropertyDefinition"::: ### Example usage @@ -58,7 +58,7 @@ The following example demonstrates how to use the embedding generator to automat ### Construct an embedding generator -For information on how to construct `Microsoft.Extensions.AI` embedding generators, see [Microsoft.Extensions.AI.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.AI.Abstractions). +For information on how to construct `Microsoft.Extensions.AI` embedding generators, see [Embeddings in .NET](../conceptual/embeddings.md). ### Generate embeddings on upsert with `IEmbeddingGenerator` @@ -70,24 +70,23 @@ For information on how to construct `Microsoft.Extensions.AI` embedding generato ## Embedding dimensions -Vector databases typically require you to specify the number of dimensions that each vector has when creating the collection. -Different embedding models typically support generating vectors with various dimension sizes. For example, OpenAI `text-embedding-ada-002` -generates vectors with 1536 dimensions. Some models also allow a developer to choose the number of dimensions they want in the -output vector. For example, Google `text-embedding-004` produces vectors with 768 dimensions by default, but allows a developer to -choose any number of dimensions between 1 and 768. +Vector databases typically require you to specify the number of dimensions that each vector has when creating the collection. Different embedding models typically support generating vectors with various dimension sizes. For example, OpenAI `text-embedding-ada-002` generates vectors with 1536 dimensions. Some models also allow you to choose the number of dimensions you want in the output vector. For example, Google `text-embedding-004` produces vectors with 768 dimensions by default, but allows you to choose any number of dimensions between 1 and 768. -It's important to ensure that the vectors generated by the embedding model have the same number of dimensions as the -matching vector in the database. +It's important to ensure that the vectors generated by the embedding model have the same number of dimensions as the matching vector in the database. -If creating a collection using the Vector Store abstractions, you need to specify the number of dimensions -required for each vector property either via annotations or via the record definition. Here are examples of both setting -the number of dimensions to 1536. +If you create a collection using the Vector Store abstractions, you need to specify the number of dimensions required for each vector property either via *annotations* or via the *record definition*. The following code shows examples of setting the number of dimensions to 1536 using both mechanisms. -:::code language="csharp" source="./snippets/embedding-generation.cs" id="EmbeddingDimensions1"::: +```csharp +[VectorStoreVector(Dimensions: 1536)] +public ReadOnlyMemory? DescriptionEmbedding { get; set; } -:::code language="csharp" source="./snippets/embedding-generation.cs" id="EmbeddingDimensions2"::: +new VectorStoreVectorProperty( + "DescriptionEmbedding", + typeof(float), + dimensions: 1536); +``` ## See also - [Defining your data model](./defining-your-data-model.md) -- [Defining your schema with a record definition](./schema-with-record-definition.md) \ No newline at end of file +- [Defining your schema with a record definition](./schema-with-record-definition.md) diff --git a/docs/ai/vector-stores/how-to/build-vector-search-app.md b/docs/ai/vector-stores/how-to/build-vector-search-app.md index 39362176f2aaf..eeb990736de75 100644 --- a/docs/ai/vector-stores/how-to/build-vector-search-app.md +++ b/docs/ai/vector-stores/how-to/build-vector-search-app.md @@ -1,7 +1,7 @@ --- title: Quickstart - Build a minimal .NET AI RAG app description: Create an AI powered app to search and integrate with vector stores using embeddings and the Microsoft.Extensions.VectorData package for .NET -ms.date: 05/29/2025 +ms.date: 02/28/2026 ms.topic: quickstart zone_pivot_groups: openai-library --- @@ -138,19 +138,19 @@ Complete the following steps to create a .NET console app that can: 1. Add a new class named `CloudService` to your project with the following properties: - :::code language="csharp" source="snippets/chat-with-data/azure-openai/CloudService.cs" ::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/CloudService.cs" ::: The attributes, such as , influence how each property is handled when used in a vector store. The `Vector` property stores a generated embedding that represents the semantic meaning of the `Description` value for vector searches. 1. In the `Program.cs` file, add the following code to create a data set that describes a collection of cloud services: - :::code language="csharp" source="snippets/chat-with-data/azure-openai/program.cs" id="DataSet"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="DataSet"::: 1. Create and configure an `IEmbeddingGenerator` implementation to send requests to an embedding AI model: :::zone target="docs" pivot="azure-openai" - :::code language="csharp" source="snippets/chat-with-data/azure-openai/program.cs" id="EmbeddingGenerator"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="EmbeddingGenerator"::: > [!NOTE] > searches for authentication credentials from your local tooling. You'll need to assign the `Azure AI Developer` role to the account you used to sign in to Visual Studio or the Azure CLI. For more information, see [Authenticate to Azure AI services with .NET](../azure-ai-services-authentication.md). @@ -159,19 +159,19 @@ Complete the following steps to create a .NET console app that can: :::zone target="docs" pivot="openai" - :::code language="csharp" source="snippets/chat-with-data/openai/program.cs" id="EmbeddingGenerator"::: + :::code language="csharp" source="../snippets/chat-with-data/openai/program.cs" id="EmbeddingGenerator"::: :::zone-end 1. Create and populate a vector store with the cloud service data. Use the `IEmbeddingGenerator` implementation to create and assign an embedding vector for each record in the cloud service data: - :::code language="csharp" source="snippets/chat-with-data/azure-openai/program.cs" id="VectorStore"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="VectorStore"::: The embeddings are numerical representations of the semantic meaning for each data record, which makes them compatible with vector search features. 1. Create an embedding for a search query and use it to perform a vector search on the vector store: - :::code language="csharp" source="snippets/chat-with-data/azure-openai/program.cs" id="Search"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="Search"::: 1. Use the `dotnet run` command to run the app: diff --git a/docs/ai/vector-stores/how-to/build-your-own-connector.md b/docs/ai/vector-stores/how-to/build-your-own-connector.md index e21633733b9c5..7246e67b494f9 100644 --- a/docs/ai/vector-stores/how-to/build-your-own-connector.md +++ b/docs/ai/vector-stores/how-to/build-your-own-connector.md @@ -1,10 +1,10 @@ --- -title: How to build your own Vector Store connector (Preview) +title: How to build your own Vector Store connector description: Describes how to build your own Vector Store connector connector ms.topic: tutorial -ms.date: 07/08/2024 +ms.date: 02/28/2026 --- -# How to build your own Vector Store connector (Preview) +# How to build your own Vector Store connector This article provides guidance for building a Vector Store connector. This article can be used by database providers that want to build and maintain their own implementation, or for anyone that wants to build and maintain an unofficial connector for a database that lacks support. @@ -18,7 +18,7 @@ To contribute your connector to the Semantic Kernel code base: Vector Store connectors are implementations of the [Vector Store abstraction](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions). Some of the decisions that were made when designing the Vector Store abstraction mean that a Vector Store connector requires certain features to provide users with a good experience. A key design decision is that the Vector Store abstraction takes a strongly typed approach to working with database records. -This means that takes a strongly typed record as input, while returns a strongly typed record. +This means that takes a strongly typed record as input, while returns a strongly typed record. The design uses C# generics to achieve the strong typing. This means that a connector has to be able to map from this data model to the storage model used by the underlying database. It also means that a connector might need to find out certain information about the record properties in order to know how to map each of these properties. For example, some vector databases (such as Chroma, Qdrant and Weaviate) require vectors to be stored in a specific structure and non-vectors in a different structure, or require record keys to be stored in a specific field. At the same time, the Vector Store abstraction also provides a generic data model that allows a developer to work with a database without needing to create a custom data model. @@ -37,7 +37,7 @@ To be considered a full implementation of the Vector Store abstractions, the fol - - - implements `IVectorSearchable`, so only two inheriting classes are required. Use the following naming convention: + implements `IVectorSearchable`, so only two inheriting classes are required. Use the following naming convention: - {database type}VectorStore : VectorStore - {database type}Collection : VectorStoreCollection\ @@ -47,7 +47,7 @@ For example: - MyDbVectorStore : VectorStore - MyDbCollection : VectorStoreCollection\ -The implementation should accept the name of the collection as a constructor parameter and each instance of it is therefore tied to a specific collection instance in the database. +The implementation should accept the name of the collection as a constructor parameter and each instance of it is therefore tied to a specific collection instance in the database. Here follows specific requirements for individual methods on these abstract base classes and interfaces. @@ -113,10 +113,10 @@ As mentioned in [Support data model attributes](#2-support-data-model-attributes be supplied via a and if supplied, the connector should avoid trying to read this information from the data model or try and validate that the data model matches the definition in any way. -The user should be able to provide a to the implementation via options. +The user should be able to provide a to the implementation via options. > [!TIP] -> Refer to [Defining your storage schema using a record definition](../schema-with-record-definition.md) for a detailed list of > all record definition settings that need to be supported. +> For a detailed list of all record definition settings that need to be supported, see [Defining your storage schema using a record definition](../schema-with-record-definition.md). ### 4. Collection and index creation @@ -130,7 +130,9 @@ additional custom strings might be accepted. For example, the goal is for a user to be able to specify a standard distance function, like `DotProductSimilarity` for any connector that supports this distance function, without needing to use different naming for each connector. -:::code language="csharp" source="./snippets/build-your-own-connector.cs" id="4CollectionAndIndexCreation"::: +```csharp +[VectorStoreVector(1536, DistanceFunction = DistanceFunction.DotProductSimilarity] +``` 4.2 A user can optionally choose whether each data property should be indexed or full text indexed. In some databases, all properties might already be filterable or full text searchable by default, however @@ -142,7 +144,7 @@ this additional indexing per property. ### 5. Data model validation Every database doesn't support every data type. To improve the user experience it's important to validate -the data types of any record properties and to do so early, for example, when an +the data types of any record properties and to do so early, for example, when an instance is constructed. This way the user will be notified of any potential failures before starting to use the database. ### 6. Storage property naming @@ -171,7 +173,7 @@ mappers. 7.1 All connectors should come with a built in mapper that can map between the user supplied data model and the storage model required by the underlying database. -7.2. All connectors should have a built in mapper that works with the . +7.2. All connectors should have a built in mapper that works with the . See [Support GenericDataModel](#8-support-genericdatamodel) for more information. ### 8. Support GenericDataModel @@ -219,22 +221,32 @@ the original exception as an inner exception. ### 11. Batching -The abstract base class includes batching overloads for Get, Upsert and Delete. -Not all underlying database clients might have the same level of support for batching. +The abstract base class includes batching overloads for Get, Upsert, and Delete. Not all underlying database clients might have the same level of support for batching. -The base batch method implementations on calls the abstract non-batch implementations in serial. -If the database supports batching natively, these base batch implementations should be overridden and implemented -using the native database support. +The base batch method implementations on calls the abstract non-batch implementations in serial. If the database supports batching natively, these base batch implementations should be overridden and implemented using the native database support. ## Recommended common patterns and practices -- Keep and implementations sealed. It's recommended to use a decorator pattern to override a default vector store behavior. +- Keep and implementations sealed. It's recommended to use a decorator pattern to override a default vector store behavior. - Always use options classes for optional settings with smart defaults. - Keep required parameters on the main signature and move optional parameters to options. -Here is an example of an constructor following this pattern. +Here is an example of an constructor following this pattern. -:::code language="csharp" source="./snippets/build-your-own-connector.cs" id="RecommendedCommonPatternsAndPractices"::: +```csharp +public sealed class MyDBCollection : VectorStoreCollection +{ + public MyDBCollection(MyDBClient myDBClient, string collectionName, MyDBCollectionOptions? options = default) + { + } + + //... +} + +public class MyDBCollectionOptions : VectorStoreCollectionOptions +{ +} +``` ## SDK changes @@ -246,15 +258,14 @@ See the following articles for a history of changes to the SDK and therefore imp ## Documentation -To share the features and limitations of your implementation, you can contribute an article to this -Microsoft Learn website. For the documentation on existing connectors, see [Out-of-the-box Vector Store connectors](../out-of-the-box-connectors/index.md). +To share the features and limitations of your implementation, you can contribute an article to this Microsoft Learn website. To create your article, create a pull request in the [.NET docs GitHub repository](https://github.com/dotnet/docs). Areas to cover: -1. An `Overview` with a standard table describing the main features of the connector. -1. An optional `Limitations` section with any limitations for your connector. -1. A `Getting started` section that describes how to import your NuGet and construct your and . -1. A `Data mapping` section showing the connector's default data mapping mechanism to the database storage model, including any property renaming it might support. -1. Information about additional features your connector supports. \ No newline at end of file +- An `Overview` with a standard table describing the main features of the connector. +- An optional `Limitations` section with any limitations for your connector. +- A `Getting started` section that describes how to import your NuGet and construct your and . +- A `Data mapping` section showing the connector's default data mapping mechanism to the database storage model, including any property renaming it might support. +- Information about additional features your connector supports. diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs deleted file mode 100644 index 5b30ff9e212a3..0000000000000 --- a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AdditionalExamples.cs +++ /dev/null @@ -1,70 +0,0 @@ -// -// A data model with two vector properties. -public class Hotel -{ - [VectorStoreKey] - public int HotelId { get; set; } - - [VectorStoreData] - public string? HotelName { get; set; } - - // Two separate embeddings for different aspects of the hotel. - [VectorStoreVector(1536)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory? AmenitiesEmbedding { get; set; } -} - -// Target the amenities embedding specifically. -var options = new VectorSearchOptions -{ - VectorProperty = r => r.AmenitiesEmbedding -}; - -IAsyncEnumerable> results = - collection.SearchAsync(queryEmbedding, top: 3, options); -// - -// -public class FinanceInfo -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData] - public string Text { get; set; } = ""; - - // Use a string type to trigger automatic embedding generation on upsert. - [VectorStoreVector(1536)] - public string EmbeddingSource { get; set; } = ""; -} -// - -// -// Check whether the collection supports hybrid search. -if (collection is IKeywordHybridSearchable hybridCollection) -{ - ReadOnlyMemory queryEmbedding = - await embeddingGenerator.GenerateVectorAsync("peaceful beachfront hotel"); - - // Provide both a vector and keywords for hybrid search. - var results = hybridCollection.HybridSearchAsync( - queryEmbedding, - keywords: ["ocean", "beach"], - top: 3); - - await foreach (var result in results) - { - Console.WriteLine($"Hotel: {result.Record.HotelName}, Score: {result.Score}"); - } -} -// - -// -// Development: in-memory -var vectorStore = new InMemoryVectorStore(); - -// Production: Azure AI Search (swap in at startup) -// var vectorStore = new AzureAISearchVectorStore(new SearchIndexClient(...)); -// diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AutoEmbedding.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AutoEmbedding.cs deleted file mode 100644 index 7a9813b7766b0..0000000000000 --- a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/AutoEmbedding.cs +++ /dev/null @@ -1,56 +0,0 @@ -using Microsoft.Extensions.AI; -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.InMemory; - -// -// When the vector property type is string (not ReadOnlyMemory), -// the vector store automatically generates embeddings using the configured IEmbeddingGenerator. -public class FinanceInfo -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData] - public string Text { get; set; } = ""; - - // The string value placed here before upsert is automatically converted to a vector. - [VectorStoreVector(1536)] - public string EmbeddingSource { get; set; } = ""; -} -// - -public static class AutoEmbeddingExample -{ - public static async Task RunAsync(IEmbeddingGenerator> embeddingGenerator) - { - // - // Configure the embedding generator at the vector store level. - // All collections in this store will use it unless overridden. - var vectorStore = new InMemoryVectorStore(new InMemoryVectorStoreOptions - { - EmbeddingGenerator = embeddingGenerator - }); - - var collection = vectorStore.GetCollection("finance"); - await collection.EnsureCollectionExistsAsync(); - - // Embeddings are generated automatically on upsert. - var records = new[] - { - new FinanceInfo { Key = 1, Text = "2024 Budget", EmbeddingSource = "The budget for 2024 is $100,000" }, - new FinanceInfo { Key = 2, Text = "2023 Budget", EmbeddingSource = "The budget for 2023 is $80,000" } - }; - - await collection.UpsertAsync(records[0]); - await collection.UpsertAsync(records[1]); - - // Embeddings for search are also generated automatically. - var results = collection.SearchAsync("What is my 2024 budget?", top: 1); - - await foreach (var result in results) - { - Console.WriteLine($"Found: Key={result.Record.Key}, Text={result.Record.Text}"); - } - // - } -} diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/RecordDefinition.cs b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/RecordDefinition.cs deleted file mode 100644 index 64da33ae4b470..0000000000000 --- a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/RecordDefinition.cs +++ /dev/null @@ -1,35 +0,0 @@ -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.InMemory; - -public static class RecordDefinitionExample -{ - public static async Task RunAsync() - { - // - // An alternative to using attributes is to define the schema programmatically. - // This is useful when you want to use the same data model with different configurations, - // or when you can't modify the data model class to add attributes. - var hotelDefinition = new VectorStoreCollectionDefinition - { - Properties = new List - { - new VectorStoreKeyProperty("HotelId", typeof(int)), - new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, - new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, - new VectorStoreVectorProperty("DescriptionEmbedding", typeof(ReadOnlyMemory?), dimensions: 1536) - { - DistanceFunction = DistanceFunction.CosineSimilarity, - IndexKind = IndexKind.Flat - } - } - }; - - // Pass the definition to GetCollection to override the attribute-based schema. - var vectorStore = new InMemoryVectorStore(); - VectorStoreCollection collection = - vectorStore.GetCollection("hotels", hotelDefinition); - - await collection.EnsureCollectionExistsAsync(); - // - } -} diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/VectorStoresExamples.csproj b/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/VectorStoresExamples.csproj deleted file mode 100644 index 418c89330cd13..0000000000000 --- a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/VectorStoresExamples.csproj +++ /dev/null @@ -1,15 +0,0 @@ - - - - Exe - net10.0 - enable - enable - - - - - - - - diff --git a/docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs b/docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs deleted file mode 100644 index 12e70cb414260..0000000000000 --- a/docs/ai/vector-stores/how-to/snippets/build-your-own-connector.cs +++ /dev/null @@ -1,19 +0,0 @@ -// <4CollectionAndIndexCreation> -[VectorStoreVector(1536, DistanceFunction = DistanceFunction.DotProductSimilarity] -public ReadOnlyMemory? Embedding { get; set; } -// - -// -public sealed class MyDBCollection : VectorStoreCollection -{ - public MyDBCollection(MyDBClient myDBClient, string collectionName, MyDBCollectionOptions? options = default) - { - } - - ... -} - -public class MyDBCollectionOptions : VectorStoreCollectionOptions -{ -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/how-to/use-vector-stores.md b/docs/ai/vector-stores/how-to/use-vector-stores.md index 8fc7d6f1262fb..12d13ec718aa2 100644 --- a/docs/ai/vector-stores/how-to/use-vector-stores.md +++ b/docs/ai/vector-stores/how-to/use-vector-stores.md @@ -2,7 +2,7 @@ title: Use vector stores in .NET AI apps description: Learn how to use Microsoft.Extensions.VectorData to store, search, and manage embeddings in vector databases for .NET AI applications. ms.topic: how-to -ms.date: 02/24/2026 +ms.date: 02/28/2026 ai-usage: ai-generated --- @@ -36,7 +36,7 @@ Define a .NET class to represent the records you want to store in the vector sto - : A data property that gets stored. Set `IsIndexed = true` to enable filtering on the property, or `IsFullTextIndexed = true` to enable full-text search on it. - : An embedding vector property. Set the `Dimensions` parameter to match your embedding model's output size. -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Hotel.cs" id="DataModel"::: +:::code language="csharp" source="../snippets/how-to/Hotel.cs" id="DataModel"::: The `Dimensions` parameter in `[VectorStoreVector]` must match the output size of the embedding model you use. Common values include 1536 for `text-embedding-3-small` and 3072 for `text-embedding-3-large`. @@ -71,19 +71,19 @@ The following tables describe all available parameters for each attribute. As an alternative to using attributes, you can define your schema programmatically using a . This approach is useful when you want to use the same data model with different configurations, or when you can't add attributes to the data model class. -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/RecordDefinition.cs" id="RecordDefinition"::: +For more information, see [Define your storage schema using a record definition](../schema-with-record-definition.md). ## Create a vector store Create an instance of the implementation for your chosen database. The following example creates an in-memory vector store: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="CreateVectorStore"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="CreateVectorStore"::: ## Get a collection Call on the to get a typed reference. Then call to create the collection if it doesn't already exist: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetCollection"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="GetCollection"::: The collection name maps to the underlying storage concept for your database (for example, a table in SQL Server, an index in Azure AI Search, or a container in Cosmos DB). @@ -91,7 +91,7 @@ The collection name maps to the underlying storage concept for your database (fo Use to insert or update records in the collection. If a record with the same key already exists, it gets updated: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="UpsertRecords"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="UpsertRecords"::: > [!IMPORTANT] > In a real app, generate the embedding vectors using an `IEmbeddingGenerator` before storing the records. For a working example with real embeddings, see [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md). @@ -100,17 +100,17 @@ Use Use to retrieve a single record by its key. To retrieve multiple records, pass an `IEnumerable` to : -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetRecord"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="GetRecord"::: To retrieve multiple records at once: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="GetBatch"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="GetBatch"::: ## Perform vector search Use to find records that are semantically similar to a query. Pass the embedding vector for your query and the number of results to return: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchBasic"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="SearchBasic"::: Each contains the matching record and a similarity score. Higher scores indicate a closer semantic match. @@ -118,7 +118,7 @@ Each contains the ma Use to filter search results before the vector comparison. You can filter on any property marked with `IsIndexed = true`: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchWithFilter"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="SearchWithFilter"::: Filters are expressed as LINQ expressions. The supported operations vary by connector, but all connectors support common comparisons like equality, inequality, and logical `&&` and `||`. @@ -126,37 +126,24 @@ Filters are expressed as LINQ expressions. The supported operations vary by conn Use to control various aspects of vector search behavior: -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="SearchOptions"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="SearchOptions"::: The following table describes the available options: -| Option | Description | -|---|---| -| `Filter` | A LINQ expression to filter records before vector comparison. | +| Option | Description | +|------------------|------------------------------------------------------------------------------------------------| +| `Filter` | A LINQ expression to filter records before vector comparison. | | `VectorProperty` | The vector property to search on. Required when the data model has multiple vector properties. | -| `Skip` | Number of results to skip before returning. Useful for paging. Default is `0`. | +| `Skip` | Number of results to skip before returning. Useful for paging. Default is `0`. | | `IncludeVectors` | Whether to include vector data in the returned records. Omitting vectors reduces data transfer. Default is `false`. | -### Target a specific vector property - -When your data model has multiple vector properties, use `VectorProperty` to specify which one to search: - -:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="TargetVectorProperty"::: +For more information, see [Vector search options](../vector-search.md#vector-search-options). ## Use built-in embedding generation Instead of generating embeddings manually before each upsert, you can configure an `IEmbeddingGenerator` on the vector store or collection. When you do, declare your vector property as a `string` type (the source text) and the store generates the embedding automatically. -:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="EmbeddingSourceModel"::: - -Then configure the embedding generator when creating the vector store: - -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/AutoEmbedding.cs" id="AutoEmbeddingVectorStore"::: - -With this approach, also accepts a `string` query directly—you don't need to generate a query embedding manually. - -> [!IMPORTANT] -> Automatic embedding generation doesn't retrieve or store the original source text. If you need to access the original text later, store it in a separate `[VectorStoreData]` property on your model. +For more information, see [Let the Vector Store generate embeddings](../embedding-generation.md#let-the-vector-store-generate-embeddings). ## Hybrid search @@ -164,35 +151,27 @@ Some vector stores support *hybrid search*, which combines vector similarity wit To use hybrid search, check whether your collection implements . Only connectors for databases that support this feature implement this interface. -:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="HybridSearch"::: - -For hybrid search to work, the data model must have at least one vector property and one text property with `IsFullTextIndexed = true`. For a list of connectors that support hybrid search, see the documentation for each connector. +For more information, see [Hybrid search using Vector Store connectors](../hybrid-search.md). ## Delete records To delete a single record by key, use : -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="DeleteRecord"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="DeleteRecord"::: ## Delete a collection To remove an entire collection from the vector store, use : -:::code language="csharp" source="./snippets/use-vector-stores/csharp/VectorStoresExamples/Program.cs" id="DeleteCollection"::: +:::code language="csharp" source="../snippets/how-to/Program.cs" id="DeleteCollection"::: ## Switch vector store connectors -Because all connectors implement the same abstract class, you can switch between them by changing the concrete type at startup. Your collection and search code remains the same. - -For example, to switch from in-memory to Azure AI Search: - -:::code language="csharp" source="./snippets/VectorStoresExamples/AdditionalExamples.cs" id="SwitchConnectors"::: - -This approach lets you develop and test locally without any external services, then deploy to a production database with minimal changes. +Because all connectors implement the same abstract class, you can switch between them by changing the concrete type at startup. Your collection and search code remains the same. This approach lets you develop and test locally without any external services, then deploy to a production database with minimal changes. ## Related content - [Vector databases for .NET AI apps](../overview.md) - [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md) - [Data ingestion](../conceptual/data-ingestion.md) -- [Embeddings in .NET](../conceptual/embeddings.md) \ No newline at end of file +- [Embeddings in .NET](../conceptual/embeddings.md) diff --git a/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md b/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md index 5520ae1db4b83..0deb26c3f2b7a 100644 --- a/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md +++ b/docs/ai/vector-stores/how-to/vector-store-data-ingestion.md @@ -2,7 +2,7 @@ title: How to ingest data into a Vector Store description: Step by step instructions on how to ingest data into a Vector Store. ms.topic: tutorial -ms.date: 07/08/2024 +ms.date: 02/28/2026 --- # How to ingest data into a Vector Store @@ -53,7 +53,7 @@ You can do this by creating a data model with attributes that describe the funct Add a new file to the project called `TextParagraph.cs` and add the following model to it. -:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="AddADataModel"::: +:::code language="csharp" source="./snippets/DataIngestion.cs" id="AddADataModel"::: The value `1536`, which is the dimension size of the vector, is passed to the . This value must match the size of vector that your chosen embedding generator produces. @@ -66,7 +66,7 @@ Next, you add the code to read the Word document and find the text of each parag Add a new file to the project called `DocumentReader.cs` and add the following class to read the paragraphs from a document. -:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="ReadTheParagraphsInTheDocument"::: +:::code language="csharp" source="./snippets/DataIngestion.cs" id="ReadTheParagraphsInTheDocument"::: ## Generate embeddings and upload the data @@ -74,26 +74,23 @@ Next, you add a new class to generate embeddings and upload the paragraphs to Re Add a new file called `DataUploader.cs` with the following contents: -:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="GenerateEmbeddingsAndUploadTheData"::: +:::code language="csharp" source="./snippets/DataIngestion.cs" id="GenerateEmbeddingsAndUploadTheData"::: ## Put it all together -Finally, you put together the different pieces. -In this example, you use standard .NET dependency injection to register the Redis vector store and the embedding generator. +Finally, you put together the different pieces. In this example, you use standard .NET dependency injection to register the Redis vector store and the embedding generator. Add the following code to your `Program.cs` file to set up the container, register the Redis vector store, and register the embedding service. Replace the text embedding generation settings with your own values. -:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="PutItAllTogether1"::: +:::code language="csharp" source="./snippets/DataIngestion.cs" id="PutItAllTogether1"::: -Lastly, add code to read the paragraphs from the Word document and call the data uploader -to generate the embeddings and upload the paragraphs. +Lastly, add code to read the paragraphs from the Word document and call the data uploader to generate the embeddings and upload the paragraphs. -:::code language="csharp" source="./snippets/vector-store-data-ingestion.cs" id="PutItAllTogether2"::: +:::code language="csharp" source="./snippets/DataIngestion.cs" id="PutItAllTogether2"::: ## See your data in Redis -Navigate to the Redis stack browser, for example, [http://localhost:8001/redis-stack/browser](http://localhost:8001/redis-stack/browser), where you should now be able to see -your uploaded paragraphs. Following is an example of what you should see for one of the uploaded paragraphs. +Navigate to the Redis stack browser, for example, [http://localhost:8001/redis-stack/browser](http://localhost:8001/redis-stack/browser), where you should now be able to see your uploaded paragraphs. Following is an example of what you should see for one of the uploaded paragraphs. ```json { @@ -102,4 +99,4 @@ your uploaded paragraphs. Following is an example of what you should see for one "Text" : "Version 1.0+ support across C#, Python, and Java means it’s reliable, committed to non breaking changes. Any existing chat-based APIs are easily expanded to support additional modalities like voice and video.", "TextEmbedding" : [...] } -``` \ No newline at end of file +``` diff --git a/docs/ai/vector-stores/hybrid-search.md b/docs/ai/vector-stores/hybrid-search.md index 23dbf07ca707c..ec7a96ca085ac 100644 --- a/docs/ai/vector-stores/hybrid-search.md +++ b/docs/ai/vector-stores/hybrid-search.md @@ -1,82 +1,73 @@ --- -title: Hybrid search using Vector Store connectors (Preview) +title: Hybrid search using Vector Store connectors description: Describes the different options you can use when doing a hybrid search using Vector Store connectors. ms.topic: concept-article -ms.date: 03/06/2025 +ms.date: 02/28/2026 --- -# Hybrid search using Vector Store connectors (Preview) +# Hybrid search using Vector Store connectors -The Microsoft.Extensions.VectorData library provides hybrid search capabilities as part of its Vector Store abstractions. This supports filtering and many other options, which this article will explain in more detail. +The library provides hybrid search capabilities, including filtering, as part of its Vector Store abstractions. -Currently the type of hybrid search supported is based on a vector search, plus a keyword search, both of which are executed in parallel, after which a union of the two result sets -are returned. Sparse vector based hybrid search is not currently supported. +The hybrid search is based on a vector search and a keyword search, both of which are executed in parallel. The hybrid search returns a union of the two result sets. (Sparse vector–based hybrid search isn't currently supported.) -To execute a hybrid search, your database schema needs to have a vector field and a string field with full text search capabilities enabled. -If you are creating a collection using the Vector Store connectors, make sure to enable the option -on the string field that you want to target for the keyword search. +To execute a hybrid search, your database schema needs to have a vector field and a string field with full-text search capabilities enabled. If you're creating a collection using the Vector Store connectors, enable the option on the string field that you want to target for the keyword search. > [!TIP] -> For more information on how to enable refer to [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstoredataattribute-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md#vectorstoredataproperty-configuration-settings) +> For more information on how to enable , see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstoredataattribute-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md#vectorstoredataproperty-configuration-settings) ## Hybrid search -The method allows searching using a vector and an `ICollection` of string keywords. It also takes an optional `HybridSearchOptions` class as input. -This method is available on the following interface: +The method searches using a vector and an `ICollection` of string keywords. It also takes an optional `HybridSearchOptions` class as input. -1. `IKeywordHybridSearchable` +Only connectors for databases that support vector plus keyword hybrid search implement [the interface](xref:Microsoft.Extensions.VectorData.IKeywordHybridSearchable`1) that provides this method. -Only connectors for databases that currently support vector plus keyword hybrid search are implementing this interface. - -Assuming you have a collection that already contains data, you can easily do a hybrid search on it. Here is an example using Qdrant. +The following example shows how to perform a hybrid search on a collection in a Qdrant database. :::code language="csharp" source="./snippets/hybrid-search.cs" id="HybridSearch"::: > [!TIP] -> For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). +> For more information on how to generate embeddings, see [embedding generation](./embedding-generation.md). ## Supported vector types - takes a generic type as the vector parameter. + takes a generic type as the vector parameter. The types of vectors supported by each data store vary. -See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. -It's also important for the search vector type to match the target vector that is being searched, for example, if you have two vectors +It's also important for the search vector type to match the target vector that's being searched. For example, if you have two vectors on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector -you are targeting. -See [VectorProperty and AdditionalProperty](#vectorproperty-and-additionalproperty) for how to pick a target vector if you have more than one per record. +you're targeting. For information on how to pick a target vector if you have more than one per record, see [VectorProperty and AdditionalProperty](#vectorproperty-and-additionalproperty). ## Hybrid search options -The following options can be provided using the `HybridSearchOptions` class. +The following options can be provided using the class. ### VectorProperty and AdditionalProperty -The `VectorProperty` and `AdditionalProperty` options can be used to specify the vector property and full text search property to target during the search. +Use the and options to specify the vector property and full-text search property to target during the search. + +When no `VectorProperty` is provided: -If no `VectorProperty` is provided and the data model contains only one vector, that vector will be used. -If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. +- If the data model contains only one vector, that vector is used. +- If the data model contains no vector or multiple vectors, the search method throws an exception. -If no `AdditionalProperty` is provided and the data model contains only one full text search property, that property will be used. -If the data model contains no full text search property or multiple full text search properties and `AdditionalProperty` is not provided, the search method will throw. +When no `AdditionalProperty` is provided: + +- If the data model contains only one full-text search property, that property is used. +- If the data model contains no full-text search property or multiple full-text search properties, the search method throws an exception. :::code language="csharp" source="./snippets/hybrid-search.cs" id="VectorPropertyAndAdditionalProperty"::: ### `Top` and `Skip` -The `Top` and `Skip` options allow you to limit the number of results to the Top n results and -to skip a number of results from the top of the resultset. -Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. +The `Top` and `Skip` options allow you to limit the number of results to the top `n` results and to skip a number of results from the top of the result set. You can use `Top` and `Skip` to do paging if you want to retrieve a large number of results using separate calls. :::code language="csharp" source="./snippets/hybrid-search.cs" id="TopAndSkip"::: -The default values for `Skip` is 0. +The default value for `Skip` is 0. ### IncludeVectors -The `IncludeVectors` option allows you to specify whether you wish to return vectors in the search results. -If `false`, the vector properties on the returned model will be left null. -Using `false` can significantly reduce the amount of data retrieved from the vector store during search, -making searches more efficient. +The `IncludeVectors` option allows you to specify whether you want to return vectors in the search results. If `false`, the vector properties on the returned model are left null. Using `false` can significantly reduce the amount of data retrieved from the vector store during search, making searches more efficient. The default value for `IncludeVectors` is `false`. @@ -84,25 +75,18 @@ The default value for `IncludeVectors` is `false`. ### Filter -The vector search filter option can be used to provide a filter for filtering the records in the chosen collection -before applying the vector search. - -This has multiple benefits: +Use the vector search filter option to provide a filter for filtering the records in the chosen collection before applying the vector search. This has multiple benefits: - Reduce latency and processing cost, since only records remaining after filtering need to be compared with the search vector and therefore fewer vector comparisons have to be done. -- Limit the resultset for for example, access control purposes, by excluding data that the user shouldn't have access to. +- Limit the result set for by excluding data that the user shouldn't have access to. This can be useful for access-control purposes. -Note that in order for fields to be used for filtering, many vector stores require those fields to be indexed first. -Some vector stores will allow filtering using any field, but might optionally allow indexing to improve filtering performance. +For fields to be used for filtering, many vector stores require them to be indexed first. Some vector stores allow filtering using any field, but might optionally allow indexing to improve filtering performance. -If creating a collection via the Vector Store abstractions and you wish to enable filtering on a field, -set the property to true when defining your data model or when creating your record definition. +If you're creating a collection via the Vector Store abstractions and you want to enable filtering on a field, set the property to `true` when defining your data model or when creating your record definition. > [!TIP] > For more information on how to set the property, see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddatafield-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md). -Filters are expressed using LINQ expressions based on the type of the data model. -The set of LINQ expressions supported will vary depending on the functionality supported -by each database, but all databases support a broad base of common expressions, for example, `equals`, `not equals`, `and`, and `or`. +Filters are expressed using LINQ expressions based on the type of the data model. The set of LINQ expressions supported varies depending on the functionality supported by each database, but all databases support a broad base of common expressions, for example, `equals`, `not equals`, `and`, and `or`. -:::code language="csharp" source="./snippets/hybrid-search.cs" id="Filter"::: \ No newline at end of file +:::code language="csharp" source="./snippets/hybrid-search.cs" id="Filter"::: diff --git a/docs/ai/vector-stores/overview.md b/docs/ai/vector-stores/overview.md index f2653e514a757..b5cf8e9e34cc7 100644 --- a/docs/ai/vector-stores/overview.md +++ b/docs/ai/vector-stores/overview.md @@ -2,7 +2,7 @@ title: "Vector databases for .NET AI apps" description: "Learn how vector databases extend LLM capabilities by storing and processing embeddings in .NET, and how to use Microsoft.Extensions.VectorData to build semantic search features." ms.topic: concept-article -ms.date: 02/24/2026 +ms.date: 02/28/2026 ai-usage: ai-assisted --- @@ -22,7 +22,7 @@ For example, you can use a vector database to: Vector databases provide vector search capabilities to find similar items based on their data characteristics rather than by exact matches on a property field. Vector search works by analyzing the vector representations of your data that you created using an AI embedding model such as the [Azure OpenAI embedding models](/azure/ai-services/openai/concepts/models#embeddings-models). The search process measures the distance between the data vectors and your query vector. The data vectors that are closest to your query vector are the ones that are most similar semantically. -Some services such as [Azure Cosmos DB for MongoDB vCore](/azure/cosmos-db/mongodb/vcore/vector-search) provide native vector search capabilities for your data. Other databases can be enhanced with vector search by indexing the stored data using a service such as Azure AI Search, which can scan and index your data to provide vector search capabilities. +Some services, such as [Azure Cosmos DB for MongoDB vCore](/azure/cosmos-db/mongodb/vcore/vector-search), provide native vector search capabilities for your data. Other databases can be enhanced with vector search by indexing the stored data using a service such as Azure AI Search, which can scan and index your data to provide vector search capabilities. ## Vector search workflows with .NET and OpenAI @@ -58,40 +58,19 @@ The library provides the following key capabilities: The `Microsoft.Extensions.VectorData.Abstractions` library exposes the following main abstract classes: - : The top-level class for a vector database. Use it to retrieve and manage collections. -- : Represents a named collection of records within a vector store. Use it to perform CRUD and search operations. Also inherits from `IVectorSearchable`. -- `IKeywordHybridSearchable`: Implemented by collections that support hybrid search combining vector similarity with keyword matching. +- : Represents a named collection of records within a vector store. Use it to perform CRUD and search operations. Also implements `IVectorSearchable`. +- `IKeywordHybridSearchable`: Implemented by collections that support hybrid search, combining vector similarity with keyword matching. For a step-by-step guide covering data model definition, CRUD operations, vector search, filtering, hybrid search, and embedding generation, see [Use vector stores in .NET AI apps](how-to/use-vector-stores.md). ## Available vector store connectors -The `Microsoft.Extensions.VectorData.Abstractions` package defines the abstractions, and separate connector packages implement them for specific vector databases. Choose the connector that matches your vector database: - -| Connector | NuGet package | -|---|---| -| In-memory (for testing/development) | [Microsoft.SemanticKernel.Connectors.InMemory](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.InMemory) | -| Azure AI Search | [Microsoft.SemanticKernel.Connectors.AzureAISearch](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.AzureAISearch) | -| Azure Cosmos DB (NoSQL) | [Microsoft.SemanticKernel.Connectors.CosmosNoSQL](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosNoSQL) | -| Azure Cosmos DB (MongoDB) | [Microsoft.SemanticKernel.Connectors.CosmosMongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosMongoDB) | -| Azure SQL / SQL Server | [Microsoft.SemanticKernel.Connectors.SqlServer](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqlServer) | -| Couchbase | [CouchbaseVectorStore.SemanticKernel](https://www.nuget.org/packages/CouchbaseVectorStore.SemanticKernel) | -| Elasticsearch | [Elastic.SemanticKernel.Connectors.Elasticsearch](https://www.nuget.org/packages/Elastic.SemanticKernel.Connectors.Elasticsearch) | -| MongoDB | [Microsoft.SemanticKernel.Connectors.MongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.MongoDB) | -| Oracle | [OracleVectorStore.SemanticKernel](https://www.nuget.org/packages/OracleVectorStore.SemanticKernel) | -| Pinecone | [Microsoft.SemanticKernel.Connectors.Pinecone](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Pinecone) | -| PostgreSQL (pgvector) | [Microsoft.SemanticKernel.Connectors.PgVector](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.PgVector) | -| Qdrant | [Microsoft.SemanticKernel.Connectors.Qdrant](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Qdrant) | -| Redis | [Microsoft.SemanticKernel.Connectors.Redis](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Redis) | -| SQLite | [Microsoft.SemanticKernel.Connectors.SqliteVec](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqliteVec) | -| Weaviate | [Microsoft.SemanticKernel.Connectors.Weaviate](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Weaviate) | - -All connectors implement the same and abstract classes, so you can switch between them without changing your application logic. +The `Microsoft.Extensions.VectorData.Abstractions` package defines the abstractions, and separate connector packages implement them for specific vector databases. Choose the connector that matches your vector database, for example, [Microsoft.SemanticKernel.Connectors.AzureAISearch](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.AzureAISearch). -> [!TIP] -> Use the in-memory connector (`Microsoft.SemanticKernel.Connectors.InMemory`) during development and testing. It doesn't require any external service or configuration, and you can swap it out for a production connector later. +All connectors implement the same and abstract classes, so you can switch between them without changing your application logic. -> [!IMPORTANT] -> Not all connectors are maintained by the Microsoft Semantic Kernel team. When evaluating a connector, review its quality, licensing, support, and compatibility to ensure it meets your requirements. +> [!TIP] +> Use the in-memory connector ([Microsoft.SemanticKernel.Connectors.InMemory](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.InMemory)) during development and testing. It doesn't require any external service or configuration, and you can swap it out for a production connector later. ## Related content @@ -99,4 +78,4 @@ All connectors implement the same class to indicate that your property is the key of the record. :::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreKeyProperty"::: #### VectorStoreKeyProperty configuration settings | Parameter | Required | Description | -|--|:-:|--| -| Name | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | -| Type | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | -| StorageName | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | - -> [!TIP] -> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). +|-----------|:--------:|-------------| +| `Name` | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| `Type` | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| `StorageName` | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | ### VectorStoreDataProperty -Use this class to indicate that your property contains general data that is not a key or a vector. +Use the class to indicate that your property contains general data that isn't a key or a vector. :::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreDataProperty"::: #### VectorStoreDataProperty configuration settings | Parameter | Required | Description | -|--|:-:|--| -| Name | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | -| Type | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | -| IsIndexed | No | Indicates whether the property should be indexed for filtering in cases where a database requires opting in to indexing per property. Default is false. | -| IsFullTextIndexed | No | Indicates whether the property should be indexed for full text search for databases that support full text search. Default is false. | -| StorageName | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | - -> [!TIP] -> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). +|-----------|:--------:|-------------| +| `Name` | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| `Type` | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| `IsIndexed` | No | Indicates whether the property should be indexed for filtering in cases where a database requires opting in to indexing per property. Default is false. | +| `IsFullTextIndexed` | No | Indicates whether the property should be indexed for full text search for databases that support full text search. Default is false. | +| `StorageName` | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | ### VectorStoreVectorProperty -Use this class to indicate that your property contains a vector. +Use the class to indicate that your property contains a vector. :::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreVectorProperty"::: #### VectorStoreVectorProperty configuration settings | Parameter | Required | Description | -|--|:-:|--| -| Name | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | -| Type | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | -| Dimensions | Yes | The number of dimensions that the vector has. This is required for creating a vector index for a collection. | -| IndexKind | No | The type of index to index the vector with. Default varies by vector store type. | -| DistanceFunction | No | The type of function to use when doing vector comparison during vector search over this vector. Default varies by vector store type. | -| StorageName | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | -| EmbeddingGenerator | No | Allows specifying a `Microsoft.Extensions.AI.IEmbeddingGenerator` instance to use for generating embeddings automatically for the decorated property. | - -> [!TIP] -> For more information on which connectors support and what alternatives are available, see [the documentation for each connector](./out-of-the-box-connectors/index.md). \ No newline at end of file +|-----------|:--------:|-------------| +| `Name` | Yes | The name of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| `Type` | No | The type of the property on the data model. Used by the mapper to automatically map between the storage schema and data model and for creating indexes. | +| `Dimensions` | Yes | The number of dimensions that the vector has. This is required for creating a vector index for a collection. | +| `IndexKind` | No | The type of index to index the vector with. Default varies by vector store type. | +| `DistanceFunction` | No | The type of function to use when doing vector comparison during vector search over this vector. Default varies by vector store type. | +| `StorageName` | No | Can be used to supply an alternative name for the property in the database. This parameter is not supported by all connectors, for example, where alternatives like `JsonPropertyNameAttribute` is supported. | +| `EmbeddingGenerator` | No | Allows specifying a `Microsoft.Extensions.AI.IEmbeddingGenerator` instance to use for generating embeddings automatically for the decorated property. | diff --git a/docs/ai/quickstarts/snippets/chat-with-data/azure-openai/CloudService.cs b/docs/ai/vector-stores/snippets/chat-with-data/azure-openai/CloudService.cs similarity index 100% rename from docs/ai/quickstarts/snippets/chat-with-data/azure-openai/CloudService.cs rename to docs/ai/vector-stores/snippets/chat-with-data/azure-openai/CloudService.cs diff --git a/docs/ai/quickstarts/snippets/chat-with-data/azure-openai/Program.cs b/docs/ai/vector-stores/snippets/chat-with-data/azure-openai/Program.cs similarity index 100% rename from docs/ai/quickstarts/snippets/chat-with-data/azure-openai/Program.cs rename to docs/ai/vector-stores/snippets/chat-with-data/azure-openai/Program.cs diff --git a/docs/ai/quickstarts/snippets/chat-with-data/azure-openai/VectorDataAI.csproj b/docs/ai/vector-stores/snippets/chat-with-data/azure-openai/VectorDataAI.csproj similarity index 100% rename from docs/ai/quickstarts/snippets/chat-with-data/azure-openai/VectorDataAI.csproj rename to docs/ai/vector-stores/snippets/chat-with-data/azure-openai/VectorDataAI.csproj diff --git a/docs/ai/quickstarts/snippets/chat-with-data/openai/CloudService.cs b/docs/ai/vector-stores/snippets/chat-with-data/openai/CloudService.cs similarity index 100% rename from docs/ai/quickstarts/snippets/chat-with-data/openai/CloudService.cs rename to docs/ai/vector-stores/snippets/chat-with-data/openai/CloudService.cs diff --git a/docs/ai/quickstarts/snippets/chat-with-data/openai/Program.cs b/docs/ai/vector-stores/snippets/chat-with-data/openai/Program.cs similarity index 100% rename from docs/ai/quickstarts/snippets/chat-with-data/openai/Program.cs rename to docs/ai/vector-stores/snippets/chat-with-data/openai/Program.cs diff --git a/docs/ai/quickstarts/snippets/chat-with-data/openai/VectorDataAI.csproj b/docs/ai/vector-stores/snippets/chat-with-data/openai/VectorDataAI.csproj similarity index 100% rename from docs/ai/quickstarts/snippets/chat-with-data/openai/VectorDataAI.csproj rename to docs/ai/vector-stores/snippets/chat-with-data/openai/VectorDataAI.csproj diff --git a/docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj b/docs/ai/vector-stores/snippets/conceptual/VectorStoreSnippets.csproj similarity index 82% rename from docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj rename to docs/ai/vector-stores/snippets/conceptual/VectorStoreSnippets.csproj index 760fc2801b1e6..8da7f054d84df 100644 --- a/docs/ai/vector-stores/snippets/VectorStoreSnippets.csproj +++ b/docs/ai/vector-stores/snippets/conceptual/VectorStoreSnippets.csproj @@ -1,4 +1,4 @@ - + net10.0 @@ -8,9 +8,10 @@ - - - + + + + diff --git a/docs/ai/vector-stores/snippets/conceptual/defining-your-data-model.cs b/docs/ai/vector-stores/snippets/conceptual/defining-your-data-model.cs new file mode 100644 index 0000000000000..f1a03abf786ca --- /dev/null +++ b/docs/ai/vector-stores/snippets/conceptual/defining-your-data-model.cs @@ -0,0 +1,27 @@ +using Microsoft.Extensions.VectorData; + +// +public class Hotel +{ + // + [VectorStoreKey] + public ulong HotelId { get; set; } + // + + // + [VectorStoreData(IsIndexed = true)] + public required string HotelName { get; set; } + // + + [VectorStoreData(IsFullTextIndexed = true)] + public required string Description { get; set; } + + // + [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] + public ReadOnlyMemory? DescriptionEmbedding { get; set; } + // + + [VectorStoreData(IsIndexed = true)] + public required string[] Tags { get; set; } +} +// diff --git a/docs/ai/vector-stores/snippets/conceptual/dynamic-data-model.cs b/docs/ai/vector-stores/snippets/conceptual/dynamic-data-model.cs new file mode 100644 index 0000000000000..c4c8f4fc499fd --- /dev/null +++ b/docs/ai/vector-stores/snippets/conceptual/dynamic-data-model.cs @@ -0,0 +1,39 @@ +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +public class DynamicDataModel +{ + public static async Task RunAsync() + { + VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); + + // + VectorStoreCollectionDefinition definition = new() + { + Properties = + [ + new VectorStoreKeyProperty("Key", typeof(string)), + new VectorStoreDataProperty("Term", typeof(string)), + new VectorStoreDataProperty("Definition", typeof(string)), + new VectorStoreVectorProperty("DefinitionEmbedding", typeof(ReadOnlyMemory), dimensions: 1536) + ] + }; + + // Use GetDynamicCollection instead of the regular GetCollection method + // to get an instance of a collection using Dictionary. + VectorStoreCollection> dynamicDataModelCollection = + vectorStore.GetDynamicCollection("glossary", definition); + + // Since schema information is available from the record definition, + // it's possible to create a collection with the right vectors, + // dimensions, indexes, and distance functions. + await dynamicDataModelCollection.EnsureCollectionExistsAsync(); + + // When retrieving a record from the collection, + // access key, data, and vector values via the dictionary entries. + Dictionary? record = await dynamicDataModelCollection.GetAsync("SK"); + Console.WriteLine(record["Definition"]); + // + } +} diff --git a/docs/ai/vector-stores/snippets/conceptual/embedding-generation.cs b/docs/ai/vector-stores/snippets/conceptual/embedding-generation.cs new file mode 100644 index 0000000000000..5cccf1bbf1d1e --- /dev/null +++ b/docs/ai/vector-stores/snippets/conceptual/embedding-generation.cs @@ -0,0 +1,191 @@ +using Microsoft.Extensions.AI; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; +using Microsoft.SemanticKernel.Connectors.Qdrant; +using OpenAI; +using Qdrant.Client; + +public class EmbeddingExamples +{ + // + [VectorStoreVector(1536)] + public required string Embedding { get; set; } + // + + public static void ConfigureEmbeddingGeneration() + { + IEmbeddingGenerator embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + + // + VectorStore vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + ownsClient: true, + new QdrantVectorStoreOptions + { + EmbeddingGenerator = embeddingGenerator + }); + // + + // + var collectionOptions = new QdrantCollectionOptions + { + EmbeddingGenerator = embeddingGenerator + }; + + var collection = new QdrantCollection( + new QdrantClient("localhost"), + "myCollection", + ownsClient: true, + collectionOptions); + // + + // + var definition = new VectorStoreCollectionDefinition + { + EmbeddingGenerator = embeddingGenerator, + Properties = + [ + new VectorStoreKeyProperty("Key", typeof(ulong)), + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) + ] + }; + + collectionOptions = new QdrantCollectionOptions + { + Definition = definition + }; + + collection = new QdrantCollection( + new QdrantClient("localhost"), + "myCollection", + ownsClient: true, + collectionOptions); + // + + // + VectorStoreVectorProperty vectorProperty = new( + "DescriptionEmbedding", + typeof(string), + dimensions: 1536) + { + EmbeddingGenerator = embeddingGenerator + }; + // + } + + private class MyRecord { } +} + +public class ExampleUsage +{ + // + + // The data model. + internal class FinanceInfo + { + [VectorStoreKey] + public string Key { get; set; } = string.Empty; + + [VectorStoreData] + public string Text { get; set; } = string.Empty; + + // Note that the vector property is typed as a string, and + // its value is derived from the Text property. The string + // value will however be converted to a vector on upsert and + // stored in the database as a vector. + [VectorStoreVector(1536)] + public string Embedding => Text; + } + + public static async Task RunAsync() + { + // Create an OpenAI embedding generator. + var embeddingGenerator = new OpenAIClient("your key") + .GetEmbeddingClient("your chosen model") + .AsIEmbeddingGenerator(); + + // Use the embedding generator with the vector store. + VectorStore vectorStore = new InMemoryVectorStore(new() + { EmbeddingGenerator = embeddingGenerator } + ); + InMemoryCollection collection = + (InMemoryCollection)vectorStore.GetCollection("finances"); + await collection.EnsureCollectionExistsAsync(); + + // Create some test data. + string[] budgetInfo = + [ + "The budget for 2020 is EUR 100 000", + "The budget for 2021 is EUR 120 000", + "The budget for 2022 is EUR 150 000", + "The budget for 2023 is EUR 200 000", + "The budget for 2024 is EUR 364 000" + ]; + + // Embeddings are generated automatically on upsert. + IEnumerable records = budgetInfo.Select( + (input, index) => new FinanceInfo { Key = index.ToString(), Text = input } + ); + await collection.UpsertAsync(records); + + // Embeddings for the search is automatically generated on search. + IAsyncEnumerable> searchResult = + collection.SearchAsync("What is my budget for 2024?", top: 1); + + // Output the matching result. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine($"Key: {result.Record.Key}, Text: {result.Record.Text}"); + } + } + // + async Task GenerateEmbeddingsAndUpsertAsync( + IEmbeddingGenerator> embeddingGenerator, + VectorStoreCollection collection) + { + // Upsert a record. + string descriptionText = "A place where everyone can be happy."; + ulong hotelId = 1; + + // Generate the embedding. + ReadOnlyMemory embedding = + (await embeddingGenerator.GenerateAsync(descriptionText)).Vector; + + // Create a record and upsert with the already generated embedding. + await collection.UpsertAsync(new Hotel + { + HotelId = hotelId, + HotelName = "Hotel Happy", + Description = descriptionText, + DescriptionEmbedding = embedding, + Tags = ["luxury", "pool"] + }); + } + // + + // + async Task GenerateEmbeddingsAndSearchAsync( + IEmbeddingGenerator> embeddingGenerator, + VectorStoreCollection collection) + { + // Upsert a record. + string descriptionText = "Find me a hotel with happiness in mind."; + + // Generate the embedding. + ReadOnlyMemory searchEmbedding = + (await embeddingGenerator.GenerateAsync(descriptionText)).Vector; + + // Search using the already generated embedding. + IAsyncEnumerable> searchResult = collection.SearchAsync(searchEmbedding, top: 1); + List> resultItems = await searchResult.ToListAsync(); + + // Print the first search result. + Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); + Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); + } + // +} diff --git a/docs/ai/vector-stores/snippets/conceptual/hybrid-search.cs b/docs/ai/vector-stores/snippets/conceptual/hybrid-search.cs new file mode 100644 index 0000000000000..b8d86c32a5b8f --- /dev/null +++ b/docs/ai/vector-stores/snippets/conceptual/hybrid-search.cs @@ -0,0 +1,188 @@ +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +public class HybridSearchExample +{ + public async static void Run() + { + // + // Create a Qdrant VectorStore object and choose + // an existing collection that already contains records. + VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); + IKeywordHybridSearchable collection = + (IKeywordHybridSearchable)vectorStore.GetCollection("skhotels"); + + // Generate a vector for your search text, using + // your chosen embedding generation implementation. + ReadOnlyMemory searchVector = await GenerateAsync("I'm looking for a hotel where customer happiness is the priority."); + + // Do the search, passing an options object with a Top value to limit results to the single top match. + IAsyncEnumerable> searchResult = + collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 1); + + // Inspect the returned hotel. + await foreach (VectorSearchResult record in searchResult) + { + Console.WriteLine($"Found hotel description: {record.Record.Description}"); + Console.WriteLine($"Found record score: {record.Score}"); + } + // + } + + private static async Task> GenerateAsync(string v) => throw new NotImplementedException(); + + public class VPAndAddlPropertyExample + { + private object searchVector; + + // + public async Task Run() + { + var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); + var collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skproducts"); + + // Create the hybrid search options and indicate that you want + // to search the DescriptionEmbedding vector property and the + // Description full text search property. + var hybridSearchOptions = new HybridSearchOptions + { + VectorProperty = r => r.DescriptionEmbedding, + AdditionalProperty = r => r.Description + }; + + // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. + IAsyncEnumerable> searchResult = + collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); + } + } + + public sealed class Product + { + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public required string Name { get; set; } + + [VectorStoreData(IsFullTextIndexed = true)] + public required string Description { get; set; } + + [VectorStoreData] + public required List FeatureList { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } + } + // + + public class TopAndSkipExample + { + private object searchVector; + private IKeywordHybridSearchable collection; + + public async Task TopAndSkipSearch() + { + // + // Create the vector search options and indicate that you want to skip the first 40 results and then pass 20 to search to get the next 20. + HybridSearchOptions hybridSearchOptions = new() + { + Skip = 40 + }; + + // This snippet assumes searchVector is already provided, + // having been created using the embedding model of your choice. + IAsyncEnumerable> searchResult = + collection.HybridSearchAsync( + searchVector, + ["happiness", "hotel", "customer"], + top: 20, + hybridSearchOptions); + + // Iterate over the search results. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine(result.Record.Description); + } + // + + // + // Create the hybrid search options and indicate that you want to include vectors in the search results. + hybridSearchOptions = new HybridSearchOptions() + { + IncludeVectors = true + }; + + // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. + searchResult = collection.HybridSearchAsync( + searchVector, + ["happiness", "hotel", "customer"], + top: 3, + hybridSearchOptions); + + // Iterate over the search results. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine(result.Record.FeatureList); + } + // + } + } + + public class FilterExample + { + private object searchVector; + private IKeywordHybridSearchable collection; + + // + public async Task FilterSearch() + { + // Create the hybrid search options and set the filter on the options. + HybridSearchOptions searchOptions = new() + { + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") + }; + + // This snippet assumes searchVector is already provided, + // having been created using the embedding model of your choice. + IAsyncEnumerable> searchResult = + collection.HybridSearchAsync( + searchVector, + ["happiness", "hotel", "customer"], + top: 3, + searchOptions); + + // Iterate over the search results. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine(result.Record.Definition); + } + } + + sealed class Glossary + { + [VectorStoreKey] + public ulong Key { get; set; } + + // Category is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public required string Category { get; set; } + + // Tags is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public required List Tags { get; set; } + [VectorStoreData] + public required string Term { get; set; } + + [VectorStoreData] + public required string Definition { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } + } + // + } +} diff --git a/docs/ai/vector-stores/snippets/conceptual/schema-with-record-definition.cs b/docs/ai/vector-stores/snippets/conceptual/schema-with-record-definition.cs new file mode 100644 index 0000000000000..f11f4bbcd98e8 --- /dev/null +++ b/docs/ai/vector-stores/snippets/conceptual/schema-with-record-definition.cs @@ -0,0 +1,20 @@ +using Microsoft.Extensions.VectorData; + +public class RecordDefinitionExample +{ + public void CreateHotelDefinition() + { + // + VectorStoreCollectionDefinition hotelDefinition = new() + { + Properties = + [ + new VectorStoreKeyProperty("HotelId", typeof(ulong)), + new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, + new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, + new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, + ] + }; + // + } +} diff --git a/docs/ai/vector-stores/snippets/conceptual/vector-search.cs b/docs/ai/vector-stores/snippets/conceptual/vector-search.cs new file mode 100644 index 0000000000000..1ddc3e25838bd --- /dev/null +++ b/docs/ai/vector-stores/snippets/conceptual/vector-search.cs @@ -0,0 +1,177 @@ +using System.Security.Cryptography.X509Certificates; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.InMemory; +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Qdrant.Client; + +public class VectorSearchExample +{ + private object searchVector; + private VectorStoreCollection collection; + + // + public async Task SearchAsync() + { + // Create a Qdrant VectorStore object and choose + // an existing collection that already contains records. + VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); + VectorStoreCollection collection = + vectorStore.GetCollection("skhotels"); + + // Generate a vector for your search text, using + // your chosen embedding generation implementation. + ReadOnlyMemory searchVector = + await GenerateAsync("I'm looking for a hotel where customer happiness is the priority."); + + // Do the search, passing an options object with + // a Top value to limit results to the single top match. + IAsyncEnumerable> searchResult = + collection.SearchAsync(searchVector, top: 1); + + // Inspect the returned hotel. + await foreach (VectorSearchResult record in searchResult) + { + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); + } + } + // + + private async Task> GenerateAsync(string v) => + throw new NotImplementedException(); + + // + public async Task VectorPropertySearch() + { + var vectorStore = new InMemoryVectorStore(); + InMemoryCollection collection = + vectorStore.GetCollection("skproducts"); + + // Create the vector search options and indicate that you want to search the FeatureListEmbedding property. + var vectorSearchOptions = new VectorSearchOptions + { + VectorProperty = r => r.FeatureListEmbedding + }; + + // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. + IAsyncEnumerable> searchResult = + collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + } + + public sealed class Product + { + [VectorStoreKey] + public int Key { get; set; } + + [VectorStoreData] + public required string Description { get; set; } + + [VectorStoreData] + public required List FeatureList { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } + } + // + // Create the vector search options and indicate + // that you want to skip the first 40 results. + VectorSearchOptions vectorSearchOptions = new() + { + Skip = 40 + }; + + // This snippet assumes searchVector is already provided, + // having been created using the embedding model of your choice. + // Pass 'top: 20' to indicate that you want to retrieve + // the next 20 results after skipping the first 40. + IAsyncEnumerable> searchResult = + collection.SearchAsync(searchVector, top: 20, vectorSearchOptions); + + // Iterate over the search results. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine(result.Record.FeatureList); + } + // + } + + public async Task IncludeVectorsSearch() + { + // + // Create the vector search options and indicate that you want to include vectors in the search results. + var vectorSearchOptions = new VectorSearchOptions + { + IncludeVectors = true + }; + + // This snippet assumes searchVector is already provided, + // having been created using the embedding model of your choice. + IAsyncEnumerable> searchResult = + collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + + // Iterate over the search results. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine(result.Record.FeatureList); + } + // + } +} + +public class FilterExample +{ + private static object searchVector; + private static VectorStoreCollection collection; + + // + public static async Task FilteredSearchAsync() + { + // Create the vector search options and set the filter on the options. + VectorSearchOptions vectorSearchOptions = new() + { + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") + }; + + // This snippet assumes searchVector is already provided, + // having been created using the embedding model of your choice. + IAsyncEnumerable> searchResult = + collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); + + // Iterate over the search results. + await foreach (VectorSearchResult result in searchResult) + { + Console.WriteLine(result.Record.Definition); + + } + } + + sealed class Glossary + { + [VectorStoreKey] + public ulong Key { get; set; } + + // Category is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public required string Category { get; set; } + + // Tags is marked as indexed, since you want to filter using this property. + [VectorStoreData(IsIndexed = true)] + public required List Tags { get; set; } + [VectorStoreData] + public required string Term { get; set; } + + [VectorStoreData] + public required string Definition { get; set; } + + [VectorStoreVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } + } + // +} diff --git a/docs/ai/vector-stores/snippets/defining-your-data-model.cs b/docs/ai/vector-stores/snippets/defining-your-data-model.cs deleted file mode 100644 index 2cde90a27efc5..0000000000000 --- a/docs/ai/vector-stores/snippets/defining-your-data-model.cs +++ /dev/null @@ -1,41 +0,0 @@ -// -using Microsoft.Extensions.VectorData; - -public class Hotel -{ - [VectorStoreKey] - public ulong HotelId { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string HotelName { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] - public ReadOnlyMemory? DescriptionEmbedding { get; set; } - - [VectorStoreData(IsIndexed = true)] - public string[] Tags { get; set; } -} -// - -// -[VectorStoreKey] -public ulong HotelId { get; set; } -// - -// -[VectorStoreData(IsIndexed = true)] -public string HotelName { get; set; } -// - -// -[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] -public ReadOnlyMemory? DescriptionEmbedding { get; set; } -// - -// -[VectorStoreVector(Dimensions: 4, DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw)] -public string DescriptionEmbedding { get; set; } -// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/dynamic-data-model.cs b/docs/ai/vector-stores/snippets/dynamic-data-model.cs deleted file mode 100644 index b2226d3a3ede8..0000000000000 --- a/docs/ai/vector-stores/snippets/dynamic-data-model.cs +++ /dev/null @@ -1,39 +0,0 @@ -// -// Create the definition to define the schema. -VectorStoreCollectionDefinition definition = new() -{ - Properties = new List - { - new VectorStoreKeyProperty("Key", typeof(string)), - new VectorStoreDataProperty("Term", typeof(string)), - new VectorStoreDataProperty("Definition", typeof(string)), - new VectorStoreVectorProperty("DefinitionEmbedding", typeof(ReadOnlyMemory), dimensions: 1536) - } -}; - -// When getting your collection instance from a vector store instance -// specify the Dictionary, using object as the key type for your database -// and also pass your record definition. -// Note that you have to use GetDynamicCollection instead of the regular GetCollection method -// to get an instance of a collection using Dictionary. -var dynamicDataModelCollection = vectorStore.GetDynamicCollection( - "glossary", - definition); - -// Since schema information is available from the record definition -// it's possible to create a collection with the right vectors, -// dimensions, indexes, and distance functions. -await dynamicDataModelCollection.EnsureCollectionExistsAsync(); - -// When retrieving a record from the collection, key, data, and vector values can -// now be accessed via the dictionary entries. -var record = await dynamicDataModelCollection.GetAsync("SK"); -Console.WriteLine(record["Definition"]); -// - -// -new AzureAISearchDynamicCollection( - searchIndexClient, - "glossary", - new() { Definition = definition }); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/embedding-generation.cs b/docs/ai/vector-stores/snippets/embedding-generation.cs deleted file mode 100644 index 652619fc98e34..0000000000000 --- a/docs/ai/vector-stores/snippets/embedding-generation.cs +++ /dev/null @@ -1,202 +0,0 @@ -// -[VectorStoreVector(1536)] -public string Embedding { get; set; } -// - -// -using Microsoft.Extensions.AI; -using Microsoft.SemanticKernel.Connectors.Qdrant; -using OpenAI; -using Qdrant.Client; - -var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - -var vectorStore = new QdrantVectorStore( - new QdrantClient("localhost"), - ownsClient: true, - new QdrantVectorStoreOptions - { - EmbeddingGenerator = embeddingGenerator - }); -// - -// -using Microsoft.Extensions.AI; -using Microsoft.SemanticKernel.Connectors.Qdrant; -using OpenAI; -using Qdrant.Client; - -var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - -var collectionOptions = new QdrantCollectionOptions -{ - EmbeddingGenerator = embeddingGenerator -}; -var collection = new QdrantCollection( - new QdrantClient("localhost"), - "myCollection", - ownsClient: true, - collectionOptions); -// - -// -using Microsoft.Extensions.AI; -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.Qdrant; -using OpenAI; -using Qdrant.Client; - -var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - -var definition = new VectorStoreCollectionDefinition -{ - EmbeddingGenerator = embeddingGenerator, - Properties = new List - { - new VectorStoreKeyProperty("Key", typeof(ulong)), - new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) - } -}; - -var collectionOptions = new QdrantCollectionOptions -{ - Definition = definition -}; -var collection = new QdrantCollection( - new QdrantClient("localhost"), - "myCollection", - ownsClient: true, - collectionOptions); -// - -// -using Microsoft.Extensions.AI; -using Microsoft.Extensions.VectorData; -using OpenAI; - -var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - -var vectorProperty = new VectorStoreVectorProperty("DescriptionEmbedding", typeof(string), dimensions: 1536) -{ - EmbeddingGenerator = embeddingGenerator -}; -// - -// - -// The data model -internal class FinanceInfo -{ - [VectorStoreKey] - public string Key { get; set; } = string.Empty; - - [VectorStoreData] - public string Text { get; set; } = string.Empty; - - // Note that the vector property is typed as a string, and - // its value is derived from the Text property. The string - // value will however be converted to a vector on upsert and - // stored in the database as a vector. - [VectorStoreVector(1536)] - public string Embedding => this.Text; -} - -// Create an OpenAI embedding generator. -var embeddingGenerator = new OpenAIClient("your key") - .GetEmbeddingClient("your chosen model") - .AsIEmbeddingGenerator(); - -// Use the embedding generator with the vector store. -var vectorStore = new InMemoryVectorStore(new() { EmbeddingGenerator = embeddingGenerator }); -var collection = vectorStore.GetCollection("finances"); -await collection.EnsureCollectionExistsAsync(); - -// Create some test data. -string[] budgetInfo = -{ - "The budget for 2020 is EUR 100 000", - "The budget for 2021 is EUR 120 000", - "The budget for 2022 is EUR 150 000", - "The budget for 2023 is EUR 200 000", - "The budget for 2024 is EUR 364 000" -}; - -// Embeddings are generated automatically on upsert. -var records = budgetInfo.Select((input, index) => new FinanceInfo { Key = index.ToString(), Text = input }); -await collection.UpsertAsync(records); - -// Embeddings for the search is automatically generated on search. -var searchResult = collection.SearchAsync( - "What is my budget for 2024?", - top: 1); - -// Output the matching result. -await foreach (var result in searchResult) -{ - Console.WriteLine($"Key: {result.Record.Key}, Text: {result.Record.Text}"); -} -// - -// -public async Task GenerateEmbeddingsAndUpsertAsync( - IEmbeddingGenerator> embeddingGenerator, - VectorStoreCollection collection) -{ - // Upsert a record. - string descriptionText = "A place where everyone can be happy."; - ulong hotelId = 1; - - // Generate the embedding. - ReadOnlyMemory embedding = - (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; - - // Create a record and upsert with the already generated embedding. - await collection.UpsertAsync(new Hotel - { - HotelId = hotelId, - HotelName = "Hotel Happy", - Description = descriptionText, - DescriptionEmbedding = embedding, - Tags = new[] { "luxury", "pool" } - }); -} -// - -// -public async Task GenerateEmbeddingsAndSearchAsync( - IEmbeddingGenerator> embeddingGenerator, - VectorStoreCollection collection) -{ - // Upsert a record. - string descriptionText = "Find me a hotel with happiness in mind."; - - // Generate the embedding. - ReadOnlyMemory searchEmbedding = - (await embeddingGenerator.GenerateEmbeddingAsync(descriptionText)).Vector; - - // Search using the already generated embedding. - IAsyncEnumerable> searchResult = collection.SearchAsync(searchEmbedding, top: 1); - List> resultItems = await searchResult.ToListAsync(); - - // Print the first search result. - Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); - Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); -} -// - -// -[VectorStoreVector(Dimensions: 1536)] -public ReadOnlyMemory? DescriptionEmbedding { get; set; } -// - -// -new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 1536); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs b/docs/ai/vector-stores/snippets/how-to/DataIngestion.cs similarity index 66% rename from docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs rename to docs/ai/vector-stores/snippets/how-to/DataIngestion.cs index b567b806db996..7f2ae3797123a 100644 --- a/docs/ai/vector-stores/how-to/snippets/vector-store-data-ingestion.cs +++ b/docs/ai/vector-stores/snippets/how-to/DataIngestion.cs @@ -1,19 +1,23 @@ -// +using System.Text; +using System.Xml; +using DocumentFormat.OpenXml.Packaging; +using Microsoft.Extensions.AI; using Microsoft.Extensions.VectorData; -namespace SKVectorIngest; +namespace VectorIngest; +// internal class TextParagraph { /// A unique key for the text paragraph. [VectorStoreKey] public required string Key { get; init; } - /// A uri that points at the original location of the document containing the text. + /// A URI that points at the original location of the document containing the text. [VectorStoreData] public required string DocumentUri { get; init; } - /// The id of the paragraph from the document containing the text. + /// The ID of the paragraph from the document containing the text. [VectorStoreData] public required string ParagraphId { get; init; } @@ -21,19 +25,13 @@ internal class TextParagraph [VectorStoreData] public required string Text { get; init; } - /// The embedding generated from the Text. + /// The embedding generated from the text. [VectorStoreVector(1536)] public ReadOnlyMemory TextEmbedding { get; set; } } // // -using System.Text; -using System.Xml; -using DocumentFormat.OpenXml.Packaging; - -namespace SKVectorIngest; - internal class DocumentReader { public static IEnumerable ReadParagraphs(Stream documentContents, string documentUri) @@ -82,7 +80,7 @@ public static IEnumerable ReadParagraphs(Stream documentContents, } // Yield a new TextParagraph if the combined text is not empty. - var combinedText = textBuilder.ToString(); + string combinedText = textBuilder.ToString(); if (!string.IsNullOrWhiteSpace(combinedText)) { Console.WriteLine("Found paragraph:"); @@ -103,12 +101,10 @@ public static IEnumerable ReadParagraphs(Stream documentContents, // // -using Microsoft.Extensions.AI; -using Microsoft.Extensions.VectorData; - -namespace SKVectorIngest; - -internal class DataUploader(VectorStore vectorStore, IEmbeddingGenerator> embeddingGenerator) +internal class DataUploader( + VectorStore vectorStore, + IEmbeddingGenerator> embeddingGenerator) { /// /// Generate an embedding for each text paragraph and upload it to the specified collection. @@ -118,14 +114,15 @@ internal class DataUploader(VectorStore vectorStore, IEmbeddingGeneratorAn async task. public async Task GenerateEmbeddingsAndUpload(string collectionName, IEnumerable textParagraphs) { - var collection = vectorStore.GetCollection(collectionName); + VectorStoreCollection collection = + vectorStore.GetCollection(collectionName); await collection.EnsureCollectionExistsAsync(); - foreach (var paragraph in textParagraphs) + foreach (TextParagraph paragraph in textParagraphs) { // Generate the text embedding. Console.WriteLine($"Generating embedding for paragraph: {paragraph.ParagraphId}"); - paragraph.TextEmbedding = (await embeddingGenerator.GenerateEmbeddingAsync(paragraph.Text)).Vector; + paragraph.TextEmbedding = (await embeddingGenerator.GenerateAsync(paragraph.Text)).Vector; // Upload the text paragraph. Console.WriteLine($"Upserting paragraph: {paragraph.ParagraphId}"); @@ -136,47 +133,3 @@ public async Task GenerateEmbeddingsAndUpload(string collectionName, IEnumerable } } // - -// -using Azure; -using Azure.AI.OpenAI; -using Microsoft.Extensions.AI; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.Redis; -using SKVectorIngest; - -// Replace with your values. -var deploymentName = "text-embedding-ada-002"; -var endpoint = "https://sksample.openai.azure.com/"; -var apiKey = "your-api-key"; - -// Register Azure OpenAI embedding generator and Redis vector store. -var services = new ServiceCollection(); -services.AddSingleton>>( - new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey)) - .GetEmbeddingClient(deploymentName) - .AsIEmbeddingGenerator()); - -services.AddRedisVectorStore("localhost:6379"); - -// Register the data uploader. -services.AddSingleton(); - -// Build the service provider and get the data uploader. -var serviceProvider = services.BuildServiceProvider(); -var dataUploader = serviceProvider.GetRequiredService(); -// - -// -// Load the data. -var textParagraphs = DocumentReader.ReadParagraphs( - new FileStream( - "vector-store-data-ingestion-input.docx", - FileMode.Open), - "file:///c:/vector-store-data-ingestion-input.docx"); - -await dataUploader.GenerateEmbeddingsAndUpload( - "sk-documentation", - textParagraphs); -// \ No newline at end of file diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Hotel.cs b/docs/ai/vector-stores/snippets/how-to/Hotel.cs similarity index 95% rename from docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Hotel.cs rename to docs/ai/vector-stores/snippets/how-to/Hotel.cs index 05629de6cacc9..632bffd5fe3ca 100644 --- a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Hotel.cs +++ b/docs/ai/vector-stores/snippets/how-to/Hotel.cs @@ -1,7 +1,7 @@ // using Microsoft.Extensions.VectorData; -public class Hotel +public record class Hotel { [VectorStoreKey] public int HotelId { get; set; } diff --git a/docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj b/docs/ai/vector-stores/snippets/how-to/HowToSnippets.csproj similarity index 71% rename from docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj rename to docs/ai/vector-stores/snippets/how-to/HowToSnippets.csproj index ba1408b632897..76610bbb55fa4 100644 --- a/docs/ai/vector-stores/how-to/snippets/HowToSnippets.csproj +++ b/docs/ai/vector-stores/snippets/how-to/HowToSnippets.csproj @@ -1,6 +1,7 @@ - + + Exe net10.0 enable enable @@ -11,8 +12,10 @@ + - + + diff --git a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Program.cs b/docs/ai/vector-stores/snippets/how-to/Program.cs similarity index 95% rename from docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Program.cs rename to docs/ai/vector-stores/snippets/how-to/Program.cs index 6577317079d05..9031980080dcc 100644 --- a/docs/ai/vector-stores/how-to/snippets/VectorStoresExamples/Program.cs +++ b/docs/ai/vector-stores/snippets/how-to/Program.cs @@ -1,5 +1,4 @@ -using Microsoft.Extensions.AI; -using Microsoft.Extensions.VectorData; +using Microsoft.Extensions.VectorData; using Microsoft.SemanticKernel.Connectors.InMemory; // @@ -50,9 +49,10 @@ } }; -await collection.UpsertAsync(hotels[0]); -await collection.UpsertAsync(hotels[1]); -await collection.UpsertAsync(hotels[2]); +foreach (Hotel h in hotels) +{ + await collection.UpsertAsync(h); +} // // diff --git a/docs/ai/vector-stores/snippets/how-to/PutItAllTogether.cs b/docs/ai/vector-stores/snippets/how-to/PutItAllTogether.cs new file mode 100644 index 0000000000000..d174c2bb45683 --- /dev/null +++ b/docs/ai/vector-stores/snippets/how-to/PutItAllTogether.cs @@ -0,0 +1,50 @@ +using Azure; +using Azure.AI.OpenAI; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.VectorData; +using Microsoft.SemanticKernel.Connectors.Redis; +using VectorIngest; + +public class PutItAllTogether +{ + public static async Task Main() + { + // + // Replace with your values. + string deploymentName = "text-embedding-ada-002"; + string endpoint = "https://sksample.openai.azure.com/"; + string apiKey = "your-api-key"; + + // Register Azure OpenAI embedding generator and Redis vector store. + var services = new ServiceCollection(); + services.AddSingleton>>( + new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey)) + .GetEmbeddingClient(deploymentName) + .AsIEmbeddingGenerator()); + + services.AddRedisVectorStore("localhost:6379"); + + // Register the data uploader. + services.AddSingleton(); + + // Build the service provider and get the data uploader. + ServiceProvider serviceProvider = services.BuildServiceProvider(); + DataUploader dataUploader = serviceProvider.GetRequiredService(); + // + + // + // Load the data. + IEnumerable textParagraphs = DocumentReader.ReadParagraphs( + new FileStream( + "vector-store-data-ingestion-input.docx", + FileMode.Open), + "file:///c:/vector-store-data-ingestion-input.docx"); + + await dataUploader.GenerateEmbeddingsAndUpload( + "sk-documentation", + textParagraphs); + + // + } +} diff --git a/docs/ai/vector-stores/snippets/hybrid-search.cs b/docs/ai/vector-stores/snippets/hybrid-search.cs deleted file mode 100644 index 686d03eadad08..0000000000000 --- a/docs/ai/vector-stores/snippets/hybrid-search.cs +++ /dev/null @@ -1,144 +0,0 @@ -// -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Microsoft.Extensions.VectorData; -using Qdrant.Client; - -// Placeholder embedding generation method. -async Task> GenerateEmbeddingAsync(string textToVectorize) -{ - // your logic here -} - -// Create a Qdrant VectorStore object and choose an existing collection that already contains records. -VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -IKeywordHybridSearchable collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skhotels"); - -// Generate a vector for your search text, using your chosen embedding generation implementation. -ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); - -// Do the search, passing an options object with a Top value to limit results to the single top match. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 1); - -// Inspect the returned hotel. -await foreach (var record in searchResult) -{ - Console.WriteLine("Found hotel description: " + record.Record.Description); - Console.WriteLine("Found record score: " + record.Score); -} -// - -// -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Microsoft.Extensions.VectorData; -using Qdrant.Client; - -var vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -var collection = (IKeywordHybridSearchable)vectorStore.GetCollection("skproducts"); - -// Create the hybrid search options and indicate that you want -// to search the DescriptionEmbedding vector property and the -// Description full text search property. -var hybridSearchOptions = new HybridSearchOptions -{ - VectorProperty = r => r.DescriptionEmbedding, - AdditionalProperty = r => r.Description -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); - -public sealed class Product -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Name { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Description { get; set; } - - [VectorStoreData] - public List FeatureList { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DescriptionEmbedding { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory FeatureListEmbedding { get; set; } -} -// - -// -// Create the vector search options and indicate that you want to skip the first 40 results and then pass 20 to search to get the next 20. -var hybridSearchOptions = new HybridSearchOptions -{ - Skip = 40 -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 20, hybridSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.Description); -} -// - -// -// Create the hybrid search options and indicate that you want to include vectors in the search results. -var hybridSearchOptions = new HybridSearchOptions -{ - IncludeVectors = true -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.FeatureList); -} -// - -// -// Create the hybrid search options and set the filter on the options. -var hybridSearchOptions = new HybridSearchOptions -{ - Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], top: 3, hybridSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.Definition); -} - -sealed class Glossary -{ - [VectorStoreKey] - public ulong Key { get; set; } - - // Category is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public string Category { get; set; } - - // Tags is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public List Tags { get; set; } - - [VectorStoreData] - public string Term { get; set; } - - [VectorStoreData(IsFullTextIndexed = true)] - public string Definition { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DefinitionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/schema-with-record-definition.cs b/docs/ai/vector-stores/snippets/schema-with-record-definition.cs deleted file mode 100644 index f4af9be040c88..0000000000000 --- a/docs/ai/vector-stores/snippets/schema-with-record-definition.cs +++ /dev/null @@ -1,30 +0,0 @@ -// -using Microsoft.Extensions.VectorData; - -var hotelDefinition = new VectorStoreCollectionDefinition -{ - Properties = new List - { - new VectorStoreKeyProperty("HotelId", typeof(ulong)), - new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, - new VectorStoreDataProperty("Description", typeof(string)) { IsFullTextIndexed = true }, - new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, - } -}; -// - -// -var collection = vectorStore.GetCollection("skhotels", hotelDefinition); -// - -// -new VectorStoreKeyProperty("HotelId", typeof(ulong)), -// - -// -new VectorStoreDataProperty("HotelName", typeof(string)) { IsIndexed = true }, -// - -// -new VectorStoreVectorProperty("DescriptionEmbedding", typeof(float), dimensions: 4) { DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw }, -// \ No newline at end of file diff --git a/docs/ai/vector-stores/snippets/vector-search.cs b/docs/ai/vector-stores/snippets/vector-search.cs deleted file mode 100644 index 24e5cb0412110..0000000000000 --- a/docs/ai/vector-stores/snippets/vector-search.cs +++ /dev/null @@ -1,139 +0,0 @@ -// -using Microsoft.SemanticKernel.Connectors.Qdrant; -using Microsoft.Extensions.VectorData; -using Qdrant.Client; - -// Placeholder embedding generation method. -async Task> GenerateEmbeddingAsync(string textToVectorize) -{ - // your logic here -} - -// Create a Qdrant VectorStore object and choose an existing collection that already contains records. -VectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost"), ownsClient: true); -VectorStoreCollection collection = vectorStore.GetCollection("skhotels"); - -// Generate a vector for your search text, using your chosen embedding generation implementation. -ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); - -// Do the search, passing an options object with a Top value to limit results to the single top match. -var searchResult = collection.SearchAsync(searchVector, top: 1); - -// Inspect the returned hotel. -await foreach (var record in searchResult) -{ - Console.WriteLine("Found hotel description: " + record.Record.Description); - Console.WriteLine("Found record score: " + record.Score); -} -// - -// -using Microsoft.Extensions.VectorData; -using Microsoft.SemanticKernel.Connectors.InMemory; - -var vectorStore = new InMemoryVectorStore(); -var collection = vectorStore.GetCollection("skproducts"); - -// Create the vector search options and indicate that you want to search the FeatureListEmbedding property. -var vectorSearchOptions = new VectorSearchOptions -{ - VectorProperty = r => r.FeatureListEmbedding -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); - -public sealed class Product -{ - [VectorStoreKey] - public int Key { get; set; } - - [VectorStoreData] - public string Description { get; set; } - - [VectorStoreData] - public List FeatureList { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DescriptionEmbedding { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory FeatureListEmbedding { get; set; } -} -// - -// -// Create the vector search options and indicate that you want to skip the first 40 results. -var vectorSearchOptions = new VectorSearchOptions -{ - Skip = 40 -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -// Pass 'top: 20' to indicate that you want to retrieve the next 20 results after skipping -// the first 40 -var searchResult = collection.SearchAsync(searchVector, top: 20, vectorSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.FeatureList); -} -// - -// -// Create the vector search options and indicate that you want to include vectors in the search results. -var vectorSearchOptions = new VectorSearchOptions -{ - IncludeVectors = true -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.FeatureList); -} -// - -// -// Create the vector search options and set the filter on the options. -var vectorSearchOptions = new VectorSearchOptions -{ - Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") -}; - -// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = collection.SearchAsync(searchVector, top: 3, vectorSearchOptions); - -// Iterate over the search results. -await foreach (var result in searchResult) -{ - Console.WriteLine(result.Record.Definition); -} - -sealed class Glossary -{ - [VectorStoreKey] - public ulong Key { get; set; } - - // Category is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public string Category { get; set; } - - // Tags is marked as indexed, since you want to filter using this property. - [VectorStoreData(IsIndexed = true)] - public List Tags { get; set; } - - [VectorStoreData] - public string Term { get; set; } - - [VectorStoreData] - public string Definition { get; set; } - - [VectorStoreVector(1536)] - public ReadOnlyMemory DefinitionEmbedding { get; set; } -} -// \ No newline at end of file diff --git a/docs/ai/vector-stores/tutorial-vector-search.md b/docs/ai/vector-stores/tutorial-vector-search.md index a3f2f50775278..337e1116f9d90 100644 --- a/docs/ai/vector-stores/tutorial-vector-search.md +++ b/docs/ai/vector-stores/tutorial-vector-search.md @@ -1,7 +1,7 @@ --- title: Tutorial - Integrate OpenAI with the RAG pattern and vector search using Azure Cosmos DB for MongoDB description: Create a simple recipe app using the RAG pattern and vector search using Azure Cosmos DB for MongoDB. -ms.date: 08/26/2025 +ms.date: 02/28/2026 ms.topic: tutorial author: alexwolfmsft ms.author: alexwolf @@ -29,27 +29,27 @@ The Cosmos Recipe Guide app allows you to perform vector and AI driven searches 1. Perform vector similarity search based on the user prompts. 1. Use the Azure OpenAI `gpt-35-turbo` completions model to compose more meaningful answers based on the search results data. - :::image type="content" source="../media/get-started-app-chat-template/contoso-recipes.png" alt-text="A screenshot showing the running sample app."::: + :::image type="content" source="../media/get-started-app-chat-template/contoso-recipes.png" alt-text="A screenshot showing the running sample app."::: ## Get started 1. Clone the following GitHub repository: - ```bash - git clone https://github.com/microsoft/AzureDataRetrievalAugmentedGenerationSamples.git - ``` + ```bash + git clone https://github.com/microsoft/AzureDataRetrievalAugmentedGenerationSamples.git + ``` 1. In the _C#/CosmosDB-MongoDBvCore_ folder, open the **CosmosRecipeGuide.sln** file. 1. In the _appsettings.json_ file, replace the following config values with your Azure OpenAI and Azure CosmosDB for MongoDb values: - ```json - "OpenAIEndpoint": "https://.openai.azure.com/", - "OpenAIKey": "", - "OpenAIEmbeddingDeployment": "", - "OpenAIcompletionsDeployment": "", - "MongoVcoreConnection": "" - ``` + ```json + "OpenAIEndpoint": "https://.openai.azure.com/", + "OpenAIKey": "", + "OpenAIEmbeddingDeployment": "", + "OpenAIcompletionsDeployment": "", + "MongoVcoreConnection": "" + ``` 1. Launch the app by pressing the **Start** button at the top of Visual Studio. @@ -59,238 +59,238 @@ When you run the app for the first time, it connects to Azure Cosmos DB and repo 1. Select **Upload recipe(s) to Cosmos DB** and press Enter. This command reads sample JSON files from the local project and uploads them to the Cosmos DB account. - The code from the _Utility.cs_ class parses the local JSON files. - - ``` C# - public static List ParseDocuments(string Folderpath) - { - List recipes = new List(); - - Directory.GetFiles(Folderpath) - .ToList() - .ForEach(f => - { - var jsonString= System.IO.File.ReadAllText(f); - Recipe recipe = JsonConvert.DeserializeObject(jsonString); - recipe.id = recipe.name.ToLower().Replace(" ", ""); - recipes.Add(recipe); - } - ); - - return recipes; - } - ``` - - The `UpsertVectorAsync` method in the _VCoreMongoService.cs_ file uploads the documents to Azure Cosmos DB for MongoDB. - - ```C# - public async Task UpsertVectorAsync(Recipe recipe) - { - BsonDocument document = recipe.ToBsonDocument(); - - if (!document.Contains("_id")) - { - Console.WriteLine("UpsertVectorAsync: Document does not contain _id."); - throw new ArgumentException("UpsertVectorAsync: Document does not contain _id."); - } - - string? _idValue = document["_id"].ToString(); - - try - { - var filter = Builders.Filter.Eq("_id", _idValue); - var options = new ReplaceOptions { IsUpsert = true }; - await _recipeCollection.ReplaceOneAsync(filter, document, options); - } - catch (Exception ex) - { - Console.WriteLine($"Exception: UpsertVectorAsync(): {ex.Message}"); - throw; - } - } - ``` + The code from the _Utility.cs_ class parses the local JSON files. + + ```csharp + public static List ParseDocuments(string Folderpath) + { + List recipes = new List(); + + Directory.GetFiles(Folderpath) + .ToList() + .ForEach(f => + { + var jsonString= System.IO.File.ReadAllText(f); + Recipe recipe = JsonConvert.DeserializeObject(jsonString); + recipe.id = recipe.name.ToLower().Replace(" ", ""); + recipes.Add(recipe); + } + ); + + return recipes; + } + ``` + + The `UpsertVectorAsync` method in the _VCoreMongoService.cs_ file uploads the documents to Azure Cosmos DB for MongoDB. + + ```csharp + public async Task UpsertVectorAsync(Recipe recipe) + { + BsonDocument document = recipe.ToBsonDocument(); + + if (!document.Contains("_id")) + { + Console.WriteLine("UpsertVectorAsync: Document does not contain _id."); + throw new ArgumentException("UpsertVectorAsync: Document does not contain _id."); + } + + string? _idValue = document["_id"].ToString(); + + try + { + var filter = Builders.Filter.Eq("_id", _idValue); + var options = new ReplaceOptions { IsUpsert = true }; + await _recipeCollection.ReplaceOneAsync(filter, document, options); + } + catch (Exception ex) + { + Console.WriteLine($"Exception: UpsertVectorAsync(): {ex.Message}"); + throw; + } + } + ``` 1. Select **Vectorize the recipe(s) and store them in Cosmos DB**. - The JSON items uploaded to Cosmos DB do not contain embeddings and therefore are not optimized for RAG via vector search. An embedding is an information-dense, numerical representation of the semantic meaning of a piece of text. Vector searches are able to find items with contextually similar embeddings. - - The `GetEmbeddingsAsync` method in the _OpenAIService.cs_ file creates an embedding for each item in the database. - - ```C# - public async Task GetEmbeddingsAsync(dynamic data) - { - try - { - EmbeddingsOptions options = new EmbeddingsOptions(data) - { - Input = data - }; - - var response = await _openAIClient.GetEmbeddingsAsync(openAIEmbeddingDeployment, options); - - Embeddings embeddings = response.Value; - float[] embedding = embeddings.Data[0].Embedding.ToArray(); - - return embedding; - } - catch (Exception ex) - { - Console.WriteLine($"GetEmbeddingsAsync Exception: {ex.Message}"); - return null; - } - } - ``` - - The `CreateVectorIndexIfNotExists` in the _VCoreMongoService.cs_ file creates a vector index, which enables you to perform vector similarity searches. - - ```C# - public void CreateVectorIndexIfNotExists(string vectorIndexName) - { - try - { - //Find if vector index exists in vectors collection - using (IAsyncCursor indexCursor = _recipeCollection.Indexes.List()) - { - bool vectorIndexExists = indexCursor.ToList().Any(x => x["name"] == vectorIndexName); - if (!vectorIndexExists) - { - BsonDocumentCommand command = new BsonDocumentCommand( - BsonDocument.Parse(@" - { createIndexes: 'Recipe', - indexes: [{ - name: 'vectorSearchIndex', - key: { embedding: 'cosmosSearch' }, - cosmosSearchOptions: { - kind: 'vector-ivf', - numLists: 5, - similarity: 'COS', - dimensions: 1536 } - }] - }")); - - BsonDocument result = _database.RunCommand(command); - if (result["ok"] != 1) - { - Console.WriteLine("CreateIndex failed with response: " + result.ToJson()); - } - } - } - } - catch (MongoException ex) - { - Console.WriteLine("MongoDbService InitializeVectorIndex: " + ex.Message); - throw; - } - } - ``` + The JSON items uploaded to Cosmos DB do not contain embeddings and therefore are not optimized for RAG via vector search. An embedding is an information-dense, numerical representation of the semantic meaning of a piece of text. Vector searches are able to find items with contextually similar embeddings. + + The `GetEmbeddingsAsync` method in the _OpenAIService.cs_ file creates an embedding for each item in the database. + + ```csharp + public async Task GetEmbeddingsAsync(dynamic data) + { + try + { + EmbeddingsOptions options = new EmbeddingsOptions(data) + { + Input = data + }; + + var response = await _openAIClient.GetEmbeddingsAsync(openAIEmbeddingDeployment, options); + + Embeddings embeddings = response.Value; + float[] embedding = embeddings.Data[0].Embedding.ToArray(); + + return embedding; + } + catch (Exception ex) + { + Console.WriteLine($"GetEmbeddingsAsync Exception: {ex.Message}"); + return null; + } + } + ``` + + The `CreateVectorIndexIfNotExists` in the _VCoreMongoService.cs_ file creates a vector index, which enables you to perform vector similarity searches. + + ```csharp + public void CreateVectorIndexIfNotExists(string vectorIndexName) + { + try + { + //Find if vector index exists in vectors collection + using (IAsyncCursor indexCursor = _recipeCollection.Indexes.List()) + { + bool vectorIndexExists = indexCursor.ToList().Any(x => x["name"] == vectorIndexName); + if (!vectorIndexExists) + { + BsonDocumentCommand command = new BsonDocumentCommand( + BsonDocument.Parse(@" + { createIndexes: 'Recipe', + indexes: [{ + name: 'vectorSearchIndex', + key: { embedding: 'cosmosSearch' }, + cosmosSearchOptions: { + kind: 'vector-ivf', + numLists: 5, + similarity: 'COS', + dimensions: 1536 } + }] + }")); + + BsonDocument result = _database.RunCommand(command); + if (result["ok"] != 1) + { + Console.WriteLine("CreateIndex failed with response: " + result.ToJson()); + } + } + } + } + catch (MongoException ex) + { + Console.WriteLine("MongoDbService InitializeVectorIndex: " + ex.Message); + throw; + } + } + ``` 1. Select the **Ask AI Assistant (search for a recipe by name or description, or ask a question)** option in the application to run a user query. - The user query is converted to an embedding using the Open AI service and the embedding model. The embedding is then sent to Azure Cosmos DB for MongoDB and is used to perform a vector search. The method in the _VCoreMongoService.cs_ file performs a vector search to find vectors that are close to the supplied vector and returns a list of documents from Azure Cosmos DB for MongoDB vCore. - - ```C# - public async Task> VectorSearchAsync(float[] queryVector) - { - List retDocs = new List(); - string resultDocuments = string.Empty; - - try - { - //Search Azure Cosmos DB for MongoDB vCore collection for similar embeddings - //Project the fields that are needed - BsonDocument[] pipeline = new BsonDocument[] - { - BsonDocument.Parse( - @$"{{$search: {{ - cosmosSearch: - {{ vector: [{string.Join(',', queryVector)}], - path: 'embedding', - k: {_maxVectorSearchResults}}}, - returnStoredSource:true - }} - }}"), - BsonDocument.Parse($"{{$project: {{embedding: 0}}}}"), - }; - - var bsonDocuments = await _recipeCollection - .Aggregate(pipeline).ToListAsync(); - - var recipes = bsonDocuments - .ToList() - .ConvertAll(bsonDocument => - BsonSerializer.Deserialize(bsonDocument)); - return recipes; - } - catch (MongoException ex) - { - Console.WriteLine($"Exception: VectorSearchAsync(): {ex.Message}"); - throw; - } - } - ``` + The user query is converted to an embedding using the Open AI service and the embedding model. The embedding is then sent to Azure Cosmos DB for MongoDB and is used to perform a vector search. The method in the _VCoreMongoService.cs_ file performs a vector search to find vectors that are close to the supplied vector and returns a list of documents from Azure Cosmos DB for MongoDB vCore. + + ```csharp + public async Task> VectorSearchAsync(float[] queryVector) + { + List retDocs = new List(); + string resultDocuments = string.Empty; + + try + { + //Search Azure Cosmos DB for MongoDB vCore collection for similar embeddings + //Project the fields that are needed + BsonDocument[] pipeline = new BsonDocument[] + { + BsonDocument.Parse( + @$"{{$search: {{ + cosmosSearch: + {{ vector: [{string.Join(',', queryVector)}], + path: 'embedding', + k: {_maxVectorSearchResults}}}, + returnStoredSource:true + }} + }}"), + BsonDocument.Parse($"{{$project: {{embedding: 0}}}}"), + }; + + var bsonDocuments = await _recipeCollection + .Aggregate(pipeline).ToListAsync(); + + var recipes = bsonDocuments + .ToList() + .ConvertAll(bsonDocument => + BsonSerializer.Deserialize(bsonDocument)); + return recipes; + } + catch (MongoException ex) + { + Console.WriteLine($"Exception: VectorSearchAsync(): {ex.Message}"); + throw; + } + } + ``` The `GetChatCompletionAsync` method generates an improved chat completion response based on the user prompt and the related vector search results. - ``` C# - public async Task<(string response, int promptTokens, int responseTokens)> GetChatCompletionAsync(string userPrompt, string documents) - { - try - { - ChatMessage systemMessage = new ChatMessage( - ChatRole.System, _systemPromptRecipeAssistant + documents); - ChatMessage userMessage = new ChatMessage( - ChatRole.User, userPrompt); - - ChatCompletionsOptions options = new() - { - Messages = - { - systemMessage, - userMessage - }, - MaxTokens = openAIMaxTokens, - Temperature = 0.5f, //0.3f, - NucleusSamplingFactor = 0.95f, - FrequencyPenalty = 0, - PresencePenalty = 0 - }; - - Azure.Response completionsResponse = - await openAIClient.GetChatCompletionsAsync(openAICompletionDeployment, options); - ChatCompletions completions = completionsResponse.Value; - - return ( - response: completions.Choices[0].Message.Content, - promptTokens: completions.Usage.PromptTokens, - responseTokens: completions.Usage.CompletionTokens - ); - - } - catch (Exception ex) - { - string message = $"OpenAIService.GetChatCompletionAsync(): {ex.Message}"; - Console.WriteLine(message); - throw; - } - } - ``` + ```csharp + public async Task<(string response, int promptTokens, int responseTokens)> GetChatCompletionAsync(string userPrompt, string documents) + { + try + { + ChatMessage systemMessage = new ChatMessage( + ChatRole.System, _systemPromptRecipeAssistant + documents); + ChatMessage userMessage = new ChatMessage( + ChatRole.User, userPrompt); + + ChatCompletionsOptions options = new() + { + Messages = + { + systemMessage, + userMessage + }, + MaxTokens = openAIMaxTokens, + Temperature = 0.5f, //0.3f, + NucleusSamplingFactor = 0.95f, + FrequencyPenalty = 0, + PresencePenalty = 0 + }; + + Azure.Response completionsResponse = + await openAIClient.GetChatCompletionsAsync(openAICompletionDeployment, options); + ChatCompletions completions = completionsResponse.Value; + + return ( + response: completions.Choices[0].Message.Content, + promptTokens: completions.Usage.PromptTokens, + responseTokens: completions.Usage.CompletionTokens + ); + + } + catch (Exception ex) + { + string message = $"OpenAIService.GetChatCompletionAsync(): {ex.Message}"; + Console.WriteLine(message); + throw; + } + } + ``` The app also uses prompt engineering to ensure Open AI service limits and formats the response for supplied recipes. - ```C# - //System prompts to send with user prompts to instruct the model for chat session - private readonly string _systemPromptRecipeAssistant = @" - You are an intelligent assistant for Contoso Recipes. - You are designed to provide helpful answers to user questions about - recipes, cooking instructions provided in JSON format below. - - Instructions: - - Only answer questions related to the recipe provided below. - - Don't reference any recipe not provided below. - - If you're unsure of an answer, say ""I don't know"" and recommend users search themselves. - - Your response should be complete. - - List the Name of the Recipe at the start of your response followed by step by step cooking instructions. - - Assume the user is not an expert in cooking. - - Format the content so that it can be printed to the Command Line console. - - In case there is more than one recipe you find, let the user pick the most appropriate recipe."; - ``` \ No newline at end of file + ```csharp + //System prompts to send with user prompts to instruct the model for chat session + private readonly string _systemPromptRecipeAssistant = @" + You are an intelligent assistant for Contoso Recipes. + You are designed to provide helpful answers to user questions about + recipes, cooking instructions provided in JSON format below. + + Instructions: + - Only answer questions related to the recipe provided below. + - Don't reference any recipe not provided below. + - If you're unsure of an answer, say ""I don't know"" and recommend users search themselves. + - Your response should be complete. + - List the Name of the Recipe at the start of your response followed by step by step cooking instructions. + - Assume the user is not an expert in cooking. + - Format the content so that it can be printed to the Command Line console. + - In case there is more than one recipe you find, let the user pick the most appropriate recipe."; + ``` diff --git a/docs/ai/vector-stores/vector-search.md b/docs/ai/vector-stores/vector-search.md index 3ce25df726482..7c9ee37927fdf 100644 --- a/docs/ai/vector-stores/vector-search.md +++ b/docs/ai/vector-stores/vector-search.md @@ -1,24 +1,24 @@ --- -title: Vector search using Vector Store connectors (Preview) +title: Vector search using Vector Store connectors description: Describes the different options you can use when doing a vector search using Vector Store connectors. ms.topic: concept-article -ms.date: 09/23/2024 +ms.date: 02/28/2026 --- -# Vector search using Vector Store connectors (Preview) +# Vector search using Vector Store connectors -The Microsoft.Extensions.VectorData library provides vector search capabilities as part of its Vector Store abstractions. This supports filtering and many other options, which this article will explain in more detail. +The library provides vector search capabilities as part of its Vector Store abstractions. These capabilities include filtering and many other options. > [!TIP] -> To see how you can search without generating embeddings yourself, see [Letting the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). +> To see how you can search without generating embeddings yourself, see [Let the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). ## Vector search -The method allows searching using data that has already been vectorized. This method takes a vector and an optional class as input. is available on the following types: +The method allows searching using data that has already been vectorized. This method takes a vector and an optional class as input. is available on the following types: - - -Note that implements from `IVectorSearchable`. +Note that implements `IVectorSearchable`. Assuming you have a collection that already contains data, you can easily search it. Here is an example using Qdrant. @@ -29,14 +29,9 @@ Assuming you have a collection that already contains data, you can easily search ## Supported vector types - takes a generic type as the vector parameter. -The types of vectors supported by each data store vary. -See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. + takes a generic type as the vector parameter. The types of vectors supported by each data store vary. -It's also important for the search vector type to match the target vector that is being searched, for example, if you have two vectors -on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector -you are targeting. -See [VectorProperty](#vectorproperty) for how to pick a target vector if you have more than one per record. +It's also important for the search vector type to match the target vector that is being searched, for example, if you have two vectors on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector you are targeting. For information on how to pick a target vector if you have more than one per record, see [VectorProperty](#vectorproperty). ## Vector search options @@ -44,28 +39,23 @@ The following options can be provided using the `VectorSearchOptions` c ### VectorProperty -The `VectorProperty` option can be used to specify the vector property to target during the search. -If none is provided and the data model contains only one vector, that vector will be used. -If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. +Use the `VectorProperty` option to specify the vector property to target during the search. If none is provided and the data model contains only one vector, that vector will be used. If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method throws an exceptoin. :::code language="csharp" source="./snippets/vector-search.cs" id="VectorProperty"::: ### `Top` and `Skip` -The `Top` and `Skip` options allow you to limit the number of results to the Top n results and -to skip a number of results from the top of the resultset. -Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. +The `Top` and `Skip` options allow you to limit the number of results to the top `n` results and +to skip a number of results from the top of the resultset. You can use `Top` and `Skip` to do paging if you want to retrieve a large number of results using separate calls. :::code language="csharp" source="./snippets/vector-search.cs" id="TopAndSkip"::: -The default value `Skip` is 0. +The default value for `Skip` is 0. ### IncludeVectors -The `IncludeVectors` option allows you to specify whether you wish to return vectors in the search results. -If `false`, the vector properties on the returned model will be left null. -Using `false` can significantly reduce the amount of data retrieved from the vector store during search, -making searches more efficient. +The `IncludeVectors` option allows you to specify whether you want to return vectors in the search results. +If `false`, the vector properties on the returned model are left null. Using `false` can significantly reduce the amount of data retrieved from the vector store during search, making searches more efficient. The default value for `IncludeVectors` is `false`. @@ -73,26 +63,18 @@ The default value for `IncludeVectors` is `false`. ### Filter -The vector search filter option can be used to provide a filter for filtering the records in the chosen collection -before applying the vector search. - -This has multiple benefits: +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection before applying the vector search. This has multiple benefits: - Reduce latency and processing cost, since only records remaining after filtering need to be compared with the search vector and therefore fewer vector comparisons have to be done. - Limit the resultset for for example, access control purposes, by excluding data that the user shouldn't have access to. -Note that in order for fields to be used for filtering, many vector stores require those fields to be indexed first. -Some vector stores will allow filtering using any field, but might optionally allow indexing to improve filtering performance. +For fields to be used for filtering, many vector stores require those fields to be indexed first. Some vector stores will allow filtering using any field, but might optionally allow indexing to improve filtering performance. -If creating a collection via the Vector Store abstractions and you wish to enable filtering on a field, -set the property to true when defining your data model or when creating your record definition. +If you're creating a collection via the Vector Store abstractions and you want to enable filtering on a field, set the property to `true` when defining your data model or when creating your record definition. > [!TIP] > For more information on how to set the property, see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstoredataattribute-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md#vectorstoredataproperty-configuration-settings). -Filters are expressed using LINQ expressions based on the type of the data model. -The set of LINQ expressions supported will vary depending on the functionality supported -by each database, but all databases support a broad base of common expressions, for example, equals, -not equals, and, or, etc. +Filters are expressed using LINQ expressions based on the type of the data model. The set of LINQ expressions supported will vary depending on the functionality supported by each database, but all databases support a broad base of common expressions, for example, equals, not equals, `and`, and `or`. -:::code language="csharp" source="./snippets/vector-search.cs" id="Filter"::: \ No newline at end of file +:::code language="csharp" source="./snippets/vector-search.cs" id="Filter"::: From 810dae21bcf2e9f3ec0c9445e410f80aa7c847b0 Mon Sep 17 00:00:00 2001 From: Genevieve Warren <24882762+gewarren@users.noreply.github.com> Date: Sat, 28 Feb 2026 19:54:54 -0800 Subject: [PATCH 11/11] fix build warnings --- .openpublishing.redirection.ai.json | 15 + docs/ai/conceptual/embeddings.md | 2 +- docs/ai/conceptual/understanding-tokens.md | 2 +- docs/ai/dotnet-ai-ecosystem.md | 2 +- docs/ai/overview.md | 2 +- docs/ai/toc.yml | 2 - .../vector-stores/defining-your-data-model.md | 14 +- docs/ai/vector-stores/dynamic-data-model.md | 2 +- docs/ai/vector-stores/embedding-generation.md | 16 +- .../how-to/build-vector-search-app.md | 40 +-- .../how-to/build-your-own-connector.md | 271 ------------------ .../vector-stores/how-to/use-vector-stores.md | 10 +- .../how-to/vector-store-data-ingestion.md | 14 +- docs/ai/vector-stores/hybrid-search.md | 12 +- docs/ai/vector-stores/overview.md | 10 +- .../schema-with-record-definition.md | 10 +- docs/ai/vector-stores/vector-search.md | 12 +- 17 files changed, 89 insertions(+), 347 deletions(-) delete mode 100644 docs/ai/vector-stores/how-to/build-your-own-connector.md diff --git a/.openpublishing.redirection.ai.json b/.openpublishing.redirection.ai.json index 6827c469e6d96..71417c5a967ec 100644 --- a/.openpublishing.redirection.ai.json +++ b/.openpublishing.redirection.ai.json @@ -18,6 +18,11 @@ "redirect_url": "/dotnet/ai/evaluation/libraries", "redirect_document_id": true }, + { + "source_path_from_root": "/docs/ai/conceptual/vector-databases.md", + "redirect_url": "/dotnet/ai/vector-stores/overview", + "redirect_document_id": true + }, { "source_path_from_root": "/docs/ai/get-started/dotnet-ai-overview.md", "redirect_url": "/dotnet/ai/overview", @@ -35,6 +40,11 @@ "source_path_from_root": "/docs/ai/how-to/work-with-local-models.md", "redirect_url": "/dotnet/ai" }, + { + "source_path_from_root": "/docs/ai/quickstarts/build-vector-search-app.md", + "redirect_url": "/dotnet/ai/vector-stores/how-to/build-vector-search-app", + "redirect_document_id": true + }, { "source_path_from_root": "/docs/ai/quickstarts/evaluate-ai-response.md", "redirect_url": "/dotnet/ai/evaluation/evaluate-ai-response", @@ -94,5 +104,10 @@ "redirect_url": "/dotnet/ai/evaluation/evaluate-with-reporting", "redirect_document_id": true } + { + "source_path_from_root": "/docs/ai/tutorials/tutorial-ai-vector-search.md", + "redirect_url": "/dotnet/ai/vector-stores/tutorial-vector-search", + "redirect_document_id": true + } ] } diff --git a/docs/ai/conceptual/embeddings.md b/docs/ai/conceptual/embeddings.md index 4f1f0f206b0be..6706eeb7c44c6 100644 --- a/docs/ai/conceptual/embeddings.md +++ b/docs/ai/conceptual/embeddings.md @@ -49,7 +49,7 @@ You generate embeddings for your raw data by using an AI embedding model, which ### Store and process embeddings in a vector database -After you generate embeddings, you'll need a way to store them so you can later retrieve them with calls to an LLM. Vector databases are designed to store and process vectors, so they're a natural home for embeddings. Different vector databases offer different processing capabilities, so you should choose one based on your raw data and your goals. For information about your options, see [Vector databases for .NET + AI](vector-databases.md). +After you generate embeddings, you'll need a way to store them so you can later retrieve them with calls to an LLM. Vector databases are designed to store and process vectors, so they're a natural home for embeddings. Different vector databases offer different processing capabilities, so you should choose one based on your raw data and your goals. For information about your options, see [Vector databases for .NET + AI](../vector-stores/overview.md). ### Using embeddings in your LLM solution diff --git a/docs/ai/conceptual/understanding-tokens.md b/docs/ai/conceptual/understanding-tokens.md index 9d9e76d55af4b..fc1c97ff6b25a 100644 --- a/docs/ai/conceptual/understanding-tokens.md +++ b/docs/ai/conceptual/understanding-tokens.md @@ -106,4 +106,4 @@ Generative AI services might also be limited regarding the maximum number of tok - [Use Microsoft.ML.Tokenizers for text tokenization](../how-to/use-tokenizers.md) - [How generative AI and LLMs work](how-genai-and-llms-work.md) - [Understand embeddings](embeddings.md) -- [Work with vector databases](vector-databases.md) +- [Work with vector databases](../vector-stores/overview.md) diff --git a/docs/ai/dotnet-ai-ecosystem.md b/docs/ai/dotnet-ai-ecosystem.md index ac6dcdcc5259e..97979820b7d6b 100644 --- a/docs/ai/dotnet-ai-ecosystem.md +++ b/docs/ai/dotnet-ai-ecosystem.md @@ -20,7 +20,7 @@ The .NET ecosystem provides many powerful tools, libraries, and services to deve ## Other AI-related Microsoft.Extensions libraries -The [📦 Microsoft.Extensions.VectorData.Abstractions package](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions/) provides a unified layer of abstractions for interacting with a variety of vector stores. It lets you store processed chunks in vector stores such as Qdrant, Azure SQL, CosmosDB, MongoDB, ElasticSearch, and many more. For more information, see [Build a .NET AI vector search app](quickstarts/build-vector-search-app.md). +The [📦 Microsoft.Extensions.VectorData.Abstractions package](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions/) provides a unified layer of abstractions for interacting with a variety of vector stores. It lets you store processed chunks in vector stores such as Qdrant, Azure SQL, CosmosDB, MongoDB, ElasticSearch, and many more. For more information, see [Build a .NET AI vector search app](vector-stores/how-to/build-vector-search-app.md). The [📦 Microsoft.Extensions.DataIngestion package](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion) provides foundational .NET building blocks for data ingestion. It enables developers to read, process, and prepare documents for AI and machine learning workflows, especially retrieval-augmented generation (RAG) scenarios. For more information, see [Data ingestion](conceptual/data-ingestion.md). diff --git a/docs/ai/overview.md b/docs/ai/overview.md index ed23e161a95cd..fbf5273042704 100644 --- a/docs/ai/overview.md +++ b/docs/ai/overview.md @@ -53,7 +53,7 @@ We recommend the following sequence of tutorials and articles for an introductio |-----------------------------|-------------------------------------------------------------------------| | Create a chat application | [Build an Azure AI chat app with .NET](./quickstarts/build-chat-app.md) | | Summarize text | [Summarize text using Azure AI chat app](./quickstarts/prompt-model.md) | -| Chat with your data | [Get insight about your data from a .NET Azure AI chat app](./quickstarts/build-vector-search-app.md) | +| Chat with your data | [Get insight about your data from a .NET Azure AI chat app](./vector-stores/how-to/build-vector-search-app.md) | | Call .NET functions with AI | [Extend Azure AI using tools and execute a local function with .NET](./quickstarts/use-function-calling.md) | | Generate images | [Generate images from text](./quickstarts/text-to-image.md) | | Train your own model | [ML.NET tutorial](https://dotnet.microsoft.com/learn/ml-dotnet/get-started-tutorial/intro) | diff --git a/docs/ai/toc.yml b/docs/ai/toc.yml index 710e6690a5846..a72d5c3917d53 100644 --- a/docs/ai/toc.yml +++ b/docs/ai/toc.yml @@ -106,8 +106,6 @@ items: href: vector-stores/how-to/build-vector-search-app.md - name: Ingest data into a vector store href: vector-stores/how-to/vector-store-data-ingestion.md - - name: Build a vector store connector - href: vector-stores/how-to/build-your-own-connector.md - name: Define your data model href: vector-stores/defining-your-data-model.md - name: Define schema with record definitions diff --git a/docs/ai/vector-stores/defining-your-data-model.md b/docs/ai/vector-stores/defining-your-data-model.md index c22417fc49c9c..c3e206daf4cce 100644 --- a/docs/ai/vector-stores/defining-your-data-model.md +++ b/docs/ai/vector-stores/defining-your-data-model.md @@ -12,12 +12,12 @@ All methods to upsert or get records use strongly typed model classes. The prope > [!TIP] > -> - For an alternative to using attributes, see [defining your schema with a record definition](./schema-with-record-definition.md). -> - For an alternative to defining your own data model, see [using Vector Store abstractions without defining your own data model](./generic-data-model.md). +> - For an alternative to using attributes, see [Define your storage schema using a record definition](./schema-with-record-definition.md). +> - For an alternative to defining your own data model, see [Use Vector Store abstractions without defining your own data model](./dynamic-data-model.md). Here's an example of a model that's decorated with these attributes. -:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="Overview"::: +:::code language="csharp" source="./snippets/conceptual/defining-your-data-model.cs" id="Overview"::: ## Attributes @@ -25,7 +25,7 @@ Here's an example of a model that's decorated with these attributes. Use the attribute to indicate that your property is the key of the record. -:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreKeyAttribute"::: +:::code language="csharp" source="./snippets/conceptual/defining-your-data-model.cs" id="VectorStoreKeyAttribute"::: #### VectorStoreKeyAttribute parameters @@ -37,7 +37,7 @@ Use the attribute Use the attribute to indicate that your property contains general data that is not a key or a vector. -:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreDataAttribute"::: +:::code language="csharp" source="./snippets/conceptual/defining-your-data-model.cs" id="VectorStoreDataAttribute"::: #### VectorStoreDataAttribute parameters @@ -51,7 +51,7 @@ Use the attribut Use the attribute to indicate that your property contains a vector. -:::code language="csharp" source="./snippets/defining-your-data-model.cs" id="VectorStoreVectorAttribute1"::: +:::code language="csharp" source="./snippets/conceptual/defining-your-data-model.cs" id="VectorStoreVectorAttribute1"::: It's also possible to use on properties that don't have a vector type, for example, a property of type `string`. When a property is decorated in this way, you need to provide an instance to the vector store. When upserting the record, the text that's in the `string` property is automatically converted and stored as a vector in the database. (It's not possible to retrieve a vector using this mechanism.) @@ -61,7 +61,7 @@ public string DescriptionEmbedding { get; set; } ``` > [!TIP] -> For more information on how to use built-in embedding generation, see [Let the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). +> For more information on how to use built-in embedding generation, see [Let the Vector Store generate embeddings](embedding-generation.md#let-the-vector-store-generate-embeddings). #### VectorStoreVectorAttribute parameters diff --git a/docs/ai/vector-stores/dynamic-data-model.md b/docs/ai/vector-stores/dynamic-data-model.md index 7c7b5e68754bc..4a73663576f7d 100644 --- a/docs/ai/vector-stores/dynamic-data-model.md +++ b/docs/ai/vector-stores/dynamic-data-model.md @@ -23,4 +23,4 @@ A record definition can be used to provide the schema information. Unlike a data To use the `Dictionary` with a connector, specify it as your data model when you create a collection. Also provide a record definition. -:::code language="csharp" source="./snippets/dynamic-data-model.cs" id="Example1"::: +:::code language="csharp" source="./snippets/conceptual/dynamic-data-model.cs" id="Example1"::: diff --git a/docs/ai/vector-stores/embedding-generation.md b/docs/ai/vector-stores/embedding-generation.md index ea3704a73136a..0fc4cb45ac2db 100644 --- a/docs/ai/vector-stores/embedding-generation.md +++ b/docs/ai/vector-stores/embedding-generation.md @@ -15,7 +15,7 @@ You can configure an embedding generator on your vector store, allowing embeddin To enable generating vectors automatically on upsert, the vector property on your data model is defined as the source type, for example, string but still decorated with a . -:::code language="csharp" source="./snippets/embedding-generation.cs" id="LetTheVectorStoreGenerateEmbeddings"::: +:::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="LetTheVectorStoreGenerateEmbeddings"::: Before upsert, the `Embedding` property should contain the string from which a vector should be generated. The type of the vector stored in the database (for example, float32, float16, etc.) will be derived from the configured embedding generator. @@ -28,31 +28,31 @@ Embedding generators that implement the `Microsoft.Extensions.AI` abstractions a You can set a default embedding generator for the entire vector store. This generator will be used for all collections and properties unless overridden. - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnTheVectorStore"::: + :::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="OnTheVectorStore"::: - **On a Collection**: You can configure an embedding generator for a specific collection, overriding the store-level generator. - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnACollection"::: + :::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="OnACollection"::: - **On a Record Definition**: When defining properties programmatically using , you can specify an embedding generator for all properties. - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnARecordDefinition"::: + :::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="OnARecordDefinition"::: - **On a Vector Property Definition**: When defining properties programmatically, you can set an embedding generator directly on the property. - :::code language="csharp" source="./snippets/embedding-generation.cs" id="OnAVectorPropertyDefinition"::: + :::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="OnAVectorPropertyDefinition"::: ### Example usage The following example demonstrates how to use the embedding generator to automatically generate vectors during both upsert and search operations. This approach simplifies workflows by eliminating the need to precompute embeddings manually. -:::code language="csharp" source="./snippets/embedding-generation.cs" id="ExampleUsage"::: +:::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="ExampleUsage"::: ## Generate embeddings yourself @@ -62,11 +62,11 @@ For information on how to construct `Microsoft.Extensions.AI` embedding generato ### Generate embeddings on upsert with `IEmbeddingGenerator` -:::code language="csharp" source="./snippets/embedding-generation.cs" id="GenerateEmbeddingsOnUpsertWithIEmbedding"::: +:::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="GenerateEmbeddingsOnUpsertWithIEmbedding"::: ### Generate embeddings on search with `IEmbeddingGenerator` -:::code language="csharp" source="./snippets/embedding-generation.cs" id="GenerateEmbeddingsOnSearchWithIEmbedding"::: +:::code language="csharp" source="./snippets/conceptual/embedding-generation.cs" id="GenerateEmbeddingsOnSearchWithIEmbedding"::: ## Embedding dimensions diff --git a/docs/ai/vector-stores/how-to/build-vector-search-app.md b/docs/ai/vector-stores/how-to/build-vector-search-app.md index a81e879ec5eeb..c9e5172d1bcde 100644 --- a/docs/ai/vector-stores/how-to/build-vector-search-app.md +++ b/docs/ai/vector-stores/how-to/build-vector-search-app.md @@ -25,13 +25,13 @@ The app uses the and attributes, such as , influence how each property is handled when used in a vector store. The `Vector` property stores a generated embedding that represents the semantic meaning of the `Description` value for vector searches. + The attributes, such as , influence how each property is handled when used in a vector store. The `Vector` property stores a generated embedding that represents the semantic meaning of the `Description` value for vector searches. 1. In the `Program.cs` file, add the following code to create a data set that describes a collection of cloud services: @@ -148,38 +148,38 @@ Complete the following steps to create a .NET console app that can: 1. Create and configure an `IEmbeddingGenerator` implementation to send requests to an embedding AI model: - :::zone target="docs" pivot="azure-openai" + :::zone target="docs" pivot="azure-openai" - :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="EmbeddingGenerator"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="EmbeddingGenerator"::: - > [!NOTE] - > searches for authentication credentials from your local tooling. You'll need to assign the `Azure AI Developer` role to the account you used to sign in to Visual Studio or the Azure CLI. For more information, see [Authenticate to Foundry Tools with .NET](../azure-ai-services-authentication.md). + > [!NOTE] + > searches for authentication credentials from your local tooling. You'll need to assign the `Azure AI Developer` role to the account you used to sign in to Visual Studio or the Azure CLI. For more information, see [Authenticate to Foundry Tools with .NET](../../azure-ai-services-authentication.md). - :::zone-end + :::zone-end - :::zone target="docs" pivot="openai" + :::zone target="docs" pivot="openai" - :::code language="csharp" source="../snippets/chat-with-data/openai/program.cs" id="EmbeddingGenerator"::: + :::code language="csharp" source="../snippets/chat-with-data/openai/program.cs" id="EmbeddingGenerator"::: - :::zone-end + :::zone-end 1. Create and populate a vector store with the cloud service data. Use the `IEmbeddingGenerator` implementation to create and assign an embedding vector for each record in the cloud service data: - :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="VectorStore"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="VectorStore"::: The embeddings are numerical representations of the semantic meaning for each data record, which makes them compatible with vector search features. 1. Create an embedding for a search query and use it to perform a vector search on the vector store: - :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="Search"::: + :::code language="csharp" source="../snippets/chat-with-data/azure-openai/program.cs" id="Search"::: 1. Use the `dotnet run` command to run the app: - ```dotnetcli - dotnet run - ``` + ```dotnetcli + dotnet run + ``` - The app prints out the top result of the vector search, which is the cloud service that's most relevant to the original query. You can modify the query to try different search scenarios. + The app prints out the top result of the vector search, which is the cloud service that's most relevant to the original query. You can modify the query to try different search scenarios. :::zone target="docs" pivot="azure-openai" @@ -194,5 +194,5 @@ If you no longer need them, delete the Azure OpenAI resource and model deploymen ## Next steps -- [Quickstart - Chat with a local AI model](chat-local-model.md) -- [Generate images from text using AI](text-to-image.md) +- [Quickstart - Chat with a local AI model](../../quickstarts/chat-local-model.md) +- [Generate images from text using AI](../../quickstarts/text-to-image.md) diff --git a/docs/ai/vector-stores/how-to/build-your-own-connector.md b/docs/ai/vector-stores/how-to/build-your-own-connector.md deleted file mode 100644 index 7246e67b494f9..0000000000000 --- a/docs/ai/vector-stores/how-to/build-your-own-connector.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -title: How to build your own Vector Store connector -description: Describes how to build your own Vector Store connector connector -ms.topic: tutorial -ms.date: 02/28/2026 ---- -# How to build your own Vector Store connector - -This article provides guidance for building a Vector Store connector. This article can be used by database providers that want to build and maintain their own implementation, or for anyone that wants to build and maintain an unofficial connector for a database that lacks support. - -To contribute your connector to the Semantic Kernel code base: - -1. Create an issue in the [Semantic Kernel Github repository](https://github.com/microsoft/semantic-kernel/issues). -1. Review the [Semantic Kernel contribution guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md). - -## Overview - -Vector Store connectors are implementations of the [Vector Store abstraction](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions). Some of the decisions that were made when designing the Vector Store abstraction mean that a Vector Store connector requires certain features to provide users with a good experience. - -A key design decision is that the Vector Store abstraction takes a strongly typed approach to working with database records. -This means that takes a strongly typed record as input, while returns a strongly typed record. -The design uses C# generics to achieve the strong typing. This means that a connector has to be able to map from this data model to the storage model used by the underlying database. It also means that a connector might need to find out certain information about the record properties in order to know how to map each of these properties. For example, some vector databases (such as Chroma, Qdrant and Weaviate) require vectors to be stored in a specific structure and non-vectors in a different structure, or require record keys to be stored in a specific field. - -At the same time, the Vector Store abstraction also provides a generic data model that allows a developer to work with a database without needing to create a custom data model. - -It's important for connectors to support different types of model and provide developers with flexibility around how they use the connector. The following section deep dives into each of these requirements. - -## Requirements - -To be considered a full implementation of the Vector Store abstractions, the following set of requirements must be met. - -### 1. Implement the core abstract base clases and interfaces - -1.1 The three core abstract base classes and interfaces that need to be implemented are: - -- -- -- - - implements `IVectorSearchable`, so only two inheriting classes are required. Use the following naming convention: - -- {database type}VectorStore : VectorStore -- {database type}Collection : VectorStoreCollection\ - -For example: - -- MyDbVectorStore : VectorStore -- MyDbCollection : VectorStoreCollection\ - -The implementation should accept the name of the collection as a constructor parameter and each instance of it is therefore tied to a specific collection instance in the database. - -Here follows specific requirements for individual methods on these abstract base classes and interfaces. - -1.2 *`VectorStore.GetCollection`* implementations should not do any checks to verify whether a collection exists or not. -The method should simply construct a collection object and return it. The user can optionally use the `CollectionExistsAsync` method to check if the collection exists in cases where this is not known. Doing checks on each invocation of `GetCollection` might add unwanted overhead for users when they are working with a collection that they know exists. - -1.3 *`VectorStoreCollection.DeleteAsync`* that takes a single key as input should succeed if the record does not exist and for any other failures an exception should be thrown. -For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. - -1.4 *`VectorStoreCollection.DeleteAsync`* that takes multiple keys as input should succeed if any of the requested records do not exist and for any other failures an exception should be thrown. -For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. - -1.5 *`VectorStoreCollection.GetAsync`* that takes a single key as input should return null and not throw if a record is not found. For any other failures an exception should be thrown. -For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. - -1.6 *`VectorStoreCollection.GetAsync`* that takes multiple keys as input should return the subset of records that were found and not throw if any of the requested records were not found. For any other failures an exception should be thrown. -For requirements on the exception types to throw, see the [standard exceptions](#10-standard-exceptions) section. - -1.7 *`VectorStoreCollection.GetAsync`* implementations should respect the `IncludeVectors` option provided via `RecordRetrievalOptions` where possible. -Vectors are often most useful in the database itself, since that is where vector comparison happens during vector searches and downloading them can be costly due to their size. -There might be cases where the database doesn't support excluding vectors in which case returning them is acceptable. - -1.8 *`IVectorSearchable.SearchAsync`* implementations should also respect the `IncludeVectors` option provided via `VectorSearchOptions` where possible. - -1.9 *`IVectorSearchable.SearchAsync`* implementations should simulate the `Top` and `Skip` functionality requested via `VectorSearchOptions` if the database -does not support this natively. To simulate this behavior, the implementation should fetch a number of results equal to Top + Skip, and then skip the first Skip number of results -before returning the remaining results. - -1.10 *`IVectorSearchable.SearchAsync`* implementations should not require `VectorPropertyName` or `VectorProperty` to be specified if only one vector exists on the data model. -In this case that single vector should automatically become the search target. If no vector or multiple vectors exists on the data model, and no `VectorPropertyName` or `VectorProperty` is provided the search method should throw. - -When using `VectorPropertyName`, if a user does provide this value, the expected name should be the -property name from the data model and not any customized name that the property might be stored under -in the database. For example, imagine that the user has a data model property called `TextEmbedding` and they -decorated the property with a `JsonPropertyNameAttribute` that indicates that it should be serialized -as `text_embedding`. Assuming that the database is json based, it means that the property should be -stored in the database with the name `text_embedding`. - When specifying the `VectorPropertyName` option, the user should always provide -`TextEmbedding` as the value. This is to ensure that where different connectors are used with the -same data model, the user can always use the same property names, even though the storage name -of the property might be different. - -### 2. Support data model attributes - -The Vector Store abstraction allows a user to use attributes to decorate their data model to indicate the type of each property and to configure the type of indexing required -for each vector property. - -This information is typically required for - -- Mapping between a data model and the underlying database's storage model. -- Creating a collection or index. -- Vector search. - -If the user does not provide a , read this information from the data model attributes using reflection. If the user did provide a -, don't use the data model as the source of truth. - -> [!TIP] -> For a detailed list of all attributes and settings that need to be supported, see [Defining your data model](../defining-your-data-model.md). - -### 3. Support record definitions - -As mentioned in [Support data model attributes](#2-support-data-model-attributes), you need information about each property to build out a connector. This information can also -be supplied via a and if supplied, the connector should avoid trying to read this information from the data model or try and validate that the -data model matches the definition in any way. - -The user should be able to provide a to the implementation via options. - -> [!TIP] -> For a detailed list of all record definition settings that need to be supported, see [Defining your storage schema using a record definition](../schema-with-record-definition.md). - -### 4. Collection and index creation - -4.1 A user can optionally choose an index kind and distance function for each vector property. -These are specified via string based settings, but where available a connector should expect -the strings that are provided as string consts on `Microsoft.Extensions.VectorData.IndexKind` -and `Microsoft.Extensions.VectorData.DistanceFunction`. Where the connector requires -index kinds and distance functions that are not available on the above mentioned static classes -additional custom strings might be accepted. - -For example, the goal is for a user to be able to specify a standard distance function, like `DotProductSimilarity` -for any connector that supports this distance function, without needing to use different naming for each connector. - -```csharp -[VectorStoreVector(1536, DistanceFunction = DistanceFunction.DotProductSimilarity] -``` - -4.2 A user can optionally choose whether each data property should be indexed or full text indexed. -In some databases, all properties might already be filterable or full text searchable by default, however -in many databases, special indexing is required to achieve this. If special indexing is required -this also means that adding this indexing will most likely incur extra cost. -The and settings allow a user to control whether to enable -this additional indexing per property. - -### 5. Data model validation - -Every database doesn't support every data type. To improve the user experience it's important to validate -the data types of any record properties and to do so early, for example, when an -instance is constructed. This way the user will be notified of any potential failures before starting to use the database. - -### 6. Storage property naming - -The naming conventions used for properties in code doesn't always match the preferred naming -for matching fields in a database. -It's therefore valuable to support customized storage names for properties. -Some databases might support storage formats that already have their own mechanism for -specifying storage names, for example, when using JSON as the storage format you can use -a `JsonPropertyNameAttribute` to provide a custom name. - -6.1 Where the database has a storage format that supports its own mechanism for specifying storage -names, the connector should preferably use that mechanism. - -6.2 Where the database does not use a storage format that supports its own mechanism for specifying -storage names, the connector must support the settings from the data model -attributes or the . - -### 7. Mapper support - -Connectors should provide the ability to map between the user supplied data model and the -storage model that the database requires, but should also provide some flexibility in how -that mapping is done. Most connectors would typically need to support the following two -mappers. - -7.1 All connectors should come with a built in mapper that can map between the user supplied -data model and the storage model required by the underlying database. - -7.2. All connectors should have a built in mapper that works with the . -See [Support GenericDataModel](#8-support-genericdatamodel) for more information. - -### 8. Support GenericDataModel - -While it is very useful for users to be able to define their own data model, in some cases -it might not be desirable, for example, when the database schema is not known at coding time and driven -by configuration. - -To support this scenario, connectors should have out of the box support for the generic data -model supplied by the abstraction package: `Microsoft.Extensions.VectorData.VectorStoreGenericDataModel`. - -In practice this means that the connector must implement a special mapper -to support the generic data model. The connector should automatically use this mapper -if the user specified the generic data model as their data model. - -### 9. Divergent data model and database schema - -The only divergence required to be supported by connector implementations are -customizing storage property names for any properties. - -Any more complex divergence is not supported, since this causes additional -complexity for filtering. For example, if the user has a filter expression that references -the data model, but the underlying schema is different to the data model, the -filter expression cannot be used against the underlying schema. - -### 10. Standard Exceptions - -The database operation methods provided by the connector should throw a set of standard -exceptions so that users of the abstraction know what exceptions they need to handle, -instead of having to catch a different set for each provider. For example, if the underlying -database client throws a `MyDBClientException` when a call to the database fails, this -should be caught and wrapped in a , preferably preserving -the original exception as an inner exception. - -11.1 For failures relating to service call or database failures the connector should throw: -`Microsoft.Extensions.VectorData.VectorStoreOperationException` - -11.2 For mapping failures, the connector should throw: -`Microsoft.Extensions.VectorData.VectorStoreRecordMappingException` - -11.3 For cases where a certain setting or feature is not supported, for example, an unsupported index type, use: -`System.NotSupportedException`. - -11.4 In addition, use `System.ArgumentException`, `System.ArgumentNullException` for argument validation. - -### 11. Batching - -The abstract base class includes batching overloads for Get, Upsert, and Delete. Not all underlying database clients might have the same level of support for batching. - -The base batch method implementations on calls the abstract non-batch implementations in serial. If the database supports batching natively, these base batch implementations should be overridden and implemented using the native database support. - -## Recommended common patterns and practices - -- Keep and implementations sealed. It's recommended to use a decorator pattern to override a default vector store behavior. -- Always use options classes for optional settings with smart defaults. -- Keep required parameters on the main signature and move optional parameters to options. - -Here is an example of an constructor following this pattern. - -```csharp -public sealed class MyDBCollection : VectorStoreCollection -{ - public MyDBCollection(MyDBClient myDBClient, string collectionName, MyDBCollectionOptions? options = default) - { - } - - //... -} - -public class MyDBCollectionOptions : VectorStoreCollectionOptions -{ -} -``` - -## SDK changes - -See the following articles for a history of changes to the SDK and therefore implementation requirements: - -- [Vector Store Changes March 2025](../../../support/migration/vectorstore-march-2025.md) -- [Vector Store Changes April 2025](../../../support/migration/vectorstore-april-2025.md) -- [Vector Store Changes May 2025](../../../support/migration/vectorstore-may-2025.md) - -## Documentation - -To share the features and limitations of your implementation, you can contribute an article to this Microsoft Learn website. - -To create your article, create a pull request in the [.NET docs GitHub repository](https://github.com/dotnet/docs). - -Areas to cover: - -- An `Overview` with a standard table describing the main features of the connector. -- An optional `Limitations` section with any limitations for your connector. -- A `Getting started` section that describes how to import your NuGet and construct your and . -- A `Data mapping` section showing the connector's default data mapping mechanism to the database storage model, including any property renaming it might support. -- Information about additional features your connector supports. diff --git a/docs/ai/vector-stores/how-to/use-vector-stores.md b/docs/ai/vector-stores/how-to/use-vector-stores.md index 12d13ec718aa2..60f48e3164885 100644 --- a/docs/ai/vector-stores/how-to/use-vector-stores.md +++ b/docs/ai/vector-stores/how-to/use-vector-stores.md @@ -15,7 +15,7 @@ This article shows you how to use the key features of the library. ## Prerequisites - [.NET 9 SDK](https://dotnet.microsoft.com/download/dotnet/9.0) or later -- An understanding of [embeddings](../conceptual/embeddings.md) and [vector databases](../overview.md) +- An understanding of [embeddings](../../conceptual/embeddings.md) and [vector databases](../overview.md) ## Install the packages @@ -94,7 +94,7 @@ Use :::code language="csharp" source="../snippets/how-to/Program.cs" id="UpsertRecords"::: > [!IMPORTANT] -> In a real app, generate the embedding vectors using an `IEmbeddingGenerator` before storing the records. For a working example with real embeddings, see [Build a .NET AI vector search app](../quickstarts/build-vector-search-app.md). +> In a real app, generate the embedding vectors using an `IEmbeddingGenerator` before storing the records. For a working example with real embeddings, see [Build a .NET AI vector search app](build-vector-search-app.md). ## Get records @@ -172,6 +172,6 @@ Because all connectors implement the same . This value must match the size of vector that your chosen embedding generator produces. > [!TIP] -> For more information on how to annotate your data model and what additional options are available for each attribute, see [defining your data model](../../../concepts/vector-store-connectors/defining-your-data-model.md). +> For more information on how to annotate your data model and what additional options are available for each attribute, see [defining your data model](../defining-your-data-model.md). ## Read the paragraphs in the document @@ -66,7 +66,7 @@ Next, you add the code to read the Word document and find the text of each parag Add a new file to the project called `DocumentReader.cs` and add the following class to read the paragraphs from a document. -:::code language="csharp" source="./snippets/DataIngestion.cs" id="ReadTheParagraphsInTheDocument"::: +:::code language="csharp" source="./snippets/conceptual/DataIngestion.cs" id="ReadTheParagraphsInTheDocument"::: ## Generate embeddings and upload the data @@ -74,7 +74,7 @@ Next, you add a new class to generate embeddings and upload the paragraphs to Re Add a new file called `DataUploader.cs` with the following contents: -:::code language="csharp" source="./snippets/DataIngestion.cs" id="GenerateEmbeddingsAndUploadTheData"::: +:::code language="csharp" source="./snippets/conceptual/DataIngestion.cs" id="GenerateEmbeddingsAndUploadTheData"::: ## Put it all together @@ -82,11 +82,11 @@ Finally, you put together the different pieces. In this example, you use standar Add the following code to your `Program.cs` file to set up the container, register the Redis vector store, and register the embedding service. Replace the text embedding generation settings with your own values. -:::code language="csharp" source="./snippets/DataIngestion.cs" id="PutItAllTogether1"::: +:::code language="csharp" source="./snippets/conceptual/DataIngestion.cs" id="PutItAllTogether1"::: Lastly, add code to read the paragraphs from the Word document and call the data uploader to generate the embeddings and upload the paragraphs. -:::code language="csharp" source="./snippets/DataIngestion.cs" id="PutItAllTogether2"::: +:::code language="csharp" source="./snippets/conceptual/DataIngestion.cs" id="PutItAllTogether2"::: ## See your data in Redis diff --git a/docs/ai/vector-stores/hybrid-search.md b/docs/ai/vector-stores/hybrid-search.md index ec7a96ca085ac..3fe601149a8c8 100644 --- a/docs/ai/vector-stores/hybrid-search.md +++ b/docs/ai/vector-stores/hybrid-search.md @@ -23,7 +23,7 @@ Only connectors for databases that support vector plus keyword hybrid search imp The following example shows how to perform a hybrid search on a collection in a Qdrant database. -:::code language="csharp" source="./snippets/hybrid-search.cs" id="HybridSearch"::: +:::code language="csharp" source="./snippets/conceptual/hybrid-search.cs" id="HybridSearch"::: > [!TIP] > For more information on how to generate embeddings, see [embedding generation](./embedding-generation.md). @@ -55,13 +55,13 @@ When no `AdditionalProperty` is provided: - If the data model contains only one full-text search property, that property is used. - If the data model contains no full-text search property or multiple full-text search properties, the search method throws an exception. -:::code language="csharp" source="./snippets/hybrid-search.cs" id="VectorPropertyAndAdditionalProperty"::: +:::code language="csharp" source="./snippets/conceptual/hybrid-search.cs" id="VectorPropertyAndAdditionalProperty"::: ### `Top` and `Skip` The `Top` and `Skip` options allow you to limit the number of results to the top `n` results and to skip a number of results from the top of the result set. You can use `Top` and `Skip` to do paging if you want to retrieve a large number of results using separate calls. -:::code language="csharp" source="./snippets/hybrid-search.cs" id="TopAndSkip"::: +:::code language="csharp" source="./snippets/conceptual/hybrid-search.cs" id="TopAndSkip"::: The default value for `Skip` is 0. @@ -71,7 +71,7 @@ The `IncludeVectors` option allows you to specify whether you want to return vec The default value for `IncludeVectors` is `false`. -:::code language="csharp" source="./snippets/hybrid-search.cs" id="IncludeVectors"::: +:::code language="csharp" source="./snippets/conceptual/hybrid-search.cs" id="IncludeVectors"::: ### Filter @@ -85,8 +85,8 @@ For fields to be used for filtering, many vector stores require them to be index If you're creating a collection via the Vector Store abstractions and you want to enable filtering on a field, set the property to `true` when defining your data model or when creating your record definition. > [!TIP] -> For more information on how to set the property, see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddatafield-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md). +> For more information on how to set the property, see [VectorStoreDataAttribute parameters](./defining-your-data-model.md#vectorstoredataattribute-parameters) or [VectorStoreDataProperty configuration settings](./schema-with-record-definition.md). Filters are expressed using LINQ expressions based on the type of the data model. The set of LINQ expressions supported varies depending on the functionality supported by each database, but all databases support a broad base of common expressions, for example, `equals`, `not equals`, `and`, and `or`. -:::code language="csharp" source="./snippets/hybrid-search.cs" id="Filter"::: +:::code language="csharp" source="./snippets/conceptual/hybrid-search.cs" id="Filter"::: diff --git a/docs/ai/vector-stores/overview.md b/docs/ai/vector-stores/overview.md index b5cf8e9e34cc7..b54923352beef 100644 --- a/docs/ai/vector-stores/overview.md +++ b/docs/ai/vector-stores/overview.md @@ -8,7 +8,7 @@ ai-usage: ai-assisted # Vector databases for .NET AI apps -Vector databases store and manage vector [*embeddings*](embeddings.md). Embeddings are numeric representations of data that preserve semantic meaning. Words, documents, images, audio, and other types of data can all be vectorized. You can use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions. +Vector databases store and manage vector [*embeddings*](../conceptual/embeddings.md). Embeddings are numeric representations of data that preserve semantic meaning. Words, documents, images, audio, and other types of data can all be vectorized. You can use embeddings to help an AI model understand the meaning of inputs so that it can perform comparisons and transformations, such as summarizing text, finding contextually related data, or creating images from text descriptions. For example, you can use a vector database to: @@ -26,7 +26,7 @@ Some services, such as [Azure Cosmos DB for MongoDB vCore](/azure/cosmos-db/mong ## Vector search workflows with .NET and OpenAI -Vector databases and their search features are especially useful in [RAG pattern](rag.md) workflows with Azure OpenAI. This pattern allows you to augment or enhance your AI model with additional semantically rich knowledge of your data. A common AI workflow using vector databases includes the following steps: +Vector databases and their search features are especially useful in [RAG pattern](../conceptual/rag.md) workflows with Azure OpenAI. This pattern allows you to augment or enhance your AI model with additional semantically rich knowledge of your data. A common AI workflow using vector databases includes the following steps: 1. Create embeddings for your data using an OpenAI embedding model. 1. Store and index the embeddings in a vector database or search service. @@ -34,7 +34,7 @@ Vector databases and their search features are especially useful in [RAG pattern 1. Run a vector search across your data, comparing the user prompt embedding to the embeddings in your database. 1. Use a language model such as GPT-4 to assemble a user-friendly completion from the vector search results. -For a hands-on example of this flow, see the [Implement Azure OpenAI with RAG using vector search in a .NET app](../tutorials/tutorial-ai-vector-search.md) tutorial. +For a hands-on example of this flow, see the [Implement Azure OpenAI with RAG using vector search in a .NET app](tutorial-vector-search.md) tutorial. Other benefits of the RAG pattern include: @@ -75,7 +75,7 @@ All connectors implement the same class to indicate that your property is the key of the record. -:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreKeyProperty"::: +:::code language="csharp" source="./snippets/conceptual/schema-with-record-definition.cs" id="VectorStoreKeyProperty"::: #### VectorStoreKeyProperty configuration settings @@ -43,7 +43,7 @@ Use the class to i Use the class to indicate that your property contains general data that isn't a key or a vector. -:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreDataProperty"::: +:::code language="csharp" source="./snippets/conceptual/schema-with-record-definition.cs" id="VectorStoreDataProperty"::: #### VectorStoreDataProperty configuration settings @@ -59,7 +59,7 @@ Use the class to Use the class to indicate that your property contains a vector. -:::code language="csharp" source="./snippets/schema-with-record-definition.cs" id="VectorStoreVectorProperty"::: +:::code language="csharp" source="./snippets/conceptual/schema-with-record-definition.cs" id="VectorStoreVectorProperty"::: #### VectorStoreVectorProperty configuration settings diff --git a/docs/ai/vector-stores/vector-search.md b/docs/ai/vector-stores/vector-search.md index 7c9ee37927fdf..1b7370fc95797 100644 --- a/docs/ai/vector-stores/vector-search.md +++ b/docs/ai/vector-stores/vector-search.md @@ -9,7 +9,7 @@ ms.date: 02/28/2026 The library provides vector search capabilities as part of its Vector Store abstractions. These capabilities include filtering and many other options. > [!TIP] -> To see how you can search without generating embeddings yourself, see [Let the Vector Store generate embeddings](./embedding-generation.md#letting-the-vector-store-generate-embeddings). +> To see how you can search without generating embeddings yourself, see [Let the Vector Store generate embeddings](./embedding-generation.md#let-the-vector-store-generate-embeddings). ## Vector search @@ -22,7 +22,7 @@ Note that impleme Assuming you have a collection that already contains data, you can easily search it. Here is an example using Qdrant. -:::code language="csharp" source="./snippets/vector-search.cs" id="VectorSearch"::: +:::code language="csharp" source="./snippets/conceptual/vector-search.cs" id="VectorSearch"::: > [!TIP] > For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). @@ -41,14 +41,14 @@ The following options can be provided using the `VectorSearchOptions` c Use the `VectorProperty` option to specify the vector property to target during the search. If none is provided and the data model contains only one vector, that vector will be used. If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method throws an exceptoin. -:::code language="csharp" source="./snippets/vector-search.cs" id="VectorProperty"::: +:::code language="csharp" source="./snippets/conceptual/vector-search.cs" id="VectorProperty"::: ### `Top` and `Skip` The `Top` and `Skip` options allow you to limit the number of results to the top `n` results and to skip a number of results from the top of the resultset. You can use `Top` and `Skip` to do paging if you want to retrieve a large number of results using separate calls. -:::code language="csharp" source="./snippets/vector-search.cs" id="TopAndSkip"::: +:::code language="csharp" source="./snippets/conceptual/vector-search.cs" id="TopAndSkip"::: The default value for `Skip` is 0. @@ -59,7 +59,7 @@ If `false`, the vector properties on the returned model are left null. Using `fa The default value for `IncludeVectors` is `false`. -:::code language="csharp" source="./snippets/vector-search.cs" id="IncludeVectors"::: +:::code language="csharp" source="./snippets/conceptual/vector-search.cs" id="IncludeVectors"::: ### Filter @@ -77,4 +77,4 @@ If you're creating a collection via the Vector Store abstractions and you want t Filters are expressed using LINQ expressions based on the type of the data model. The set of LINQ expressions supported will vary depending on the functionality supported by each database, but all databases support a broad base of common expressions, for example, equals, not equals, `and`, and `or`. -:::code language="csharp" source="./snippets/vector-search.cs" id="Filter"::: +:::code language="csharp" source="./snippets/conceptual/vector-search.cs" id="Filter":::