The need to efficiently search and retrieve documents based on their semantic content is more important than ever—especially when building Generative AI systems that implement the Retrieval Augmented Generation (RAG) design pattern. One robust approach to achieve this is by using vector representations of text data. These vectors, generated using advanced embedding models like Azure OpenAI’s text-embedding-ada-002, allow for similarity search based on vector distances, resulting in highly relevant results.

In this article, we discuss what a vector search engine is, why you might choose this approach for your .NET applications, and how to integrate a lightweight, in-memory vector database using the Build5Nines.SharpVector library. We also cover how to combine SharpVector with Azure OpenAI embeddings to enhance the quality of your vector representations.

What is a Vector Search Engine?

A vector search engine is a specialized system designed to index and retrieve documents based on their vectorized representations. Rather than relying solely on keyword matching, a vector search engine converts text data into high-dimensional numerical vectors. These vectors can then be compared using metrics such as cosine similarity or Euclidean distance to find documents that are semantically similar.

Key operations include:

• Vectorization: Transforming text into vectors using methods like word embeddings or transformer-based models (e.g., Azure OpenAI text-embedding-ada-002).
• Similarity Search: Retrieving documents whose vector representations are closest to a given query vector.

Why use a Vector Search Engine?

There are several compelling reasons to adopt a vector search approach:

• Semantic Search: By capturing the underlying meaning of the text, vector search engines deliver more accurate and contextually relevant results compared to traditional keyword-based methods.
• Flexibility: They can process various types of data, as long as it can be transformed into vectors.
• Performance: Designed for high-dimensional operations, these engines provide efficient and scalable solutions for similarity search tasks.

Integrating Azure OpenAI for embedding generation further enhances your search capabilities by providing state-of-the-art semantic representations.

Why use Azure OpenAI for Embeddings?

While the Build5Nines.SharpVector library does have built-in embedding generation, the built-in capability is really simplistic compared to using a model like OpenAI’s text-embedding-ada-002. By integrating SharpVector with an OpenAI embeddings model the accuracy and contextual understanding of the embeddings is greatly improved. This will result in the text vector similarity search to be greatly improved as well. Essentially, OpenAI’s text embedding models, like text-embedding-ada-002, generate more meaningful vector representations, leading to better search results.

Benefits of Azure OpenAI Embeddings:

✅ Improved Search Accuracy – Better vector representations result in more relevant search results.
✅ Pre-trained on Vast Data – Azure OpenAI’s models have been trained on extensive datasets, making them more robust.
✅ Optimized for Performance – OpenAI’s embeddings are optimized for fast retrieval and indexing.

Getting Started with SharpVector and Azure OpenAI

The Build5Nines.SharpVector library is an open-source, lightweight solution for creating an in-memory vector database in .NET applications. When paired with Azure OpenAI embeddings, you can generate high-quality vectors that improve the accuracy of your similarity searches.

The Build5Nines.SharpVector project has two Nuget packages currently:

• Build5Nines.SharpVector: This is the core SharpVector library with the main in-memory, vector database and similarity search functionality. This includes the basic, local vector generation too.
• Build5Nines.SharpVector.OpenAI: This library extends the core SharpVector library with an in-memory, vector database that uses the OpenAI.Embeddings.EmbeddingClient to connect to an OpenAI endpoint for generating embeddings.

Build5Nines.SharpVector.OpenAI NuGet Package Installation

When using SharpVector with the Build5Nines.SharpVector.OpenAI library, you only need to add a reference to the Build5Nines.SharpVetor.OpenAI library to your project. This will automatically include the additional dependencies it has to the Build5Nines.SharpVector and OpenAI Nuget packages as well.

To add a reference for `SharpVector.OpenAI` to your project, install it via NuGet.

Using the .NET CLI:

dotnet add package Build5Nines.SharpVector.OpenAI

Using the Package Manager Console in Visual Studio:

Install-Package Build5Nines.SharpVector.OpenAI

Setting up OpenAI Embeddings in SharpVector

To leverage Azure OpenAI embeddings with SharpVector, we need to set up an embedding client that can generate high-quality vector representations of text. While SharpVector provides built-in text vectorization, using Azure OpenAI’s text-embedding-ada-002 significantly improves the semantic accuracy of the embeddings, leading to more precise vector similarity searches.

Using OpenAI Embeddings Model

If you’re using OpenAI’s API directly, you can set up an embedding client using the OpenAI API key. This approach allows you to leverage OpenAI’s text-embedding-ada-002 model to generate high-quality vector embeddings. These embeddings improve semantic search accuracy by converting text into a numerical representation that captures context and meaning.

using OpenAI;

// Setup the OpenAI connection information
var openAIKey = "xxxxxxxxxx";
var modelName = "text-embedding-ada-002";

// Create OpenAI Client using the API key
var openAIClient = new OpenAIClient(openAIKey);

// Create Embedding Client for the OpenAI Embeddings Model
var embeddingClient = openAIClient.GetEmbeddingClient(modelName);

Using Azure OpenAI Embeddings Model

For applications running in Azure, you can use Azure OpenAI Service to generate embeddings instead of relying on OpenAI’s public API. This approach provides enterprise-grade security, compliance, and regional availability within your Azure environment.

using Azure;
using Azure.AI.OpenAI;

// Setup the Azure OpenAI connection information
var openAIUri = new Uri("https://{name}.openai.azure.com/");
var openAIKey = "xxxxxxxxxx";
var modelName = "text-embedding-ada-002";

// Create Azure OpenAI Client using the API key
var openAIClient = new AzureOpenAIClient(openAIUri, new AzureKeyCredential(openAIKey));

// Create Embedding Client for the Azure OpenAI Embeddings Model
var embeddingClient = openAIClient.GetEmbeddingClient(modelName);

Initialize the Vector Database using Embedding Client

Now that the OpenAI Embeddings Client has been setup, Build5Nines.SharpVector.OpenAI provides the BasicOpenAIMemoryVectorDatabase class, which is an in-memory vector database that utilizes the OpenAI embeddings model for generating the text vector embeddings.

using Build5Nines.SharpVector.OpenAI;

// Initialize the in-memory vector database with OpenAI Embeddings Client
var vectorDatabase = new BasicOpenAIMemoryVectorDatabase(embeddingClient);

Adding Text Data to the Vector Database

To perform searches, text documents needs to first be vectorized and stored in the SharpVector database. This can be done using either the .AddText or .AddTextAsync methods. These methods accept two parameters, first the text document and second the text metadata to store alongside the document.

vectorDatabase.AddText(documentText, metadataText);
async vectorDatabase.AddTextAsync(documentText, metadataText);

The metadata text stored alongside the document is not vectorized. It is stored with the document and retrieved along with the document text when performing a vector search. This enables the storage and association of any metadata to the document that’s needed. For example, the name of the document or even a JSON string could be the metadata.

Here’s an example of adding a couple text descriptions of movies with the name of the movie as the metadata:

await vectorDatabase.AddTextAsync("The Matrix is a sci-fi movie about AI and reality.", "The Matrix");
await vectorDatabase.AddTextAsync("Inception is a movie that explores dreams within dreams.", "Inception");
await vectorDatabase.AddTextAsync("Interstellar is about space exploration and time relativity.", "Interstellar");

Perform Vector Similarity Search

Once the text documents / data has been stored in the vector database, then the .Search or .SearchAsync method can be used to perform a vector similarity search and query the SharpVector database.

var query = "space and time";
var results = await vectorDatabase.SearchAsync(query);

The Search methods also support the following additional parameters to help with searching the vector database:

var results = await vectorDatabase.SearchAsync(queryText,
  threshold: 0.001f // 0.2f - Cosine Similarity
  pageIndex: 0, // page index of search results (default: 0)
  pageCount: 10 // Number of results per page to return (default: no limit)
);

The supported parameters of the .Search and .SearchAsync methods are as follows:

• queryText: The text query to search within the vector database
• threshold: The similarity threshold to use for searching matches using Cosine Similarity method.
• pageIndex: The page index of search results to return. This defaults to 0 for the first page.
• pageCount: The number of results to return per page. This defaults to returning all results.

Reading Search Results

The results of the search are returned as a VectorTextResult object that contains the returns page of search results from the vector Cosine Similarity search. The VectorTextResult object has the following properties available:

• .IsEmpty: Returns true / false whether the search yielded results. True if empty / no results.
• .TotalCount: Returns the total count of search results found.
• .PageIndex: Returns the page index of the page of results returned.
• .TotalPages: Returns the total count of pages of search results available.
• .Texts: Returns the current page of results as VectorResultItem instances.

The .Texts property returns the current page of text results that match the vector search performed:

var texts = results.Texts;

Each VectorTextResultItem contains the following properties to access the details of the text item returned:

• .Text: This is the text of the document that was vectorized.
• .Metadata: This is the metadata stored alongside the document in the vector database.
• .VectorComparison: This is the float that represents the vector comparison of how closely the document matches the search query when the Cosine Similarity was performed.

Here’s an example of looping through the search results of a vector database query and displaying the results using a console app:

if (searchResults.HasResults)
{
    Console.WriteLine("Similar Movies Found:");

    foreach (var item in searchResults.Texts)
    {
        Console.WriteLine($"Movie: {item.Metadata}");
        Console.WriteLine($"Vector Score: {item.VectorComparison}");
        Console.WriteLine($"Description: {item.Text}");
        Console.WriteLine();
    }
}
else
{
    Console.WriteLine("No matching movies found.");
}

Optimizing using Text Chunking for Better Search Optimization

For large documents, the Build5Nines.SharpVector library also offers a TextDataLoader class that supports text chunking. This process splits a document into smaller segments (by paragraphs, sentences, or fixed lengths) before adding them to the database, ensuring optimal performance for lengthy texts.

var loader = new Build5Nines.SharpVector.Data.TextDataLoader<int, string>(vectorDatabase);
loader.AddDocument(documentContent, new TextChunkingOptions<string>
{
    Method = TextChunkingMethod.Paragraph,
    RetrieveMetadata = (chunk) => "{ \"filename\": \"" + filename + "\" }"
});

When adding text to the vector database, you can attach metadata (such as a JSON string with file details, URLs, or additional context) to each document. This metadata is retrieved alongside the search results, providing additional context for your application.

Conclusion

Integrating Azure OpenAI embeddings with Build5Nines.SharpVector provides a powerful solution for semantic search and vector similarity search in .NET applications. By leveraging OpenAI’s text-embedding-ada-002 or other models, high-quality vector representations can be generated that improves the accuracy and efficiency of text-based searches.

With Build5Nines.SharpVector, you get a lightweight, in-memory vector database that seamlessly integrates with OpenAI embeddings, making it easy to store, search, and retrieve text based on semantic meaning rather than just keyword matching. The support for text chunking further optimizes searches by breaking down large documents into smaller, more manageable pieces for indexing and retrieval.

Original Article Source: Enhanced In-Memory Text Vector Search in .NET with SharpVector and OpenAI Embeddings written by Chris Pietschmann (If you're reading this somewhere other than Build5Nines.com, it was republished without permission.)