Vector databases have become an essential architecture component in the era of artificial intelligence, particularly as AI applications involving large language models, generative AI, and semantic search continue to rapidly increase. With a human brain analogy Vector database, artificial intelligence is broken into:
- specialised memory engrams and encodings (Vector Embeddings), - cognitive functions (as leveraged by Large Language Models), - memory recall (embraced by Vector Databases), and - neurological pathways (Data Pipelines).What is a Vector?
A vector in the simplest of terms is a mathematical entity that represents both a magnitude (how much) and a direction (where). It is often represented as an arrow, where the length of the arrow shows the magnitude and the direction of the arrow shows the direction.
What is a Vector Database?
Vector databases, also known as vector stores, are specialised databases designed for storage, retrieval, and querying of high-dimensional vector data.
What is high dimensional vector data?
A vector embedding at its core is the ability to represent a piece of data as a mathematical equation in n-dimensional space so that similar data points cluster together. A single embedding is like a neuron in our brain. And as a single neuron doesn’t make a brain and a single embedding doesn’t make an AI system. The more embeddings, and the more relationships those embeddings have give the ability to have more and more complex cognitive abilities.
In the context of machine learning, high-dimensional vectors are vectors where each element represents a specific feature or attribute of the data. Data points are represented as vectors in a space with a large number of dimensions. The dimensionality of a high-dimensional vector is characterised by the number of dimensions (or elements) it contains. As the number of dimensions increases, the size of the vector space grows exponentially, which can present challenges in terms of storage, computational complexity, data sparsity and efficient searching.for example, in text or natural language processing context, each element of the vector represents a feature associated with a word or a document, allowing for computations and comparisons based on semantic similarity. Similarly, in the image context, the dimensionality of the vector would be equal to the total number of pixels or features in the image.
scale of dimensionality
In some cases, a vector with thousands of dimensions might be considered high-dimensional, while in other cases, it might refer to vectors with millions or billions of dimensions. Now, we can correlate the number of features or elements vector database stores inside it.
hypernear - scalable and fast vector database
efficient similarity search, specialised indexing structures & algorithms, graph-based structures, hashing techniques for efficient indexing and search
highlights
hypernear with LLMs and RAG
Retrieval-augmented generation (RAG) for large language models (LLMs) aims to improve prediction quality by using hypernear at inference time to augment a richer prompt that includes some combination of context, history, and recent and relevant knowledge. Combining the flexibility of parametric-only approach with the performance of nonparametric memory. This combination employs a form of late fusion to integrate knowledge from all retrieved documents.
3 steps to leverage hypernear
Advantage of hypernear
- Fast similarity search: leverage distance or similarity measures, quickly identify similar vectors enabling efficient retrieval of relevant items or recommendations.
- Scalability: hypernear is designed to handle large-scale vector datasets and high dimensional vector data, allowing for efficient storage, indexing, and retrieval of millions or even billions of vectors.
- Versatile: supports numerical vectors, embeddings, feature vectors, or textual representations, to support a wide range of applications.
- Integration with AI ecosystem: hypernear is providing seamless integration with popular and leading AI ecosystem frameworks and platforms.
The Process
So, how does it work?
As Meta calls it; answering with both closed and open book
take user prompt
Rather than passing the uuser input directly to the generator, send it to the vector search solution to find the relevant information
augment prompt
Once it has that relevant information it will construct a “prompt” that contains the question the user asked, the information received from the vector search
search & generate
augmented prompt is creted to make the LLM respond how you’d like. Once this is done, all of that information is sent to the LLM.