Google Drive
Gist
Clipboard
Welcome to the Vector Database Course! In this course, you'll learn how to create, integrate, and query vector databases. We will create a simple application that will have a that is connected to OpenAI's GPT but augement it's knowledge with your own data via a vector database.
Potential use cases for vector databases include:
Throughout this course, you will acquire the following essential skills:
Embeddings are a form of feature engineering used predominantly in machine learning to convert text or other data into numerical vectors. These vectors capture semantic meaning and contextual relationships among words or items, making it easier for machine learning algorithms to understand text. Various algorithms like Word2Vec, GloVe, and BERT offer different methods for generating embeddings.
For instance, in the case of the word 'happy', it gets converted into a numerical array, say [0.12, 0.25, -0.47, ...], in a 300-dimensional space. This high-dimensionality allows the model to understand nuanced relationships, like how 'happy' is closer in meaning to 'joyful' than to 'sad'.
Embeddings are not just for word representation; they are pivotal in various NLP tasks such as sentiment analysis, text classification, and even machine translation.
A vector database is a specialized data storage and retrieval system designed to manage high-dimensional vectors like the ones generated by embeddings. Unlike traditional databases that are optimized for text or numerical data, vector databases are engineered to perform fast similarity searches within the high-dimensional space where these vectors reside.
To draw a parallel, if embeddings are the language that machine learning algorithms use to understand text, then a vector database is like a highly efficient library for this language. This 'library' can quickly identify which 'words' (vectors) are similar to a given 'word' (query vector), making it incredibly useful for tasks such as recommendation systems, similarity searches, and even enhancing chatbot intelligence.
For instance, after converting a text query into its corresponding vector using embeddings, you could search a vector database for the most similar vectors. These could then be used to generate chatbot responses, recommend products, or classify the text.
In order to follow along with this course, you will need to have the following:
Here are the technologies we will be using in this tutorial:
In the first step, we will create a Supabase project and enable it to store vector datatypes.
If you already have a Supabase account, you can skip this step. Otherwise, go to supabase.com and create an account.
Once you have created an account, you can create a new project.
Click on the "New project" button.
Select an organization and fill out the project details.
Finally click on the "Create New Project" button.
Now that you have created a project, you can proceed to enable it to store vector datatypes.
When you create a new Supabase project, with a PostgreSQL database you will not be able to store vector datatypes. To enable PostgreSQL to work with vector datatypes, you can follow these steps:
In the sidebar of your Supabase project, click on the SQL editor button.
Click on Quickstarts.
Select "LangChain" from the Quickstarts tiles.
Click on the "Run" button.
After running the query successfully, you will receive the following message: "Success. No rows returned."
Now that you have enabled PostgreSQL to work with vector datatypes, you can proceed to set up the boilerplate application.
In this step, we will set up the boilerplate application and run it. You can either use Codespaces or your local machine to set up the boilerplate. If want to use Codespaces follow the steps below otherwise skip to the next section (2.1.2).
GitHub Codespaces provides a complete, ready-to-use cloud-based dev environment in your browser. It saves you from the need for local setup, allowing you to concentrate on learning and building.
Create a GitHub account if you don't have one.
To create a new Codespace with the boilerplate, go to the
Vectordatabase-boilerplate repository.
When opening the Codespace, it will automatically install the dependencies.
git clone https://github.com/dacadeorg/vector-database-boilerplate.git
cd vector-database-boilerplate
npm install
Open the .env.example file and add your OpenAI API key, Supabase project API key, and Supabase reference ID.
1. OpenAI API Key
Your OPENAI_API_KEY key can be located in your OpenAI Dashboard.
2. Supabase Reference ID
To get your SUPABASE_REFERENCE_ID, click on the settings icon on the sidebar menu of your Supabase project, copy the Reference ID, and assign it to SUPABASE_REFERENCE_ID.
SUPABASE_REFERENCE_ID= -
3. Supabase Project API Key
To get SUPABASE_PROJECT_API_KEY, click on API, click copy, and assign it to your SUPABASE_PROJECT_API_KEY in your .env.example file. Copy only the key with anon and public labels.
4. Rename .env.example to .env
Rename .env.example to .env.
npm run dev
Access the application by navigating to http://localhost:3000. The boilerplate application should now be live.
The boilerplate application is a simple chatbot that uses OpenAI's GPT-3 model to generate responses. It doesn't use any data from the vector database yet. To test it, type a message in the input field and press enter. The chatbot will respond with a message generated by the unaugmented GPT-3 model. We will now augment the chatbot with our own data.
If you ask it what the Javascript toolkit "bun" is for example it will not know the answer. Since it's knowledge cut off date is 2022-01-01.
Now you can enter content that you want to vectorize and store in your vector database. For example, content of the bun documentation.
After you have entered the content, you can click on the "Vectorize" button. This will vectorize the content and store it in your vector database. Then you can ask the chatbot about the bun toolkit again. This time, it will know the answer because it has been augmented with your data.
In this step, we will add content to our vector database via the terminal. For most of your projects you will probably not have a UI to add content to your vector database, here we will use a script.
We will use the uploadEmbedding.js file from the scripts folder. This file will read the content from the document.txt file from the scripts/content folder and upload it to the vector database.
To run the script, you can use the following command:
node ./scripts/uploadEmbedding.js
Don't worry about this warning: "No storage option exists to persist the session, which may result in unexpected behavior when using auth. If you want to set persistSession to true, please provide a storage option, or you may set persistSession to false to disable this warning." Just wait until you see "Uploaded" logged in the terminal.
After this is done you are ready to test the chatbot again. This time you can ask it about the bun toolkit and it will know the answer because it has been augmented with your data.
In this chapter we will explore the code of the boilerplate application. We will especially focus on the pages/api/openai.js and pages/api/embed.js files.
pages/api/openai.jsThis is the handler function of the /api/openai endpoint that is responsible for accepting request from client and respond back with an actual answer from openai model. Here we will add our own data to the OpenAI model.
On line 34 we initialize the OpenAI model with an object parameter with three items: temperature, openAIApiKey, stream.
const model = new OpenAI({
temperature: 1,
openAIApiKey: process.env.OPENAI_API_KEY,
streaming: false,
});
On line 40 we initialize the vector store with the Supabase client and the table where we stored our vectors. This is responsible for giving us access to the vector database.
const vectorStore = await SupabaseVectorStore.fromExistingIndex(
new OpenAIEmbeddings({ openAIApiKey: process.env.OPENAI_API_KEY }),
{
client: supabase,
tableName: "documents",
queryName: "match_documents",
}
);
ConversationalRetrievalQAChain is a type of chain in LangChain that is specifically designed for conversational question answering. It works by combining a retrieval component with a large language model (LLM).
Chains in LangChain are reusable components that can be linked together to perform complex tasks. Chains can be formed using various types of components, such as prompt templates, LLMs, arbitrary functions, and other chains.
In a ConversationalRetrievalQAChain, the retrieval component searches for relevant documents from a knowledge base based on the user's question. The LLM then uses these documents to generate a comprehensive and informative answer.
On line 49 we use the fromLLM method to create a ConversationalRetrievalQAChain from an LLM and a retriever. The LLM is the OpenAI model we initialized earlier. The retriever is the vector store we initialized earlier.
const chain = ConversationalRetrievalQAChain.fromLLM(
model,
vectorStore.asRetriever()
);
Finally on line 54, we make the final call to get the response.
We need to pass an object that contains the question and the chat history. The chat history is not required to have elements, however if you pass the chat history, the model will be aware of the history. The format should be like this: [ { role: "assistant', content: "" }, { role: "user', content: "" } ];
const answer = await chain.call({ question: question, chat_history: [] });
return res.status(200).json({ data: answer });
Now we passed the question to the chain which will use the vector store and the OpenAI model to generate an answer. The answer will be returned to the client.
pages/api/embed.js// Import necessary libraries and modules
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import {
MarkdownTextSplitter,
RecursiveCharacterTextSplitter,
} from "langchain/text_splitter";
import { v4 as uuidv4 } from "uuid";
import { SupabaseVectorStore } from "langchain/vectorstores/supabase";
import { supabase } from "@/utils/supabase";
// Define an asynchronous function to handle incoming requests
export default async function handler(req, res) {
try {
// Extract the 'payload' field from the request body
const content = req.body.payload;
// Create a content text splitter instance
const splitter = new RecursiveCharacterTextSplitter({
chunkSize: 150,
chunkOverlap: 2,
});
const document = await splitter.createDocuments([content]);
// Generate and store embeddings for the text documents
const response = await generateAndStoreEmbedding(document, {
chat_id: uuidv4(),
});
// Send a JSON response with the result
res.status(200).json({ result: response });
} catch (error) {
// Handle any errors and send a 500 Internal Server Error response
res.status(500).json({ error: JSON.stringify(error) });
}
}
// Define an asynchronous function to generate and store embeddings
const generateAndStoreEmbedding = async (docs, fields) => {
// Modify the metadata of each chunked document by adding specified fields
const splitedDocs = docs.map((e) => ({
...e,
metadata: { ...e.metadata, ...fields },
}));
// Generate and store embeddings using SupabaseVectorStore and OpenAIEmbeddings
return await SupabaseVectorStore.fromDocuments(
splitedDocs,
new OpenAIEmbeddings({
openAIApiKey: process.env.OPENAI_API_KEY,
}),
{
client: supabase,
tableName: "documents",
}
);
};
This file is responsible for handling the request that comes from the client and
transform the content into vectors and store them in our vector database.
This file contains two functions, the handler function and the generateAndStoreEmbedding function.
The handler function is the main function that is responsible for handling the request that comes from the client and respond back with the result.
On line 18, we use RecursiveCharacterTextSplitter to create a text splitter instance.
const splitter = new RecursiveCharacterTextSplitter({
chunkSize: 150,
chunkOverlap: 2,
});
RecursiveCharacterTextSplitter is responsible for splitting the content into chunks. It takes two parameters: chunkSize and chunkOverlap. The chunkSize parameter specifies the maximum number of characters in each chunk. The chunkOverlap parameter specifies the number of characters that each chunk should overlap with the previous chunk. Overlap in text splitting allows adjacent chunks to share common information, which can improve performance and accuracy of text processing tasks.
On line 26, we use generateAndStoreEmbedding from the Langchain library to generate and store embeddings for the text documents.
const response = await generateAndStoreEmbedding(document, {
chat_id: uuidv4(),
});
generateAndStoreEmbedding takes two parameters: The text passage to generate an embedding for and the vector store to store the embedding in.
pages/api/embed.js This is endpoint is responsible for two major role transforming our conent into vectors and uploading them to our vector database.
In this file we have two function first function that handle the content that came from the request payload and second function that is invoked inside the handler function to generate vectors and store vector them.
RecursiveCharacterTextSplitter(): will help us split the content into chunks, and those chunks will be transformed into vectors, that are going to be stored in our vector database, and each chunk is called a document.
More information can be found Here.
SupabaseVectorStore.fromDocuments: This is an instance that merges your Supabase client with the logic of Langchain and the OpenAI model that converts documents into vectors and uploads them to vector database.
SplittedDocs: This function receive two important parament, first splittedDocs or chunks.
OpenAIEmbeddings: This will initialize openai instance with text-embedding-ada-002 model and that will transform our content into text as vectors and the out gets to be uploaded to the vector database.
This is how text-embedding-ada-002 is working.
Lastly we pass an object that contains supabase client with the table that will store our vectors.
await SupabaseVectorStore.fromDocuments(
splittedDocs,
new OpenAIEmbeddings({
openAIApiKey: process.env.OPENAI_API_KEY,
}),
{
client: supabase,
tableName: "documents",
}
);
In this course, you learned how to create, integrate, and query vector databases. You learned how to create a vector database using Supabase, how to integrate it with OpenAI's GPT-3 model, and how to query it to retrieve vectors. You also learned how to use vector databases to augment chatbots with your own data.
-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
Subscribe