This project demonstrates how to integrate Box and Pinecone via a python script. This coincides with a blog post on Medium. Please see that posting for more insructuctions on how to use this solution.
You will need accounts for Box, Pinecone, and OpenAI to appropriate run this demo. You will also need an OAuth Box custom application created in the Box Developer Console.
Important
Once files leave Box, they are no longer under the protection of Box. It is incumbant upon the developer to ensure proper access rights. In this example, we are using OAuth2 to ensure the user can only access files they have access to, and on the Pinecone side, we are creating a namespace for each user based on their Box user ID, as well as adding their user ID to the metadata. This should ensure only the user can access their vectors.
-
Clone the repository:
git clone https://github.com/yourusername/box-pinecone-sample.git
cd box-pinecone-sample -
Create and activate a virtual environment:
python3 -m venv venv source venv/bin/activate -
Install dependencies:
pip install -r requirements.txt
-
Create config.py file.
cp sample_config.py config.py
-
Open the code in an editor of your choice. Update the credentials/Box folder ID with the information you create based on the above linked Medium Post. Save the file.
Important
DO NOT input 0 as the folder id. This will attempt to index your entire Box account. Not only is this not recommended, it will exceed rate limits, cost lots of money, and probably break.
-
To create embeddings:
python main.py
-
To answer queries about the created embeddings:
python query.py
New to this repository is a Pinecone Assistant demo. It relies on the same configuration as the other examples, so no additional setup is needed for the repo. You will need to make sure you have accessed the Pinecone Console and accepted the terms for using assistants.
The example will download the files from the prescribed folder in Box and upload the files to the assistant. You will then have access to a commandline chatbot to ask questions about the files you uploaded. Simply type quit or exit to end the chatbot, at which time the assistant and all the files uploaded will be deleted.
To run:
python assistant.py