Ollama

The guide was put together in collaboration with members of the Quivr Discord, Using Quivr fully locally thread. That is a good place to discuss it. https://discord.com/invite/HUpRgp2HG8

​

Ollama

Ollama is a tool that allows you to run LLMs locally. We are using it to run Llama2, MistralAI and others locally.

​

Install Ollama

Install Ollama from their website.

Then run the following command to run Ollama in the background:

ollama run llama2

​

Run Migrations

mv supabase/migrations/local_20240107152745_ollama.sql supabase/migrations/20240107152745_ollama.sql
supabase db reset

​

Add Ollama Model to Quivr

Now that you have your model running locally, you need to add it to Quivr.

In order to allow the user to choose between the Ollama, we need to add a new model to the Quivr backend.

Go to supabase and in the table user_settings either add by default or to your user the following value to the models column:

[
  "ollama/llama2",
  "ollama/mistral"
]

This will add the Ollama model to the list of models that the user can choose from.

By adding this as default, it means that all new users will have this model by default. If you want to add it to your user only, you can add it to the models column in the user_settings table. In order for the change to take effect if you put as default your need to drop the entire table with the following command:

DELETE TABLE user_settings;