A simple way to run llm locally and chat with your documents

TL;DR
In this tutorial, we'll be utilizing ollama, open webui, and AnythingLLM to run LLM locally and chat with your documents using RAG technology. You can install these tools now or follow along as I guide you through the process.

Installing ollama

Ollama is a local LLM solution that offers a variety of available open-source models. For this tutorial, we'll be using mistral, which currently provides the best balance between quality and performance. To install ollama, visit their website and download the version for your operating system. Once downloaded, install it by following the instructions provided. To confirm a successful installation, open your terminal and enter the command ollama --version. If you see an output displaying the ollama version number, then the installation was successful.

Next, we'll install mistral using ollama. In your terminal, run the command ollama run mistral. During the first use, ollama will download the model to your computer. After the installation is complete, you can chat with mistral. Try saying "hello" and press enter. Mistral will then begin generating text. To exit, type /bye into your terminal. Now you have a local LLM that you can use to ask questions. However, this LLM does not yet know anything about you. Let's give it access to your documents so that it can use them to answer questions.

Installing open webui

Before installing open webui, ensure that docker is already installed on your system. For open webui installation instructions, please refer to the open webui GitHub page. In a nutshell, you need to run the following command in your terminal:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

After running the command, go to your browser and visit http://localhost:3000/. You'll be prompted to create a user; feel free to create any username as this is a local program that won't be accessible to others.

Now you can select mistral from the list of models at the top, as we have already installed it via ollama. Next, on the left-hand side menu, click on "Documents" and import the documents you want your LLM to use. Go back to the chat and type #. Open webui will ask you to select which document you'd like to use for the conversation. You can choose all of them. Now, you can start using mistral with open webui.

Installing AnythingLLM

AnythingLLM functions similarly to open webui but includes one unique feature: if you go to your workspace settings -> chat settings -> chat mode and set it to "Query," AnythingLLM will notify you that it cannot find information in the documents if there is no relevant information regarding your query. By default, the chat mode is "Chat," meaning that if there is no information in the documents, AnythingLLM will use mistral's knowledge base to answer your question.

To install AnythingLLM, watch this YouTube video for instructions and then return here.

In conclusion

These three programs provide an intriguing way to use LLM on your local computer. The most significant advantage is that it's completely private, can work offline once the models are downloaded, and utilizes free licenses.