Getting Started with LLaMA

August 2023 Update: If you're new to Llama and local LLMs, this post is for you. This guide has been updated with the latest information, including the simplest ways to get started. You can skip the sections on manually installing with text generation web UI, which was part of the old original guide from six months ago. These sections are everything below the Old Guide header.

If you're looking for the link to the new Discord server, it's here: https://discord.gg/Y8H8uUtxc3

If you're looking for the subreddit list of models, go to the wiki: https://www.reddit.com/r/LocalLLaMA/wiki/models.

LLaMA FAQ

Q: What is r/LocalLLaMA about?

LocalLLaMA is a subreddit to discuss about Llama, the family of large language models created by Meta AI. It was created to foster a community around Llama similar to communities dedicated to open source like Stable Diffusion. Discussion of other local LLMs is welcome.

To learn more about Llama, read the Wikipedia page.

Q: Is Llama like ChatGPT?

A: The foundational Llama models are not fine-tuned for dialogue or question answering like ChatGPT. They should be prompted so that the expected answer is the natural continuation of the prompt. Fine-tuned Llama models have scored high on benchmarks and can resemble GPT-3.5-Turbo. Llama models are not yet GPT-4 quality.

Q: How to get started? Will this run on my [insert computer specs here?]

A: To get started, keep reading. You can very likely run Llama based models on your hardware even if it's not good.

System Requirements

8-bit Model Requirements for GPU inference

^{*System RAM, not VRAM, required to load the model, in addition to having enough VRAM. Not required to run the model. You can use swap space if you do not have enough RAM.}

4-bit Model Requirements for GPU inference

^{*System RAM, not VRAM, required to load the model, in addition to having enough VRAM. Not required to run the model. You can use swap space if you do not have enough RAM.}

llama.cpp Requirements for CPU inference

^{As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them. At the moment, memory and disk requirements are the same.}

Projects and Installation

Since the unveil of LLaMA several months ago, the tools available for use have become better documented and simpler to use. There are three main projects that this community uses: text generation web UI, llama.cpp, and koboldcpp. This section contains information on each one.

text generation web UI

^{A gradio web UI for running Large Language Models like LLaMA, llama.cpp, GPT-J, Pythia, OPT, and GALACTICA.}

The developer of the project has created extensive documentation for installation and other information, and the old guide for manual installation is no longer necessary. To get started, all you have to do is download the one-click installer for the OS of your choice then download a model. For the full documentation, check here.

llama.cpp

^{Inference of LLaMA model in pure C/C}

This is the preferred option for CPU inference. For building on Linux or macOS, view the repository for usage. If you're on Windows, you can download the latest release from the releases page and immediately start using.

For all of the other info on using, the documentation here explains the different options and interaction.

koboldcpp

^{A self contained distributable from Concedo that exposes llama.cpp function bindings, allowing it to be used via a simulated Kobold API endpoint. You get llama.cpp with a fancy UI, persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer.}

The koboldcpp wiki explains everything you need to know to get started.

Models

To find known good models to download, including the base LLaMA and Llama 2 models, visit this subreddit's wiki: https://www.reddit.com/r/LocalLLaMA/wiki/models. You can also search Hugging Face.

Although there have been several fine-tuned models to be released, not all have the same quality. For the best first time experience, it's recommended to start with the official Llama 2 Chat models released by Meta AI or Vicuna v1.5 from LMSYS. They are the most similar to ChatGPT.

If you need a locally run model for coding, use Code Llama or a fine-tuned derivative of it. 7B, 13B, and 34B Code Llama models exist. If you're looking for visual instruction, then use LLaVA or InstructBLIP with Vicuna.

Other Info and FAQ

Q: Do these models provide refusals like ChatGPT?

A: This depends on the model. Some, like the Vicuna models trained on ShareGPT data, inherits refusals from ChatGPT for certain queries. Other models never provide refusals at all. If this is important for your use case, you can experiment with different choices to find your preferred option.

Q: How can I train a LoRA for a specific task or purpose?

A: Read this guide. If you have any questions after reading all of that, then you can ask in this subreddit.

Q: Where can I keep up with the latest news for local LLMS?

A: This subreddit! While the name of this subreddit is r/LocalLLaMA and focuses on LLaMA, discussion of all local LLMs is allowed and encouraged. You can be sure that the latest news and resources will be shared here.

​

Old Guide

Everything below this point is the old guide which was the original post, and you can skip everything here. Most of it has been deleted now, including the tips, resources, and LoRA tutorial, but the manual steps for the web UI will remain as a reference for anyone who wants it or anyone curious about how the process used to be. This old guide below and its information will no longer be updated.

Installing Windows Subsystem for Linux (WSL)

WSL installation is optional. If you do not want to install this, you can skip over to the Windows specific instructions below for 8-bit or 4-bit. This section requires an NVIDIA GPU.

On Windows, you may receive better performance when using WSL. To install WSL using the instructions below, first ensure you are running at least Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11. To check for this, type info in the search box on your taskbar and then select System Information. Alternatively, hit Windows R, type msinfo32 into the "Open" field, and then hit enter. Look at "Version" to see what version you are running.

Instructions:

1. Open Powershell in administrator mode
2. Enter the following command then restart your machine: wsl --install

This command will enable WSL, download and install the lastest Linux Kernel, use WSL2 as default, and download and install the Ubuntu Linux distribution.

1. After restart, Windows will finish installing Ubuntu. You'll be asked to create a username and password for Ubuntu. It has no bearing on your Windows username.

2. Windows will not automatically update or upgrade Ubuntu. Update and upgrade your packages by running the following command in the Ubuntu terminal (search for Ubuntu in the Start menu or taskbar and open the app): sudo apt update && sudo apt upgrade

3. You can now continue by following the Linux setup instructions for LLaMA. Check the necessary troubleshooting info below to resolve errors. If you plan on using 4-bit LLaMA with WSL, you will need to install the WSL-Ubuntu CUDA toolkit using the instructions below.

Extra tips:

To install conda, run the following inside the Ubuntu environment:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

To find the name of a WSL distribution and uninstall it (afterward, you can create a new virtual machine environment by opening the app again):

wsl -l
wsl --unregister <DistributionName>

To access the web UI from another device on your local network, you will need to configure port forwarding:

netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=7860 connectaddress=localhost connectport=7860

Troubleshooting:

If you will use 4-bit LLaMA with WSL, you must install the WSL-Ubuntu CUDA toolkit, and it must be 11.7. This CUDA toolkit will not overwrite your WSL2 driver unlike the default CUDA toolkit. Follow these steps:

sudo apt-key del 7fa2af80 
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda-repo-wsl-ubuntu-11-7-local_11.7.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-11-7-local_11.7.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-11-7-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda

In order to avoid a CUDA error when starting the web UI, you will need to apply the following fix as seen in this comment and issue #400:

cd /home/USERNAME/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/
cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so
conda install cudatoolkit

If for some reason installing the WSL-Ubuntu CUDA toolkit does not work for you, this alternate fix should resolve any errors relating to that.

You may also need to create symbolic links to get everything working correctly. Do not do this if the above commands resolve your errors. To create the symlinks, follow the instructions here then restart your machine.

Installing 8-bit LLaMA with text-generation-webui

Linux:

1. Follow the instructions here under "Installation"
2. Download the desired Hugging Face converted model for LLaMA here
3. Copy the entire model folder, for example llama-13b-hf, into text-generation-webui\models
4. Run the following command in your conda environment: python server.py --model llama-13b-hf --load-in-8bit

Windows:

1. Install miniconda
2. Activate conda via powershell, replacing USERNAME with your username: powershell -ExecutionPolicy ByPass -NoExit -Command "& 'C:\Users\USERNAME\miniconda3\shell\condabin\conda-hook.ps1' ; conda activate 'C:\Users\USERNAME\miniconda3' "
3. Follow the instructions here under "Installation", starting with the step "Create a new conda environment."
4. Download the desired Hugging Face converted model for LLaMA here
5. Copy the entire model folder, for example llama-13b-hf, into text-generation-webui\models
6. Download libbitsandbytes_cuda116.dll and put it in C:\Users\xxx\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\
7. In \bitsandbytes\cuda_setup\main.py search for: if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None and replace with: if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None
8. In \bitsandbytes\cuda_setup\main.py search for this twice: self.lib = ct.cdll.LoadLibrary(binary_path) and replace with: self.lib = ct.cdll.LoadLibrary(str(binary_path))
9. Run the following command in your conda environment: python server.py --model llama-13b-hf --load-in-8bit

Note: for decapoda-research models, you must change "tokenizer_class": "LLaMATokenizer" to "tokenizer_class": "LlamaTokenizer" in text-generation-webui/models/llama-13b-hf/tokenizer_config.json

Installing 4-bit LLaMA with text-generation-webui

Linux:

1. Follow the instructions here under "Installation"
2. Continue with the 4-bit specific instructions here

Windows (Step-by-Step):

1. Install Build Tools for Visual Studio 2019 (has to be 2019) here. Check "Desktop development with C " when installing.
2. Install miniconda
3. Install Git from the website or simply with cmd prompt: winget install --id Git.Git -e --source winget
4. Open "x64 native tools command prompt" as admin
5. Activate conda, replacing USERNAME with your username: powershell -ExecutionPolicy ByPass -NoExit -Command "& 'C:\Users\USERNAME\miniconda3\shell\condabin\conda-hook.ps1' ; conda activate 'C:\Users\USERNAME\miniconda3' "
6. conda create -n textgen python=3.10.9
7. conda activate textgen
8. conda install cuda -c nvidia/label/cuda-11.3.0 -c nvidia/label/cuda-11.3.1
9. git clone https://github.com/oobabooga/text-generation-webui
10. cd text-generation-webui
11. pip install -r requirements.txt
12. pip install torch==1.12 cu113 -f https://download.pytorch.org/whl/torch_stable.html
13. mkdir repositories
14. cd repositories
15. git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa --branch cuda --single-branch
16. cd GPTQ-for-LLaMa
17. git reset --hard c589c5456cc1c9e96065a5d285f8e3fac2cdb0fd
18. pip install ninja
19. $env:DISTUTILS_USE_SDK=1
20. python setup_cuda.py install
21. Download the 4-bit model of your choice and place it directly into your models folder. For instance, models/llama-13b-4bit-128g. The links for the updated 4-bit models are listed below in the models directory section. If you will use 7B 4-bit, download without group-size. For 13B 4-bit and up, download with group-size.
22. Run the following command in your conda environment: ^{without group-size} python server.py --model llama-7b-4bit --wbits 4 --no-stream ^{with group-size} python server.py --model llama-13b-4bit-128g --wbits 4 --groupsize 128 --no-stream

Note: If you get the error "CUDA Setup failed despite GPU being available", do the patch in steps 6-8 of the 8-bit instructions above.

For a quick reference, here is an example chat with LLaMA 13B:

https://preview.redd.it/yhyuzgg6d8oa1.png?width=634&format=png&auto=webp&v=enabled&s=e703895ad863c57239d9d4602f105558ce6f1a8f