llama2 inference numbers

#llama #large_language_models #technology #performance

%at=2023-07-25T01:18:10

remember, the gold SUBJECTIVE target is 10tok/s (i think thats what chatgpt runs at, probably 40tok/s, i dont run those numbers yet), which is around 100ms per token. going slower might be good enough for you though!!

all experiments running with llama.cpp

arae is same as arae from llama inference numbers, except cublas now gets a

• nvidia mx350 gpu (2gb vram, allegedly)

wall is the same as wall from llama inference numbers. no hardware changes for those runs

setting up cublas #

i hate the christ that is Python Packaging. as seen by my python packaging patterns

1. install miniconda somewhere. get miniconda here. run the script
   1. i dont like doing conda init. activate it manually
   2. for fish, i run eval "$(/home/luna/tmp/miniconda3/bin/conda shell.fish hook)" manually when i want to enter conda
2. conda new -n llamacpp
3. conda activate llamacpp
4. conda install -c 'nvidia/label/cuda-12.2.0' cuda-toolkit
   1. around 10gb disk space needed
5. get llama.cpp
6. edit its makefile's LDFLAGS to point it to your conda environment (it assumes CUDA toolkit is on /opt/cuda, we dont do that because this method of installing CUDA is non-standard! but it's easily reproducible at least.).
   1. e.g add -L/home/luna/tmp/miniconda3/envs/llamacpp/lib
7. make LLAMA_CUBLAS=1
8. wget 'https://huggingface.co/TheBloke/llama2_7b_chat_uncensored-GGML/resolve/main/llama2_7b_chat_uncensored.ggmlv3.q4_K_M.bin'
   1. other quantization methods may be wanted, like q5 for larger ram systems??
9. the compiled ./main will have successfully linked to CUDA libraries in the conda path, but your linker does not know about those. that means diretcly using the binary leads to ./main: error while loading shared libraries: libcublas.so.12: cannot open shared object file: No such file or directory. use LD_LIBRARY_PATH overrides to fix this
10. full run looks like this env LD_LIBRARY_PATH=/home/luna/tmp/miniconda3/envs/llamacpp/lib ./main -m ./llama2_7b_chat_uncensored.ggmlv3.q4_K_M.bin --color-b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1 -t 7 -ngl 12 -p 'Write a paragraph about the hit game Among Us'
    1. -ngl is the amount of model layers sent to the gpu. for my 400ms/tok number i used -ngl 12, anything longer is an OOM (my gpu has 2gb vram as reported by nvidia-smi)
    2. those are bad parameters for actual inference. if you want to actually use this for anything, choose different parameters
       1. this might help you

setting up without cublas #

1. get llama.cpp
2. make. you need gcc and g++
3. wget 'https://huggingface.co/TheBloke/llama2_7b_chat_uncensored-GGML/resolve/main/llama2_7b_chat_uncensored.ggmlv3.q4_K_M.bin'
4. ./main -m ./llama2_7b_chat_uncensored.ggmlv3.q4_K_M.bin --color-b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1 -t 7 -p 'Write a paragraph about the hit game Among Us'
   1. same thing as with cuBLAS, but without -ngl and LD_LIBRARY_PATH