performance

#llama #large_language_models #technology #performance | hostname | device | model | time per token | | - | - | - | - | | arae | cpu | 7b q4 (non-gptq) | 1000ms | | arae | cpu | 7b q4 (gptq) | TODO | | wall | cpu | 7b q4 (non-gptq) | 160ms | | wall | cpu…

see the others: - [llama inference numbers] - [llama2 inference numbers] i am not running perplexity scores on these (that's more important for a quantization comparison chart, rather than performance parameter tweaking), just want to find out how …

#llama #large_language_models #technology #performance %at=2023-07-25T01:18:10 remember, the gold **SUBJECTIVE** target is 10tok/s (i think thats what chatgpt runs at, probably 40tok/s, i dont run those numbers yet), which is around 100ms per token. goi…

#llama #large_language_models #technology #performance WIP DRAFT %at=2024-05-01T17:13:07.283Z to continue the tradition from llama1 and llama2, here is the llama3 numbers. this time i made a script: [llamabench](https://github.com/lun-4/llamabench) to s…