Widely considered the best quality-per-FLOP trade-off in the Llama 2 family, the 13B chat model scores 54.8% on MMLU and 61.9% on TruthfulQA, closing the gap with the 70B variant far more than its parameter count would suggest. At 13 billion parameters with 40 layers and a 5120-dimension hidden state, it runs on a single consumer GPU with quantization.
Llama 2 13B Chat scores 54.8% on MMLU (5-shot), roughly 10 points above Llama 2 7B Chat (45.3%) and 14 points below Llama 2 70B Chat (68.9%) on the same sheet. On TruthfulQA it reaches 61.9%, competitive with much larger models due to RLHF tuning. It represents the best quality-per-FLOP tradeoff in the Llama 2 family, running on a single consumer GPU with quantization.
The cost for running the model with Telnyx Inference is $0.0003 per 1,000 tokens. For instance, analyzing 1,000,000 customer chats, assuming each chat is 1,000 tokens long, would cost $300.
Discover the power and diversity of large language models available with Telnyx. Explore the options below to find the perfect model for your project.
| Organization | Model Name | Tasks | Languages Supported | Context Length | Parameters | Model Tier | License |
|---|---|---|---|---|---|---|---|
| No data available at this time, please try again later. |
Powered by our own GPU infrastructure, select a large language model, add a prompt, and chat away. For unlimited chats, sign up for a free account on our Mission Control Portal here.
Check out our helpful tools to help get you started.
Llama 2 13B Chat is the mid-sized model in Meta's Llama 2 family, offering more capability than the 7B variant while remaining manageable for single-GPU deployment. It is fine-tuned for dialogue using RLHF.
Yes, Llama 2 13B is released under Meta's community license for free commercial use. Weights are available on Hugging Face and through hosted inference platforms.
Llama 2 13B Chat can be loaded through Hugging Face Transformers or deployed locally with tools like Ollama. For production workloads, hosted inference provides API access without GPU management.
Meta provides Llama models through the API and open weights, not a consumer chat interface. Third-party tools and inference providers offer chat-style interfaces for interacting with Llama models.
Llama 2 13B provides noticeably better reasoning and factual accuracy than the 7B variant, at roughly double the compute cost. For tasks requiring more nuanced responses, the 13B model is worth the additional resources.
Llama 2 13B requires approximately 26GB of VRAM at full precision, or 8-10GB with 4-bit quantization. An RTX 4090 or A6000 can handle the quantized version for local development.
