About Machine Learning Model Llama-2

Llama-2 is an advanced collection of pretrained and fine-tuned large language models (LLMs), developed and released by Meta AI. With a parameter range from 7 billion to 70 billion, Llama 2 stands as an updated version of its predecessor, Llama 1. It represents a significant step forward in the field of LLMs, particularly for dialogue use cases through its fine-tuned variants, Llama 2-Chat. This family of models not only surpasses the performance of existing open-source chat models on several benchmarks but also matches the level of some closed-source models in terms of helpfulness and safety, as evaluated by human testing. Llama 2's development involved substantial changes from the original Llama, including an expanded pretraining corpus by 40%, a doubled context length, and the adoption of grouped-query attention. The team behind Llama 2 focused on increasing the safety of these models through specific data annotation and tuning techniques, red-teaming, and iterative evaluations.

Model Card for Llama-2

• Model Details:

  • Developers: Meta AI
  • Model Dates: January 2023 to July 2023
  • Variations: 7B, 13B, and 70B parameter sizes, both pretrained and fine-tuned
  • Architecture: Auto-regressive language model using an optimized transformer architecture. Fine-tuned versions employ supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF).
  • License: Custom commercial license (ai.meta.com/resources/models-and-libraries/llama-downloads/)
  • Comments: Feedback can be provided through the model README or the GitHub repository (github.com/facebookresearch/llama)

• Intended Use:

  • Primary Uses: Commercial and research applications in English; assistant-like chat for tuned models and various natural language generation tasks for pretrained models
  • Out-of-Scope Uses: Any use that violates applicable laws or regulations, use in languages other than English, or any use prohibited by the Acceptable Use Policy and Licensing Agreement for Llama 2

• Training Data:

  • Overview: Trained on 2 trillion tokens from publicly available sources, not including Meta user data. Fine-tuning data includes publicly available instruction datasets and over one million new human-annotated examples.
  • Data Freshness: Pretraining data cut off in September 2022, with some tuning data up to July 2023

• Ethical Considerations and Limitations:

  • Llama 2 carries inherent risks due to its new technology. Testing has been limited to English and cannot cover all scenarios. The potential outputs of Llama 2 cannot be predicted in advance and may produce inaccurate or objectionable responses. Developers are advised to perform safety testing and tuning tailored to their specific applications of the model.

• Hardware and Software:

  • Utilized custom training libraries, Meta’s Research Super Cluster, and production clusters for pretraining. Fine-tuning, annotation, and evaluation were also performed on third-party cloud compute.

• Carbon Footprint:

  • Pretraining involved 3.3M GPU hours on A100-80GB hardware. Estimated total emissions were 539 tCO2.

References

• Llama 2: Open Foundation and Fine-Tuned Chat Models - Abstract
• Llama 2: Open Foundation and Fine-Tuned Chat Models - Full Paper