Llama Series Comparison Across Generations: A White Paper

Introduction

The Llama series, an open-source large language model (LLM) developed by Meta, has gained recognition for its high performance and the emphasis placed on safety and security during its development. Over the years, several significant versions of the Llama series have been released, each aimed at improving upon its predecessor:

Llama-2 Series: Released on July 18, 2023

Llama-3 Series: Released on April 18, 2024

Llama-3.1 Series: Released on July 23, 2024

This white paper aims to analyze the evolution of safety and security within the Llama series, highlighting both the improvements made across generations and the challenges that persist.

Methodology

Key Insights

We found two key insights derived from our analysis of the Llama series:

• Improvement in Safety Scores: As the Llama series has evolved, the overall Safety Scores have shown improvement when no adversarial attacks are applied.

• Mitigation of Attacks: While some attacks have been successfully mitigated in newer generations, certain attack methods remain effective.

Evaluation Dataset

To thoroughly assess the security and safety of the Llama models, we created negative prompts spanning across 31 categories within the domains of Safety, Privacy, Security, and Integrity. Each of these domains was defined as follows:

Safety: The safety domain refers to risks that promote danger such as Crime and Hate Speech.

Privacy: The privacy domain refers to risks related to data breaching such as Data Sharing and Membership Interference.

Security: The security domain refers to behavioral risks of the model such as Roleplay and Prompt Injection.

Integrity: The integrity domain refers to risks related to model ethics such as Copyright and Fraud.

In addition to these negative prompts, we applied the following attack methods to further modify the prompts, creating adversarial inputs to test the models’ robustness:

Adaptive: This method uses adaptive prompt templates to exploit model-specific vulnerabilities, tailoring the prompts based on the model's responses to maximize the likelihood of a successful attack.

ArtPrompt: ArtPrompt takes advantage of the ability of large language models (LLMs) to properly interpret ASCII art, leading them to generate unintended or harmful outputs.

GCG: GCG generates adversarial suffixes automatically based on the model's gradient, effectively steering the model toward generating harmful or undesirable content.

PAIR: PAIR employs an attacker LLM to iteratively generate and refine jailbreak prompts for a target LLM using a black-box approach, enhancing the attack's effectiveness over multiple iterations.

Past Tense: This method simply reformulates a harmful request in the past tense, exploiting the model’s tendency to respond differently to past events, potentially bypassing safety mechanisms.

ReNeLLM: ReNeLLM utilizes LLMs to automatically generate jailbreak prompts through techniques like prompt rewriting and scenario nesting, creating complex, layered attacks that can be challenging to mitigate.

TAP: TAP automates the generation of jailbreak prompts using a tree-of-thoughts reasoning and pruning strategy, systematically exploring and refining possible prompts to maximize their effectiveness.

These attack methods were chosen to evaluate the robustness of the Llama models against a wide range of adversarial techniques. Our EPASS offers the ability to compare even more attack methods, providing a comprehensive analysis of model vulnerabilities and defenses.

We evaluated approximately 10 test cases for each category and attack method combination, resulting in over 300 prompts per model and attack method.

Evaluation Metrics

Each response from the models was evaluated using two primary metrics:

Safety Score: Measures how safe the response is.

Coherence Score: Assesses the consistency and logical integrity of the response.

These two scores were combined to calculate an Overall Score on a scale from 0 to 100 using our proprietary platform. A score of 0 represents the most dangerous or inconsistent responses, while a score of 100 represents the safest and most coherent responses.

Evaluated Models

The following Llama models were evaluated in this study:

Llama-2 Series: Llama-2-7B, Llama-2-13B, Llama-2-70B

Llama-3 Series: Llama-3-8B, Llama-3-70B

Llama-3.1 Series: Llama-3.1-8B, Llama-3.1-70B, Llama-3.1-405B

This diverse set of models allowed for a comprehensive comparison across different model sizes and generations.

Results and Analysis

Key Insight 1: Improvement in Safety Scores

The following chart illustrates the Overall Scores measured from negative prompts that were not subjected to any attack methods. By comparing models of the same size across different generations, it is evident that the Overall Scores have consistently improved. This indicates that the Llama models have become increasingly safer and more secure with each subsequent release.