78 lines
4.5 KiB
Markdown
78 lines
4.5 KiB
Markdown
---
|
|
license: llama3.3
|
|
---
|
|
# Llama 3.3 8B Instruct
|
|
|
|
Yes, this is official, and yes, this is, to my knowledge, a real version of Llama 3.3 8B!
|
|
|
|
**I would highly recommend trying both this model, and [a version with the Llama 3.3 70B config applied to extend the context length to 128k](/shb777/Llama-3.3-8B-Instruct). I am unsure as to which one of these is closest to real; while the original copy I downloaded came with the 8k context configuration, benchmarks seem to slightly improve on the 128k version.**
|
|
|
|
|
|
## Benchmarks
|
|
|
|
| | Llama 3.1 8B Instruct | Llama 3.3 8B Instruct (as downloaded from Facebook) | [Llama 3.3 8B Instruct w/ Llama 3.3 70B RoPE config to extend to 128k context](/shb777/Llama-3.3-8B-Instruct)
|
|
|-|-|-|-|
|
|
|IFEval (1 epoch, score avged across all strict/loose instruction/prompt accuracies to follow Llama 3 paper)|78.2|81.95|84.775
|
|
|GPQA Diamond (3 epochs)|29.3|37.0|37.5
|
|
|
|
All benchmarks done in OpenBench at 1.0 temp.
|
|
|
|
<details>
|
|
|
|
|
|
## Rationale
|
|
|
|
Facebook has a [Llama API](https://llama.developer.meta.com) available that allows for inference of the other Llama models (L3.3 70B, L4 Scout and Maverick), but *also* includes a special, new (according to the original press release) "Llama 3.3 8B" that didn't exist anywhere else and was stuck behind the Facebook API!
|
|
|
|
However. The Llama API supports finetuning L3.3... *and downloading the final model in HF format.* Problem solved, right?
|
|
|
|
Wellllllllllllllll. Not really. The finetuning API was hidden behind layers of support tickets. I tried when the original API dropped in April, and was just told "We'll think about it and send you any updates" (there never were any updates).
|
|
|
|
Flash forward to December, on a whim I decide to look at the API again. And... by god... the finetuning tab was there. I could click on it and start a job (please ignore that I have no idea how it works, and in fact the finetuning tab actually disappeared after the first time I clicked on it, though I could still manually go to the page).
|
|
|
|
Apparently, this was not very well tested, as there were a good few bugs, the UI was janky, and the download model function did not actually work due to CORS (I had to manually curl things to get the CDN link).
|
|
|
|
But... by god... the zip file downloaded, and I had my slightly finetuned model.
|
|
|
|
To my shock and delight, however, they also provide the adapter that they merged into the model. That means I can *subtract* that adapter and get the original model. And... here we are!
|
|
|
|
## More cursed discoveries
|
|
|
|
1. Apparently the context length of the *original* Llama 3.3 model (i.e. the regular one that the Llama API serves) is 128k, while the finetunable version is only 8k. This is true across both the downloaded version *and* the version of the finetune served by the API (refer to screenshot with 10k tokens worth of `a` as input). This does not really make any coherent sense.
|
|
|
|
|
|
## Are you sure this is really Llama 3.3?
|
|
|
|
As far as I'm aware! It has stylistic tics that differ from Llama 3 and 3.1.
|
|
|
|
There are a few weird artifacts, however, of the base model being the... original Llama 3(???)
|
|
|
|
The original ZIP included an `original_repo_id.json` that contained:
|
|
|
|
```json
|
|
{
|
|
"repo_id": "meta-llama/Meta-Llama-3-8B-Instruct"
|
|
}
|
|
```
|
|
|
|
and, furthermore, the `adapter_config.json` *also* had Llama 3 as the base model. However, the models clearly act different and know different things! Furthermore, I tested across the Llama API and my copy, and they both share the same differences from L3 and L3.1.
|
|
|
|
Suffice it to say, I'm pretty sure this is really Llama 3.3 8B.
|
|
|
|
## Is this legal?
|
|
|
|
According to the [T&S of the Llama API as of December 29th, 2025](https://archive.is/y7KSR):
|
|
|
|
> For example, via the Llama API, you may receive access to the Llama 3.3 8b model, which is considered a Llama AI model and part of the Meta AI Materials; when downloaded, and not accessed via the Llama API, the Llama 3.3 8b model is subject to the Llama 3.3 Community License Agreement and Acceptable Use Policy.
|
|
|
|
The Llama 3.3 8b model (after downloading) is subject to the regular L3.3 license, which allows for redistribution. So... as far as I can tell, yes, this is perfectly legal to redistribute!
|
|
|
|
---
|
|
|
|
Llama 3.3 is licensed under the Llama 3.3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved
|
|
|
|
Please email me at `fizzarolli [at] riseup.net` for any conerns. If Meta would like me to take this model down, please have someone email me and ask from an official Meta address.
|
|
|
|
|
|
|
|
</details> |