60 lines
3.1 KiB
Markdown
60 lines
3.1 KiB
Markdown
---
|
|
license: apache-2.0
|
|
datasets:
|
|
- JetBrains/KExercises
|
|
base_model: JetBrains/CodeLlama-7B-Kexer
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
dataset:
|
|
name: MultiPL-HumanEval (Kotlin)
|
|
type: openai_humaneval
|
|
metrics:
|
|
- name: pass@1
|
|
type: pass@1
|
|
value: 42.24
|
|
tags:
|
|
- code
|
|
---
|
|
|
|
# CodeLlama-7B-Kexer-GGUF
|
|
This is quantized version of [JetBrains/CodeLlama-7B-Kexer](https://huggingface.co/JetBrains/CodeLlama-7B-Kexer) created using llama.cpp
|
|
|
|
# Model Description
|
|
|
|
Kexer models are a collection of open-source generative text models fine-tuned on the [Kotlin Exercices](https://huggingface.co/datasets/JetBrains/KExercises) dataset.
|
|
This is a repository for the fine-tuned **CodeLlama-7b** model in the *Hugging Face Transformers* format.
|
|
|
|
|
|
# Training setup
|
|
|
|
The model was trained on one A100 GPU with the following hyperparameters:
|
|
|
|
| **Hyperparameter** | **Value** |
|
|
|:---------------------------:|:----------------------------------------:|
|
|
| `warmup` | 10% |
|
|
| `max_lr` | 1e-4 |
|
|
| `scheduler` | linear |
|
|
| `total_batch_size` | 256 (~130K tokens per step) |
|
|
| `num_epochs` | 4 |
|
|
|
|
More details about fine-tuning can be found in the technical report (coming soon!).
|
|
|
|
# Fine-tuning data
|
|
|
|
For tuning this model, we used 15K exmaples from the synthetically generated [Kotlin Exercices](https://huggingface.co/datasets/JetBrains/KExercises) dataset. Every example follows the HumanEval format. In total, the dataset contains about 3.5M tokens.
|
|
|
|
# Evaluation
|
|
|
|
For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).
|
|
|
|
Here are the results of our evaluation:
|
|
|
|
| **Model name** | **Kotlin HumanEval Pass Rate** |
|
|
|:---------------------------:|:----------------------------------------:|
|
|
| `CodeLlama-7B` | 26.89 |
|
|
| `CodeLlama-7B-Kexer` | **42.24** |
|
|
|
|
# Ethical considerations and limitations
|
|
|
|
CodeLlama-7B-Kexer is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-Kexer's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-Kexer, developers should perform safety testing and tuning tailored to their specific applications of the model. |