91 lines
3.9 KiB
Markdown
91 lines
3.9 KiB
Markdown
---
|
||
license: apache-2.0
|
||
datasets:
|
||
- JetBrains/KExercises
|
||
base_model: deepseek-ai/deepseek-coder-6.7b-base
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
dataset:
|
||
name: MultiPL-HumanEval (Kotlin)
|
||
type: openai_humaneval
|
||
metrics:
|
||
- name: pass@1
|
||
type: pass@1
|
||
value: 55.28
|
||
tags:
|
||
- code
|
||
---
|
||
|
||
# Kexer models
|
||
|
||
Kexer models are a collection of open-source generative text models fine-tuned on the [Kotlin Exercices](https://huggingface.co/datasets/JetBrains/KExercises) dataset.
|
||
This is a repository for the fine-tuned **Deepseek-coder-6.7b** model in the *Hugging Face Transformers* format.
|
||
|
||
# How to use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
# Load pre-trained model and tokenizer
|
||
model_name = 'JetBrains/deepseek-coder-6.7B-kexer'
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
|
||
|
||
# Create and encode input
|
||
input_text = """\
|
||
This function takes an integer n and returns factorial of a number:
|
||
fun factorial(n: Int): Int {\
|
||
"""
|
||
input_ids = tokenizer.encode(
|
||
input_text, return_tensors='pt'
|
||
).to('cuda')
|
||
|
||
# Generate
|
||
output = model.generate(
|
||
input_ids, max_length=60, num_return_sequences=1,
|
||
early_stopping=True, pad_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
# Decode output
|
||
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
|
||
print(generated_text)
|
||
```
|
||
|
||
As with the base model, we can use FIM. To do this, the following format must be used:
|
||
```
|
||
'<｜fim▁begin｜>' + prefix + '<｜fim▁hole｜>' + suffix + '<｜fim▁end｜>'
|
||
```
|
||
|
||
# Training setup
|
||
|
||
The model was trained on one A100 GPU with following hyperparameters:
|
||
|
||
| **Hyperparameter** | **Value** |
|
||
|:---------------------------:|:----------------------------------------:|
|
||
| `warmup` | 10% |
|
||
| `max_lr` | 1e-4 |
|
||
| `scheduler` | linear |
|
||
| `total_batch_size` | 256 (~130K tokens per step) |
|
||
| `num_epochs` | 4 |
|
||
|
||
More details about fine-tuning can be found in the technical report (coming soon!).
|
||
|
||
# Fine-tuning data
|
||
|
||
For tuning this model, we used 15K exmaples from the synthetically generated [Kotlin Exercices](https://huggingface.co/datasets/JetBrains/KExercises) dataset. Every example follows the HumanEval format. In total, the dataset contains about 3.5M tokens.
|
||
|
||
# Evaluation
|
||
|
||
For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).
|
||
|
||
Here are the results of our evaluation:
|
||
|
||
| **Model name** | **Kotlin HumanEval Pass Rate** |
|
||
|:---------------------------:|:----------------------------------------:|
|
||
| `Deepseek-coder-6.7B` | 40.99 |
|
||
| `Deepseek-coder-6.7B-kexer` | **55.28** |
|
||
|
||
# Ethical considerations and limitations
|
||
|
||
Deepseek-coder-6.7B-kexer is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Deepseek-coder-6.7B-kexer's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of Deepseek-coder-6.7B-kexer, developers should perform safety testing and tuning tailored to their specific applications of the model. |