232 lines
7.0 KiB
Markdown
232 lines
7.0 KiB
Markdown
---
|
||
license: other
|
||
base_model: deepseek-ai/deepseek-coder-1.3b-base
|
||
tags:
|
||
- axolotl
|
||
- generated_from_trainer
|
||
model-index:
|
||
- name: deepseek-coder-1.3b-typescript
|
||
results: []
|
||
datasets:
|
||
- bigcode/the-stack-dedup
|
||
widget:
|
||
- text: "class Person {\n constructor(public name:"
|
||
example_title: "class"
|
||
- text: "function quickSort"
|
||
example_title: "function"
|
||
---
|
||
|
||
<p align="center">
|
||
<img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="codegpt-deepseek-typescript.png?raw=true">
|
||
</p>
|
||
<p align="center"><a href="https://codegpt.co/">[CodeGPT.co]</a> | <a href="https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript">[🦙 Ollama]</a> | <a href="https://discord.gg/fKyyJX5pne">[Discord]</a> | <a href="https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt">[VSCode Extension]</a> </p>
|
||
<hr>
|
||
|
||
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
||
<details><summary>See axolotl config</summary>
|
||
|
||
axolotl version: `0.3.0`
|
||
```yaml
|
||
base_model: deepseek-ai/deepseek-coder-1.3b-base
|
||
model_type: AutoModelForCausalLM
|
||
trust_remote_code: true
|
||
load_in_8bit: false
|
||
load_in_4bit: false
|
||
strict: false
|
||
|
||
|
||
datasets:
|
||
- path: CodeGPTPlus/typescript-0-500000-seq1024
|
||
type: completion
|
||
field: text
|
||
|
||
|
||
val_set_size: 0.001
|
||
output_dir: ./fft-out
|
||
|
||
sequence_len: 1024
|
||
|
||
adapter:
|
||
lora_model_dir:
|
||
lora_r:
|
||
lora_alpha:
|
||
lora_dropout:
|
||
lora_target_linear:
|
||
lora_fan_in_fan_out:
|
||
lora_modules_to_save:
|
||
|
||
wandb_project: deepseek_1.3_fft
|
||
wandb_entity:
|
||
wandb_watch:
|
||
wandb_name: aws_a10g
|
||
wandb_log_model: end
|
||
|
||
|
||
gradient_accumulation_steps: 2
|
||
micro_batch_size: 20
|
||
num_epochs: 1
|
||
optimizer: adamw_bnb_8bit
|
||
adam_beta1: 0.9
|
||
adam_beta2: 0.999
|
||
adam_epsilon: 0.000001
|
||
max_grad_norm: 1.0
|
||
weight_decay: 0.1
|
||
lr_scheduler: cosine
|
||
learning_rate: 0.00002
|
||
train_on_inputs: false
|
||
group_by_length: false
|
||
bf16: true
|
||
fp16: false
|
||
tf32: false
|
||
gradient_checkpointing: true
|
||
early_stopping_patience:
|
||
resume_from_checkpoint:
|
||
local_rank:
|
||
logging_steps: 1
|
||
xformers_attention:
|
||
flash_attention: true
|
||
|
||
loss_watchdog_threshold: 5.0
|
||
loss_watchdog_patience: 3
|
||
|
||
hub_model_id: CodeGPTPlus/deepseek_coder_1.3b_typescript
|
||
hub_strategy: every_save
|
||
warmup_ratio: 0.01
|
||
evals_per_epoch: 20
|
||
saves_per_epoch: 3
|
||
debug:
|
||
deepspeed:
|
||
|
||
fsdp:
|
||
fsdp_config:
|
||
special_tokens:
|
||
bos_token: "<｜begin▁of▁sentence｜>"
|
||
eos_token: "<｜end▁of▁sentence｜>"
|
||
pad_token: "<｜end▁of▁sentence｜>"
|
||
```
|
||
|
||
</details><br>
|
||
|
||
# deepseek-coder-1.3b-typescript
|
||
|
||
CodeGPTPlus/deepseek-coder-1.3b-typescript, emerges as a fine-tuned iteration of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base), meticulously crafted by the CodeGPT team to excel in generating expert code in TypeScript. With specific fine-tuning for TypeScript and a dataset of 0.5B tokens, this model excels in producing precise and efficient solutions in this programming language.
|
||
|
||
The 16K window size and an additional fill-in-the-middle task are employed to deliver project-level code completion.
|
||
|
||
This new model stands as the ideal choice for those seeking a specialized code generator for TypeScript, backed by the expertise of the CodeGPT team.
|
||
|
||
It achieves the following results on the evaluation set:
|
||
- Loss: 0.7681
|
||
|
||
**Model Developers** CodeGPT Team
|
||
|
||
**Variations** 1.3B
|
||
|
||
**Input** Models input text only.
|
||
|
||
**Output** Models generate text only.
|
||
|
||
## How to Use
|
||
This model is for completion purposes only. Here give some examples of how to use the model.
|
||
|
||
#### Running the model on a GPU
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
tokenizer = AutoTokenizer.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript",
|
||
trust_remote_code=True)
|
||
model = AutoModelForCausalLM.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript",
|
||
trust_remote_code=True).cuda()
|
||
|
||
input_text = """<｜fim▁begin｜>function quickSort(arr: number[]): number[] {
|
||
if (arr.length <= 1) {
|
||
return arr;
|
||
}
|
||
const pivot = arr[0];
|
||
const left = [];
|
||
const right = [];
|
||
<｜fim▁hole｜>
|
||
return [...quickSort(left), pivot, ...quickSort(right)];
|
||
}<｜fim▁end｜>"""
|
||
|
||
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
|
||
outputs = model.generate(**inputs, max_length=256)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
### Running with Ollama
|
||
**Model:** https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript
|
||
|
||
```ollama run codegpt/deepseek-coder-1.3b-typescript```
|
||
|
||
### Running with Ollama and CodeGPT Autocomplete in VSCode
|
||
|
||
**Documentation:** https://docs.codegpt.co/docs/tutorial-features/code_autocompletion
|
||
|
||
Select "Ollama - codegpt/deepseek-coder-1.3b-typescript" in the autocomplete model selector.
|
||
|
||
Then, write any code or comment in the vscode text editor, and the model will provide you with code suggestions through the CodeGPT code autocomplete.
|
||
|
||
<img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="ollama_autocomplete_codegpt.gif">
|
||
|
||
### Fill In the Middle (FIM)
|
||
```python
|
||
<｜fim▁begin｜>function quickSort(arr: number[]): number[] {
|
||
if (arr.length <= 1) {
|
||
return arr;
|
||
}
|
||
const pivot = arr[0];
|
||
const left = [];
|
||
const right = [];
|
||
<｜fim▁hole｜>
|
||
return [...quickSort(left), pivot, ...quickSort(right)];
|
||
}<｜fim▁end｜>
|
||
```
|
||
|
||
## Training procedure
|
||
|
||
### Training hyperparameters
|
||
|
||
The following hyperparameters were used during training:
|
||
- learning_rate: 2e-05
|
||
- train_batch_size: 20
|
||
- eval_batch_size: 20
|
||
- seed: 42
|
||
- gradient_accumulation_steps: 2
|
||
- total_train_batch_size: 40
|
||
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
|
||
- lr_scheduler_type: cosine
|
||
- lr_scheduler_warmup_steps: 261
|
||
- num_epochs: 1
|
||
|
||
### Training results
|
||
|
||
| Training Loss | Epoch | Step | Validation Loss |
|
||
|:-------------:|:-----:|:-----:|:---------------:|
|
||
| 1.0745 | 0.0 | 1 | 0.8681 |
|
||
| 1.2267 | 0.05 | 1308 | 0.8130 |
|
||
| 1.1594 | 0.1 | 2616 | 0.8018 |
|
||
| 0.7674 | 0.15 | 3924 | 0.7942 |
|
||
| 0.6443 | 0.2 | 5232 | 0.7889 |
|
||
| 0.9155 | 0.25 | 6540 | 0.7847 |
|
||
| 0.7501 | 0.3 | 7848 | 0.7819 |
|
||
| 0.8835 | 0.35 | 9156 | 0.7792 |
|
||
| 0.7261 | 0.4 | 10464 | 0.7769 |
|
||
| 0.9746 | 0.45 | 11772 | 0.7748 |
|
||
| 0.6884 | 0.5 | 13080 | 0.7734 |
|
||
| 0.6104 | 0.55 | 14388 | 0.7722 |
|
||
| 0.8876 | 0.6 | 15696 | 0.7710 |
|
||
| 0.9567 | 0.65 | 17004 | 0.7703 |
|
||
| 0.6915 | 0.7 | 18312 | 0.7696 |
|
||
| 0.8874 | 0.75 | 19620 | 0.7691 |
|
||
| 0.6124 | 0.8 | 20928 | 0.7686 |
|
||
| 0.8147 | 0.85 | 22236 | 0.7684 |
|
||
| 0.8021 | 0.9 | 23544 | 0.7683 |
|
||
| 0.8665 | 0.95 | 24852 | 0.7681 |
|
||
|
||
|
||
### Framework versions
|
||
|
||
- Transformers 4.37.0.dev0
|
||
- Pytorch 2.0.1+cu118
|
||
- Datasets 2.16.1
|
||
- Tokenizers 0.15.0 |