+
+## Model Details
+The model is designed for translating short text segments between any pair of the supported languages.
+
+Supported language pairs:
+
+- Korean ↔ Vietnamese
+- Korean ↔ Indonesian
+- Korean ↔ Cambodian
+- Korean ↔ Thai
+- Vietnamese ↔ Indonesian
+- Vietnamese ↔ Cambodian
+- Vietnamese ↔ Thai
+- Indonesian ↔ Cambodian
+- Indonesian ↔ Thai
+- Cambodian ↔ Thai
+
+### Model Description
+
+This model is specifically optimized for Southeast Asian language translation needs, focusing on enabling communication between these specific language communities.
+
+The extensive training data of 20M examples (1M for each translation direction) provides a robust foundation for handling common expressions and basic conversations across these languages.
+
+### Model Architecture
+
+Base Model: meta-llama/Llama-3.1-8B-Instruct
+
+
+## Bias, Risks, and Limitations
+
+- Performance is limited to short sentences and phrases
+- May not handle complex or lengthy text effectively
+- Translation quality may vary depending on language pair and content complexity
+
+## Evaluation results
+
+| Source Language | Target Language | BLEU Score | ROUGE-1 | ROUGE-L |
+|----------------|-----------------|------------|---------|---------|
+| Korean | Vietnamese | 56.70 | 81.64 | 76.66 |
+| Korean | Cambodian | 71.69 | 89.26 | 88.20 |
+| Korean | Indonesian | 58.32 | 80.39 | 76.63 |
+| Korean | Thai | 63.26 | 78.88 | 72.29 |
+| Vietnamese | Korean | 49.01 | 75.57 | 72.74 |
+| Vietnamese | Cambodian | 78.26 | 90.74 | 90.32 |
+| Vietnamese | Indonesian | 65.96 | 83.08 | 81.46 |
+| Vietnamese | Thai | 65.93 | 81.09 | 76.57 |
+| Cambodian | Korean | 49.10 | 72.67 | 69.75 |
+| Cambodian | Vietnamese | 63.42 | 81.56 | 79.09 |
+| Cambodian | Indonesian | 61.41 | 79.67 | 77.75 |
+| Cambodian | Thai | 70.91 | 81.85 | 77.66 |
+| Indonesian | Korean | 53.61 | 77.14 | 74.29 |
+| Indonesian | Vietnamese | 68.21 | 85.41 | 83.10 |
+| Indonesian | Cambodian | 78.84 | 90.81 | 90.35 |
+| Indonesian | Thai | 67.12 | 81.54 | 77.19 |
+| Thai | Korean | 45.59 | 72.48 | 69.46 |
+| Thai | Vietnamese | 61.55 | 81.01 | 78.24 |
+| Thai | Cambodian | 78.52 | 91.47 | 91.16 |
+| Thai | Indonesian | 58.99 | 78.56 | 76.40 |
+
+## Example
+
+```py
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+ "MLP-KTLim/llama-3.1-Asian-Bllossom-8B-Translator",
+ torch_dtype="auto",
+ device_map="auto",
+)
+
+tokenizer = AutoTokenizer.from_pretrained(
+ "MLP-KTLim/llama-3.1-Asian-Bllossom-8B-Translator",
+)
+
+input_text = "안녕하세요? 아시아 언어 번역 모델 입니다."
+
+def get_input_ids(source_lang, target_lang, message):
+ assert source_lang in ["Korean", "Vietnamese", "Indonesian", "Thai", "Cambodian"]
+ assert target_lang in ["Korean", "Vietnamese", "Indonesian", "Thai", "Cambodian"]
+
+ input_ids = tokenizer.apply_chat_template(
+ conversation=[
+ {"role": "system", "content": f"You are a useful translation AI. Please translate the sentence given in {source_lang} into {target_lang}."},
+ {"role": "user", "content": message},
+ ],
+ tokenize=True,
+ return_tensors="pt",
+ add_generation_prompt=True,
+ )
+ return input_ids
+
+input_ids = get_input_ids(
+ source_lang="Korean",
+ target_lang="Vietnamese",
+ message=input_text,
+)
+
+output = model.generate(
+ input_ids.to(model.device),
+ max_new_tokens=128,
+)
+
+print(tokenizer.decode(output[0][len(input_ids[0]):], skip_special_tokens=True))
```
-如果您是本模型的贡献者，我们邀请您根据模型贡献文档，及时完善模型卡片内容。
\ No newline at end of file + +## Contributor +- 원인호 (wih1226@seoultech.ac.kr) +- 김민준 (mjkmain@seoultech.ac.kr) + + diff --git a/configuration.json b/configuration.json new file mode 100644 index 0000000..159097f --- /dev/null +++ b/configuration.json @@ -0,0 +1 @@ +{"framework": "pytorch", "task": "others", "allow_remote": true} \ No newline at end of file diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q2_K.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q2_K.gguf new file mode 100644 index 0000000..cac6e35 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8ce6cc376fe14246c75dfc2950f3e2ef7dab02d223120227b8d8c6b261f6435a +size 3179136352 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_L.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_L.gguf new file mode 100644 index 0000000..ee551b8 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a2d63db545c414e0abc5de575204f0167d6724d78a2342350fe809f98feeaa55 +size 4321961312 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_M.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_M.gguf new file mode 100644 index 0000000..7a3af40 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:45e235d6b8bf298862a031bc1ba238b3436e5ba5de0fe26a91810d54368ea2e5 +size 4018922848 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_S.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_S.gguf new file mode 100644 index 0000000..259a9ac --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ebe9c4f006becd324c1559435ebb1fe7c5566d01410f38313c09c7fdc16c350d +size 3664504160 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0.gguf new file mode 100644 index 0000000..40cc321 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:889b19180b29ca4bb0567426a44d4a0d717e1e09e4593463d03a1b8a24c5570f +size 4661216608 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_4_4.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_4_4.gguf new file mode 100644 index 0000000..a5d3f0e --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_4_4.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:73be4caaaee2a624fa0155f926513408c59e1d9955d1e98e7285531372e1f011 +size 4661216608 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_4_8.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_4_8.gguf new file mode 100644 index 0000000..b2b9440 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_4_8.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c8cd82427285627c7ede558a36f2c6f90fb4707116091e2df096d823849e449f +size 4661216608 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_8_8.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_8_8.gguf new file mode 100644 index 0000000..feaef43 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_0_8_8.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f4544c4505bda00cd930184fcf2bf56fb0d9b343ddac941cd68b0504b0a57d7e +size 4661216608 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_1.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_1.gguf new file mode 100644 index 0000000..022a785 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5210ebd599dd5968e858b49ccfa2538d6685c5de07d6e32b27cd5157e426ee94 +size 5130257760 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_K_M.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_K_M.gguf new file mode 100644 index 0000000..4526c75 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e996e6c9da143aa13770925f840b5f150b85bd057fd5c563a7e10ee621c0fe01 +size 4920739168 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q4_K_S.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_K_S.gguf new file mode 100644 index 0000000..24adf83 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8c111ab784eb742c1eb27158637ab92441c27ab7ebf4ed1d9948d38a08ef6635 +size 4692673888 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q5_0.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_0.gguf new file mode 100644 index 0000000..20eff28 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5700fa0c48db5e996759ad38c0d0a5ac10d0059e252933f252ad1585d002d565 +size 5599298912 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q5_1.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_1.gguf new file mode 100644 index 0000000..b4e55a2 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:71de48c86a8233b90e705cef65de2d4affa3d9e8c44ada2111169d36687aee81 +size 6068340064 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q5_K_M.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_K_M.gguf new file mode 100644 index 0000000..6e11187 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5710fea256f67ce48cb2213b3444224c9521dd33d456b1341a8b53a286ee8828 +size 5732992352 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q5_K_S.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_K_S.gguf new file mode 100644 index 0000000..5bc516f --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5de5422d74760becd20d924ca4f42ecc8699ae0946c45484ff0930cd41215a71 +size 5599298912 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q6_K.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q6_K.gguf new file mode 100644 index 0000000..6cc1306 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2ac05cc8867ae60b8206a2c942d09172d8079bbf658799cd57de6c67c02c80fa +size 6596011360 diff --git a/llama-3.1-Asian-Bllossom-8B-Translator.Q8_0.gguf b/llama-3.1-Asian-Bllossom-8B-Translator.Q8_0.gguf new file mode 100644 index 0000000..ea293f2 --- /dev/null +++ b/llama-3.1-Asian-Bllossom-8B-Translator.Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:619ae723e4e3c56c25f04202982e612cf0846e0c2b59295bb91e258241b7fa37 +size 8540775776