初始化项目，由ModelHub XC社区提供模型

Model: llm-jp/llm-jp-1.3b-v1.0 Source: Original Platform
2026-07-12 19:13:06 +08:00
commit dbcbe1d941
9 changed files with 283 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,50 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+ 
+ 
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+ 
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.gguf* filter=lfs diff=lfs merge=lfs -text
+*.ggml filter=lfs diff=lfs merge=lfs -text
+*.llamafile* filter=lfs diff=lfs merge=lfs -text
+*.pt2 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+
+pytorch_model.bin filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,155 @@
+---
+license: apache-2.0
+language:
+  - en
+  - ja
+programming_language:
+  - C
+  - C++
+  - C#
+  - Go
+  - Java
+  - JavaScript
+  - Lua
+  - PHP
+  - Python
+  - Ruby
+  - Rust
+  - Scala
+  - TypeScript
+library_name: transformers
+pipeline_tag: text-generation
+inference: false
+---
+# llm-jp-1.3b-v1.0
+
+This repository provides large language models developed by [LLM-jp](https://llm-jp.nii.ac.jp/), a collaborative project launched in Japan.
+
+| Model Variant | 
+| :--- |
+|**Instruction models**|
+| [llm-jp-13b-instruct-full-jaster-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-instruct-full-jaster-v1.0) |
+| [llm-jp-13b-instruct-full-jaster-dolly-oasst-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-instruct-full-jaster-dolly-oasst-v1.0) |
+| [llm-jp-13b-instruct-full-dolly-oasst-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-instruct-full-dolly-oasst-v1.0) |
+| [llm-jp-13b-instruct-lora-jaster-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-instruct-lora-jaster-v1.0) | 
+| [llm-jp-13b-instruct-lora-jaster-dolly-oasst-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-instruct-lora-jaster-dolly-oasst-v1.0) | 
+| [llm-jp-13b-instruct-lora-dolly-oasst-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-instruct-lora-dolly-oasst-v1.0) | 
+
+
+|  | 
+| :--- |
+|**Pre-trained models**|
+| [llm-jp-13b-v1.0](https://huggingface.co/llm-jp/llm-jp-13b-v1.0) | 
+| [llm-jp-1.3b-v1.0](https://huggingface.co/llm-jp/llm-jp-1.3b-v1.0) | 
+Checkpoints format: Hugging Face Transformers (Megatron-DeepSpeed format models are available [here](https://huggingface.co/llm-jp/llm-jp-13b-v1.0-mdsfmt))
+
+
+## Required Libraries and Their Versions
+
+- torch>=2.0.0
+- transformers>=4.34.0
+- tokenizers>=0.14.0
+- accelerate==0.23.0
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+tokenizer = AutoTokenizer.from_pretrained("llm-jp/llm-jp-1.3b-v1.0")
+model = AutoModelForCausalLM.from_pretrained("llm-jp/llm-jp-1.3b-v1.0", device_map="auto", torch_dtype=torch.float16)
+text = "自然言語処理とは何か"
+tokenized_input = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    output = model.generate(
+        tokenized_input,
+        max_new_tokens=20,
+        do_sample=True,
+        top_p=0.90,
+        temperature=0.7,
+    )[0]
+print(tokenizer.decode(output))
+```
+
+
+## Model Details
+
+- **Model type:** Transformer-based Language Model
+- **Total seen tokens:** 300B
+
+|Model|Params|Layers|Hidden size|Heads|Context length|
+|:---:|:---:|:---:|:---:|:---:|:---:|
+|13b model|13b|40|5120|40|2048|
+|1.3b model|1.3b|24|2048|16|2048|
+
+
+## Training
+
+- **Pre-training:**
+  - **Hardware:** 96 A100 40GB GPUs ([mdx cluster](https://mdx.jp/en/))
+  - **Software:** Megatron-DeepSpeed
+
+- **Instruction tuning:**
+  - **Hardware:** 8 A100 40GB GPUs ([mdx cluster](https://mdx.jp/en/))
+  - **Software:** [TRL](https://github.com/huggingface/trl), [PEFT](https://github.com/huggingface/peft), and [DeepSpeed](https://github.com/microsoft/DeepSpeed)
+
+## Tokenizer
+The tokenizer of this model is based on [huggingface/tokenizers](https://github.com/huggingface/tokenizers) Unigram byte-fallback model.
+The vocabulary entries were converted from [`llm-jp-tokenizer v2.1 (50k)`](https://github.com/llm-jp/llm-jp-tokenizer/releases/tag/v2.1).
+Please refer to [README.md](https://github.com/llm-jp/llm-jp-tokenizer) of `llm-ja-tokenizer` for details on the vocabulary construction procedure.
+- **Model:** Hugging Face Fast Tokenizer using Unigram byte-fallback model which requires `tokenizers>=0.14.0`
+- **Training algorithm:** SentencePiece Unigram byte-fallback
+- **Training data:** A subset of the datasets for model pre-training
+- **Vocabulary size:** 50,570 (mixed vocabulary of Japanese, English, and source code)
+
+
+## Datasets
+
+### Pre-training
+
+The models have been pre-trained using a blend of the following datasets.
+
+| Language | Dataset | Tokens|
+|:---:|:---:|:---:|
+|Japanese|[Wikipedia](https://huggingface.co/datasets/wikipedia)|1.5B
+||[mC4](https://huggingface.co/datasets/mc4)|136B
+|English|[Wikipedia](https://huggingface.co/datasets/wikipedia)|5B
+||[The Pile](https://huggingface.co/datasets/EleutherAI/pile)|135B
+|Codes|[The Stack](https://huggingface.co/datasets/bigcode/the-stack)|10B
+
+The pre-training was continuously conducted using a total of 10 folds of non-overlapping data, each consisting of approximately 27-28B tokens.
+We finalized the pre-training with additional (potentially) high-quality 27B tokens data obtained from the identical source datasets listed above used for the 10-fold data.
+
+### Instruction tuning
+
+The models have been fine-tuned on the following datasets.
+ 
+| Language | Dataset | description |
+|:---|:---:|:---:|
+|Japanese|[jaster](https://github.com/llm-jp/llm-jp-eval)| An automatically transformed data from the existing Japanese NLP datasets |
+||[databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)| A translated one by DeepL in LLM-jp |
+||[OpenAssistant Conversations Dataset](https://huggingface.co/datasets/OpenAssistant/oasst1)| A translated one by DeepL in LLM-jp |
+
+
+## Evaluation
+You can view the evaluation results of several LLMs on this [leaderboard](http://wandb.me/llm-jp-leaderboard). We used [llm-jp-eval](https://github.com/llm-jp/llm-jp-eval) for the evaluation.
+
+## Risks and Limitations
+
+The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
+
+
+## Send Questions to
+
+llm-jp(at)nii.ac.jp
+
+
+## License
+
+[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+
+## Model Card Authors
+*The names are listed in alphabetical order.*
+
+Hirokazu Kiyomaru, Hiroshi Matsuda, Jun Suzuki, Namgi Han, Saku Sugawara, Shota Sasaki, Shuhei Kurita, Taishi Nakamura, Takumi Okamoto.
--- a/config.json
+++ b/config.json
@@ -0,0 +1,33 @@
+{
+  "activation_function": "gelu",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 7,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 7,
+  "gradient_checkpointing": false,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 2048,
+  "n_head": 16,
+  "n_inner": 8192,
+  "n_layer": 24,
+  "n_positions": 2048,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float16",
+  "transformers_version": "4.34.0",
+  "use_cache": true,
+  "vocab_size": 50688
+}
--- a/configuration.json
+++ b/configuration.json
@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 7,
+  "eos_token_id": 7,
+  "transformers_version": "4.34.0"
+}
--- a/pytorch_model.bin
+++ b/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ed5e7d03afb86318ccf73475a3151192a0f25994ab05c8d2396cce9e74953e9
+size 2633307101
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,13 @@
+{
+    "additional_special_tokens": [
+        "</s|LLM-jp>"
+    ],
+    "bos_token": "<s|LLM-jp>",
+    "cls_token": "<CLS|LLM-jp>",
+    "eod_token": "<EOD|LLM-jp>",
+    "eos_token": "<EOD|LLM-jp>",
+    "mask_token": "<mask|LLM-jp>",
+    "pad_token": "<pad|LLM-jp>",
+    "sep_token": "<SEP|LLM-jp>",
+    "unk_token": "<unk|LLM-jp>"
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fefc427dff3323dd8a2fd66f392b90a62896db3b11a031463ad0f4c70fb1de9c
+size 3245878
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,19 @@
+{
+  "unk_token": "<unk|LLM-jp>",
+  "bos_token": "<s|LLM-jp>",
+  "eos_token": "<EOD|LLM-jp>",
+  "pad_token": "<pad|LLM-jp>",
+  "cls_token": "<CLS|LLM-jp>",
+  "sep_token": "<SEP|LLM-jp>",
+  "eod_token": "<EOD|LLM-jp>",
+  "mask_token": "<mask|LLM-jp>",
+  "extra_ids": 0,
+  "additional_special_tokens": [
+    "</s|LLM-jp>"
+  ],
+  "sp_model_kwargs": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "clean_up_tokenization_spaces": false,
+  "special_tokens_map_file": null,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}
				`@@ -0,0 +1 @@`
				`{"framework": "pytorch", "task": "text-generation", "allow_remote": true}`