初始化项目，由ModelHub XC社区提供模型

Model: KnutJaegersberg/megatron-GPT-2-345m-EvolInstruct
Source: Original Platform

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,42 @@
+---
+license: cc-by-nc-4.0
+---
+
+Fine tuned small megatron-GPT-2 on Evol-Instruct dataset of WizardLM. 
+Megatron-GPT-2-345m is competetive with GPT-2 large in some benchmarks. 
+Megatron-GPT-2-345m performance on Wikitext-103 is 19.31, GPT2-large (1.5B) is 17.48 and GPT2-medium (762M) is 22.05. 
+
+Code example: 
+
+from transformers import AutoModel, GPT2LMHeadModel, AutoTokenizer
+
+model2 = GPT2LMHeadModel.from_pretrained('KnutJaegersberg/megatron-GPT-2-345m-EvolInstruct')
+
+tokenizer = AutoTokenizer.from_pretrained("KnutJaegersberg/megatron-GPT-2-345m-EvolInstruct")
+
+
+text = "Below is an instruction that describes a task. Write a response that appropriately completes the request. \n\n### Instruction:\n Give me a list of 5 political commentators, compare and contrast their beliefs. \n### Response:\n"
+encoded_input = tokenizer(text, return_tensors='pt').input_ids.to('cuda:0')
+
+
+from transformers import pipeline
+generator = pipeline('text-generation', model=model2, tokenizer=tokenizer, device=0)
+
+generator(text, penalty_alpha=0.6, top_k=4, max_length=1000)
+
+
+output: 
+"1. Rush Limbaugh - Religious conservative\n2. Bernard Adjaye - African American\n3. Juan Williams - Hispanic\n4. Rachel Maddow - Journalist\n5. Bill O'Reilly - Political Commentator\nLimbaugh and Adjaye are both religious conservative politicians, but they differ in their beliefs on issues such as abortion, same-sex marriage, and immigration. Adjaye is a strong advocate for gun rights and has been criticized for his pro-choice stance on social issues. However, he has also been critical of President Trump's handling of the COVID-19 pandemic and has suggested that the country needs to focus on mental health issues rather than politics.\nMaddow, on the other hand, is a journalist and talk show host who has made a name for herself in the media. She has been critical of Trump's handling of the COVID-19 pandemic and has suggested that the country needs to focus on mental health issues rather than politics. However, she has also been critical of the government's response to the pandemic and has suggested that the government needs to do more to address the issue.\nO'Reilly, on the other hand, is a political commentary and talk show host who has made a name for himself in the media. He has been critical of Trump's handling of the COVID-19 pandemic and has suggested that the country needs to focus on mental health issues rather than politics. However, he has also been critical of the government's response to the pandemic and has suggested that the country needs to take care of its citizens who are sick.\nOverall, each of these political commentators has their own unique perspective on the political landscape, and their beliefs are shaped by a variety of factors, including their religious beliefs, political affiliations, and personal experiences. It is important to note that these opinions are not universal and may vary across different demographics and regions of the country. Therefore, it is always best to consult with a qualified political analyst or news organization for a comprehensive understanding of the political landscape. Additionally, it is important to be respectful of others' opinions and not try to influence them. By doing so, we can work together to create a more just and equitable society for all.\nSources:\nLimbaugh, R. (2020). The rise of religion in America. Christianity Today, www.cchurch.com/content/dam/2021/08/the-rise-of-religion-in-america. Retrieved from https://www. ChristianityToday.com/blog/how-religion-is-becoming-a-part-of-america/\nAdjaye, B. (2020). Black Lives Matter: A Call to Action. National Book Critics, www.nrdc.org/books/britannica/article/2020/08/black-lives-matter-a-call-to-action.html\nWright, J. (2020). Climate change and the economy. American Psychological Association, www.apa.org/publication/climate-change-and-economy/2020/08/council-member-wright-jeff-kincaid-reviews-opinions-on-policies-to-reform-climate-change.html\nMegan, M. (2020). The future of healthcare: What we know and don't know. Healthline, www.healthline.com/healthline/2020/08/what-we-know-and-don't-know.html\nO'Reilly, R. (2020). Donald Trump's presidency. Fox News, www.foxnews.com/politics/presidential-race.mp3\nMaddow, R. (2020). The media is biased against the right wing. The New York Times, www.nytimes.com/2020/08/29/us/politics/the-media-is-biased-against-the-right-wing.html\nO'Reilly, R. (2020). The 2020 U.S. presidential election. CNN, www.cnn.com/2020/08/29/us/politics/the-2020-presidential-election.html\nMaddow, M. (2020). The COVID-19 pandemic is a wake-up call for the world. The Wall Street Journal, www.bloomberg.com/news/2020/08/causes-and-benefits-of-the-coVID-19-vaccine.html\nO'Reilly, R. (2020). It's time to get"
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_KnutJaegersberg__megatron-GPT-2-345m-EvolInstruct)
+
+| Metric                | Value                     |
+|-----------------------|---------------------------|
+| Avg.                  | 26.35   |
+| ARC (25-shot)         | 24.06          |
+| HellaSwag (10-shot)   | 35.12    |
+| MMLU (5-shot)         | 24.48         |
+| TruthfulQA (0-shot)   | 41.25   |
+| Winogrande (5-shot)   | 54.78   |
+| GSM8K (5-shot)        | 0.38        |
+| DROP (3-shot)         | 4.39         |

config.json

@@ -0,0 +1,34 @@
+{
+  "_name_or_path": "robowaifudev/megatron-gpt2-345m",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "gradient_checkpointing": false,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": 4096,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.29.0.dev0",
+  "use_cache": true,
+  "vocab_size": 50257
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.29.0.dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e78322d3f378c081991089541520869bea187755f3c93475c44265fe106c651
+size 1444493800

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6f675ed4b0cd2dc42c7d3940e9d9c9ceb4baccd55acdaffe910e8688a3bc54a
+size 1444569373

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04cd1421600ec4effb75934f047af73b38bbdb8feeae142b7ede71a9daa62fa1
+size 3963