Batch Size: 4 (effective 16 with gradient accumulation)
Usage
fromtransformersimportAutoTokenizer,AutoModelForCausalLMmodel_name="gitcoreai/gpt-2-small-sft"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForCausalLM.from_pretrained(model_name)prompt="user: hello, how are you?\nassistant: <s>"inputs=tokenizer(prompt,return_tensors="pt")outputs=model.generate(**inputs,max_new_tokens=60)print(tokenizer.decode(outputs[0],skip_special_tokens=True))