Jarvis_granite_4.1_3b_finetuned_16bit/README.md

base_model, tags, license, language, datasets, metrics, model-index

base_model	tags	license	language	datasets	metrics	model-index
unsloth/granite-4.1-3b	text-generation-inference transformers unsloth granite jarvis persona fine-tuned qlora ddp	apache-2.0	en	roger33303/jarvis_deep_dataset	bertscore bleu rouge	name results Jarvis_granite_4.1_3b_finetuned_16bit task dataset metrics type name text-generation Text Generation name type split args roger33303/jarvis_deep_dataset roger33303/jarvis_deep_dataset validation num_few_shot 0 type value name verified bertscore 0.8641 BERTScore F1 false type value name verified bleu 0.1555 BLEU false type value name verified rouge 0.2408 ROUGE-L false

Jarvis - Granite 4.1 3B Fine-tuned (16bit)

A JARVIS-persona fine-tune of IBM's Granite 4.1 3B, trained to respond in the dry, formal, and subtly witty style of J.A.R.V.I.S. from the Iron Man universe.

"I am fully operational and, as ever, at your disposal, sir."

Evaluation Results

Evaluated on the full 6,983-row validation split using the HuggingFace evaluate library. Inference was run using vLLM (tensor_parallel_size=2) on 2x NVIDIA T4 GPUs.

To properly measure the lift from fine-tuning, the base model (unsloth/granite-4.1-3b, no fine-tuning) was also evaluated on the identical validation set, prompted with the same JARVIS system prompt used to generate the training data. This isolates what fine-tuning actually contributed versus what prompting alone can achieve.

Metric	Base + system prompt	Fine-tuned 16bit	Notes
BERTScore F1	0.8420	0.8641	Primary metric: semantic and stylistic alignment
BLEU	0.0806	0.1555	Lexical overlap with reference outputs
ROUGE-L	0.1803	0.2408	Longest common subsequence recall
Persona adherence delta	-0.1174	+0.0093	Model vs. reference persona score (0 is ideal)
Length ratio	2.390	1.232	Model output length divided by reference length (1.0 is ideal)
Self-BLEU	0.1017	0.0584	Lower indicates healthier diversity, less templated output

Why BERTScore is the primary metric: for persona fine-tuning, lexical overlap (BLEU/ROUGE) is a poor proxy for style fidelity. BERTScore captures semantic and contextual alignment at the embedding level, making it far more sensitive to whether the model has genuinely adopted the JARVIS voice rather than just copying words.

Interpretation:

• Persona delta is the clearest signal of fine-tuning's impact. The base model, even when given the exact JARVIS system prompt, undershoots the target persona by -0.117, meaning it inconsistently uses "sir," contractions, and JARVIS-style phrasing. Fine-tuning flips this to +0.009, essentially matching the target persona intensity.
• Length ratio tells a similar story. The base model is 2.39 times more verbose than the reference responses, rambling without fine-tuned guidance on appropriate response length. Fine-tuning brings this down to 1.232, much closer to the target.
• BERTScore F1 improved by 0.022, confirming genuine semantic and stylistic alignment gains, not just surface-level word overlap.
• Self-BLEU dropped from 0.1017 to 0.0584, meaning fine-tuned outputs are less repetitive and templated than the base model's. The base model leans on generic catchphrases when prompted into a persona it has not learned.
• A BERTScore F1 of 0.864 at only approximately 35% training completion signals strong style transfer with room to improve further on a full training run.

Base vs. fine-tuned 16bit, all metrics

Length ratio, base vs. fine-tuned 16bit

Persona delta, base vs. fine-tuned 16bit

The JARVIS Persona

The model was fine-tuned to follow a 10-rule JARVIS dialect style guide. Every training example was verified against these rules.

1. Always addresses the user as "sir," with no exceptions
2. No contractions: "I am" not "I'm," "do not" not "don't," "cannot" not "can't"
3. Formal vocabulary substitutions: "ascertain" over "find out," "endeavour" over "try," "assist" over "help"
4. Six dry wit archetypes:
   • Understatement: describing catastrophic events with mild detachment
   • Deadpan observation: noting obvious things with academic precision
   • Polite sarcasm: technically correct but pointed
   • Reluctant compliance: following absurd instructions with dignified resignation
   • Ironic precision: pedantic accuracy that highlights absurdity
   • Understated concern: alarm expressed through studied calm
5. Response length matching: short inputs get short outputs; complex queries get proportionate depth
6. Sentence structure preferences: subordinate clauses, measured pacing, no bullet-point lists for conversational responses
7. Banned casual words: "okay," "sure," "awesome," "great," "totally," "literally," "basically," "cool," "wow," "yeah," and similar
8. Out-of-character handling: breaks persona gracefully when genuinely unable to help, without breaking immersion unnecessarily
9. Tone calibration by situation: crisis is focused and efficient, casual engages dry wit, technical is precise and formal
10. Complete examples: every rule illustrated with before and after examples in the training data generation prompt

Example exchange:

User: What is the weather like today?

JARVIS: I am afraid I do not have access to your current location's
meteorological data at this moment, sir. You may wish to consult a
dedicated weather service, or if you provide your coordinates I can
make the appropriate enquiry on your behalf.

Model Details

Property	Value
Base model	unsloth/granite-4.1-3b
Architecture	Granite 4.1, 3.40B parameters, 40 layers
Fine-tuning method	QLoRA (4-bit quantization during training)
LoRA rank	r=16, alpha=32
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, shared_mlp.input_linear, shared_mlp.output_linear
Max sequence length	1536 tokens
Training steps completed	3,000 of 8,476 (approximately 35% of full run)
Saved as	bfloat16 merged weights (full model, no adapter required)
License	Apache 2.0
Developed by	roger33303

Training Dataset

Fine-tuned on roger33303/jarvis_deep_dataset, a 69,836-row instruction-response dataset constructed by rewriting WizardLMTeam/WizardLM_evol_instruct_70k into the JARVIS persona using the DeepSeek API with 400 concurrent async workers and fault-tolerant resume logic.

Split	Rows
Train	60,853
Validation	6,983
Test	2,000
Total	69,836

Splits were constructed using quantile-based stratified sampling on token length (n_bins=3, boundaries via np.percentile) to ensure balanced representation of short, medium, and long instructions across all splits. No hardcoded length boundaries; fully data-driven.

Training Configuration

Training was performed using Unsloth and HuggingFace TRL with Accelerate DDP across 2x NVIDIA T4 GPUs on Kaggle.

SFTConfig(
    per_device_train_batch_size = 1,       # DDP stability (Unsloth known issue with batch>1)
    gradient_accumulation_steps = 4,       # effective batch = 1 x 4 x 2 GPUs = 8
    num_train_epochs            = 1,
    learning_rate               = 1e-4,
    optim                       = "adamw_8bit",
    lr_scheduler_type           = "cosine",
    warmup_steps                = 100,
    weight_decay                = 0.01,
    eval_strategy               = "steps",
    eval_steps                  = 500,     # roughly 17 evals across the run
    save_steps                  = 250,
    max_seq_length              = 1536,
    ddp_find_unused_parameters  = False,   # required for DDP + LoRA
    mixed_precision             = "bf16",
)

DDP setup (Accelerate config):

compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
num_processes: 2
num_machines: 1
mixed_precision: bf16
use_cpu: false

Key DDP requirements:

• device_map must not be passed to FastLanguageModel.from_pretrained, as it conflicts with DDP process placement
• All logging, hub pushes, and checkpoint saves gated to PartialState().is_main_process
• ddp_find_unused_parameters=False required for LoRA and DDP to avoid hanging

Throughput: approximately 4.34 s/it (2x T4 DDP) versus approximately 7.9 s/it (single T4), roughly a 1.8x speedup.

Training on responses only: used train_on_responses_only() with Granite's role tokens.

instruction_part = "<|start_of_role|>user<|end_of_role|>"
response_part    = "<|start_of_role|>assistant<|end_of_role|>"

Checkpoint resume: automatically detects latest checkpoint in output_dir and resumes, with no manual configuration needed between Kaggle sessions.

Usage

With Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "roger33303/Jarvis_granite_4.1_3b_finetuned_16bit"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain quantum entanglement."}]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
)

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

With vLLM (recommended for multi-GPU inference)

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("roger33303/Jarvis_granite_4.1_3b_finetuned_16bit")

llm = LLM(
    model="roger33303/Jarvis_granite_4.1_3b_finetuned_16bit",
    tensor_parallel_size=2,
    max_model_len=1536,
    gpu_memory_utilization=0.85,
    dtype="float16",   # T4 does not support bfloat16
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

messages = [{"role": "user", "content": "What is the status of the reactor?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text.strip())

Reproducibility

All code used to train and evaluate this model is publicly available.

Resource	Link
Training notebook (with outputs)	Kaggle: parallel-training-on-multiple-gpus
Evaluation notebook (with outputs)	Kaggle: vllm-evaluate-16-bit-model-evaluation
Training dataset	roger33303/jarvis_deep_dataset

Evaluation Pipeline

The evaluation pipeline used vLLM for fast batched inference (batch_size=64) with checkpoint-resume support, then computed all metrics using the HuggingFace evaluate library.

Persona adherence score is a composite metric:

score = (
    0.35 * has_sir          +   # addresses user as "sir"
    0.30 * no_contractions  +   # avoids contractions
    0.20 * no_banned_words  +   # avoids casual vocabulary
    0.15 * has_preferred_phrase # uses formal JARVIS phrases
)

The persona delta (model score minus reference score) of +0.009 indicates the model matches the target persona intensity, neither undershooting nor overshooting into caricature.

The base model baseline was generated using the identical JARVIS system prompt (from the dataset-creation pipeline's prompt_lib.py) paired directly with each validation instruction, with no fine-tuning and no rewrite wrapper, to test whether prompting alone can achieve the persona that fine-tuning was designed to teach.

Limitations

• Training was stopped at step 3,000 of 8,476 (approximately 35%) due to Kaggle session timeout. A full training run is expected to further improve all metrics.
• The model was trained on instruction-following data rewritten into the JARVIS persona. It may occasionally break character on highly technical or ambiguous queries.
• Outputs tend to be approximately 23% longer than reference responses. This is consistent with JARVIS's verbosity but may be reduced with additional training.
• The base model (Granite 4.1 3B) has a knowledge cutoff; the fine-tune does not extend factual knowledge.

Acknowledgements

Trained with Unsloth for memory-efficient QLoRA fine-tuning on Kaggle's free GPU tier.