======================================================================================== QUINTUS WEIGHT AUDIT ======================================================================================== base model : Qwen/Qwen3-1.7B-Base distilled model : iamrahulreddy/Quintus alpha : 0.3 device : cuda | dtype: torch.bfloat16 python : 3.12.13 | torch: 2.11.0+cu128 ======================================================================================== [01] Resolve checkpoints Downloading 'iamrahulreddy/Quintus' from HuggingFace Hub... Done in 0.2s base model commit : distilled commit : snapshot root : /snapshots Filename Size Modified .gitattributes 1.533 KiB 2026-06-14 09:19 README.md 5.388 KiB 2026-06-14 09:19 added_tokens.json 707 B 2026-06-14 09:19 chat_template.jinja 4.070 KiB 2026-06-14 09:19 config.json 1.328 KiB 2026-06-14 09:19 generation_config.json 143 B 2026-06-14 09:19 model.safetensors 3.205 GiB 2026-06-14 09:21 special_tokens_map.json 613 B 2026-06-14 09:19 tokenizer.json 10.893 MiB 2026-06-14 09:19 tokenizer_config.json 5.277 KiB 2026-06-14 09:19 vocab.json 2.648 MiB 2026-06-14 09:19 total 3.218 GiB [02] Architecture configuration Loading base config... Loading distilled config... -- Base label : base (Qwen/Qwen3-1.7B-Base) model_type : qwen3 architecture : Qwen3ForCausalLM Vocabulary vocab_size : 151,936 bos / eos / pad : 151643 / 151643 / None Positional encoding max_position_embeddings: 32,768 rope_theta : N/A rope_scaling : None Transformer dimensions hidden_size : 2048 num_hidden_layers : 28 intermediate_size : 6144 Attention num_attention_heads : 16 num_key_value_heads : 8 head_dim : 128 GQA ratio : 2:1 attention_bias : False use_qk_norm : True sliding_window : None Feed-forward hidden_act : silu mlp_bias : False Misc rms_norm_eps : 1e-06 tie_word_embeddings : True use_cache : True torch_dtype : float32 initializer_range : 0.02 -- Distilled label : distilled (iamrahulreddy/Quintus) model_type : qwen3 architecture : Qwen3ForCausalLM Vocabulary vocab_size : 151,936 bos / eos / pad : 151643 / 151643 / None Positional encoding max_position_embeddings: 32,768 rope_theta : N/A rope_scaling : None Transformer dimensions hidden_size : 2048 num_hidden_layers : 28 intermediate_size : 6144 Attention num_attention_heads : 16 num_key_value_heads : 8 head_dim : 128 GQA ratio : 2:1 attention_bias : False use_qk_norm : True sliding_window : None Feed-forward hidden_act : silu mlp_bias : False Misc rms_norm_eps : 1e-06 tie_word_embeddings : True use_cache : True torch_dtype : float32 initializer_range : 0.02 -- Config diff (ignoring: _name_or_path, transformers_version) No differences — configs identical (expected for same-architecture KD). [03] Parameter accounting -- Base base raw (all named) : 2,031,739,904 (2.031740 B) embedding : 311,164,928 (311.164928 M) lm_head : 311,164,928 (311.164928 M) tied : True unique (deduped) : 1,720,574,976 (1.720575 B) non-embedding : 1,409,410,048 (1.409410 B) per layer (approx) : 50,336,000 -- Distilled distilled raw (all named) : 2,031,739,904 (2.031740 B) embedding : 311,164,928 (311.164928 M) lm_head : 311,164,928 (311.164928 M) tied : True unique (deduped) : 1,720,574,976 (1.720575 B) non-embedding : 1,409,410,048 (1.409410 B) per layer (approx) : 50,336,000 -- Delta unique param delta : +0 (+0.0000 %) non-embed param delta : +0 [04] Load weights device: cuda | dtype: torch.bfloat16 Loading base model : Qwen/Qwen3-1.7B-Base Done in 15.4s Loading distilled : iamrahulreddy/Quintus Done in 13.1s base tensors : 311 distilled tensors : 311 weight tying confirmed (embed == lm_head): True base weight memory : 3.784 GiB distilled memory : 3.784 GiB [05] Per-tensor weight statistics (distilled) Layer Shape Mean Std Min Max Sparse KurtD OutlR RowL2 DeadR -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- model.embed_tokens.weight [151936, 2048] -0.0001 0.0354 -0.3613 0.2988 0.0000 -0.00 0.0003 1.5789 0 model.layers.0.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0356 -0.6641 0.6680 0.0000 -0.00 0.0017 1.5730 0 model.layers.0.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0362 -0.8320 0.6172 0.0000 -0.00 0.0019 1.5972 0 model.layers.0.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0340 -0.2275 0.2227 0.0000 -0.00 0.0005 1.5332 0 model.layers.0.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0328 -0.4512 0.5156 0.0000 -0.00 0.0031 1.4652 0 model.layers.0.self_attn.q_norm.weight [128] 1.6407 0.8456 -0.4355 6.9688 0.0000 0.00 0.0156 nan N/A model.layers.0.self_attn.k_norm.weight [128] 3.0374 6.5282 0.0679 68.0000 0.0000 0.00 0.0234 nan N/A model.layers.0.mlp.gate_proj.weight [6144, 2048] 0.0002 0.0390 -0.6484 0.6367 0.0000 -0.00 0.0012 1.7427 0 model.layers.0.mlp.up_proj.weight [6144, 2048] -0.0000 0.0318 -0.4160 0.3945 0.0000 -0.00 0.0004 1.4374 0 model.layers.0.mlp.down_proj.weight [2048, 6144] 0.0000 0.0336 -0.6367 0.5156 0.0000 -0.00 0.0007 2.6247 0 model.layers.0.input_layernorm.weight [2048] 0.0895 0.0521 0.0576 1.0078 0.0000 0.00 0.0142 nan N/A model.layers.0.post_attention_layernorm.weight [2048] 0.1976 0.1459 -0.6406 0.8047 0.0000 0.00 0.0005 nan N/A model.layers.1.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0349 -0.5039 0.6094 0.0000 -0.00 0.0027 1.5187 0 model.layers.1.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0341 -0.5078 0.4082 0.0000 -0.00 0.0022 1.4952 0 model.layers.1.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0350 -0.2305 0.2617 0.0000 -0.00 0.0005 1.5767 0 model.layers.1.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0329 -0.3398 0.3711 0.0000 -0.00 0.0011 1.4763 0 model.layers.1.self_attn.q_norm.weight [128] 1.4082 0.7994 -2.1875 4.5938 0.0000 0.00 0.0078 nan N/A model.layers.1.self_attn.k_norm.weight [128] 2.1907 2.8109 -2.9688 30.6250 0.0000 0.00 0.0078 nan N/A model.layers.1.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0453 -0.4043 0.5273 0.0000 -0.00 0.0007 1.9884 0 model.layers.1.mlp.up_proj.weight [6144, 2048] -0.0000 0.0289 -0.4688 0.5234 0.0000 -0.00 0.0014 1.2540 0 model.layers.1.mlp.down_proj.weight [2048, 6144] -0.0000 0.0294 -0.6289 0.5312 0.0000 -0.00 0.0014 2.2993 0 model.layers.1.input_layernorm.weight [2048] 0.0283 0.0940 -0.3125 0.5703 0.0000 0.00 0.0029 nan N/A model.layers.1.post_attention_layernorm.weight [2048] 0.5232 0.1638 -1.1953 5.2500 0.0000 -0.00 0.0068 nan N/A model.layers.2.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0347 -0.5156 0.6406 0.0000 -0.00 0.0029 1.5091 0 model.layers.2.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0338 -0.2910 0.3613 0.0000 -0.00 0.0022 1.4704 0 model.layers.2.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0349 -0.2119 0.2422 0.0000 -0.00 0.0007 1.5588 0 model.layers.2.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0319 -0.4121 0.4414 0.0000 -0.00 0.0018 1.4350 0 model.layers.2.self_attn.q_norm.weight [128] 1.4967 0.7808 -0.1245 4.9688 0.0000 0.00 0.0156 nan N/A model.layers.2.self_attn.k_norm.weight [128] 2.3197 3.2107 -0.0057 35.5000 0.0000 0.00 0.0078 nan N/A model.layers.2.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0469 -0.4805 0.5000 0.0000 -0.00 0.0005 2.0895 0 model.layers.2.mlp.up_proj.weight [6144, 2048] 0.0000 0.0299 -0.5859 0.7031 0.0000 -0.00 0.0007 1.3430 0 model.layers.2.mlp.down_proj.weight [2048, 6144] -0.0000 0.0309 -0.8203 0.6445 0.0000 -0.00 0.0007 2.4146 0 model.layers.2.input_layernorm.weight [2048] 0.1257 0.0469 -0.2061 0.6133 0.0000 -0.00 0.0210 nan N/A model.layers.2.post_attention_layernorm.weight [2048] 0.5019 0.1475 -0.5039 5.5000 0.0000 0.13 0.0029 nan N/A model.layers.3.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0352 -0.6797 0.5078 0.0000 -0.00 0.0031 1.5280 0 model.layers.3.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0345 -0.6484 0.7969 0.0000 -0.00 0.0025 1.4915 0 model.layers.3.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0371 -0.3926 0.4844 0.0000 -0.00 0.0007 1.6652 0 model.layers.3.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0338 -0.4004 0.4453 0.0000 -0.00 0.0012 1.5201 0 model.layers.3.self_attn.q_norm.weight [128] 1.8103 0.8711 0.0004 7.0000 0.0000 0.00 0.0156 nan N/A model.layers.3.self_attn.k_norm.weight [128] 1.7619 1.9032 -0.0018 20.2500 0.0000 -0.00 0.0078 nan N/A model.layers.3.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0463 -0.5352 0.4258 0.0000 -0.00 0.0005 2.0712 0 model.layers.3.mlp.up_proj.weight [6144, 2048] 0.0000 0.0306 -0.4414 0.4688 0.0000 -0.00 0.0005 1.3758 0 model.layers.3.mlp.down_proj.weight [2048, 6144] 0.0000 0.0308 -0.4160 0.6641 0.0000 -0.00 0.0005 2.4052 0 model.layers.3.input_layernorm.weight [2048] 0.3191 0.1033 -0.6133 1.5625 0.0000 0.00 0.0229 nan N/A model.layers.3.post_attention_layernorm.weight [2048] 0.5742 0.1119 -0.5938 2.0000 0.0000 0.00 0.0068 nan N/A model.layers.4.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0346 -0.4961 0.4805 0.0000 -0.00 0.0028 1.5056 0 model.layers.4.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0334 -0.5742 0.4141 0.0000 -0.00 0.0023 1.4488 0 model.layers.4.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0364 -0.2402 0.2441 0.0000 -0.00 0.0006 1.6342 0 model.layers.4.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0331 -0.6250 0.6641 0.0000 -0.00 0.0013 1.4884 0 model.layers.4.self_attn.q_norm.weight [128] 1.0624 0.5927 0.0052 3.7500 0.0000 0.00 0.0078 nan N/A model.layers.4.self_attn.k_norm.weight [128] 3.1512 3.6376 -0.0025 36.5000 0.0000 0.00 0.0156 nan N/A model.layers.4.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0449 -0.6172 0.6328 0.0000 -0.00 0.0005 2.0082 0 model.layers.4.mlp.up_proj.weight [6144, 2048] 0.0000 0.0309 -0.4668 0.4395 0.0000 -0.00 0.0005 1.3882 0 model.layers.4.mlp.down_proj.weight [2048, 6144] -0.0000 0.0308 -0.5898 0.6328 0.0000 -0.00 0.0006 2.4082 0 model.layers.4.input_layernorm.weight [2048] 0.3279 0.0863 -0.5820 1.2969 0.0000 0.00 0.0186 nan N/A model.layers.4.post_attention_layernorm.weight [2048] 0.6251 0.1225 -0.8164 1.6094 0.0000 -0.00 0.0068 nan N/A model.layers.5.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0358 -0.4375 0.5195 0.0000 -0.00 0.0026 1.5625 0 model.layers.5.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0339 -0.5469 0.5781 0.0000 -0.00 0.0016 1.4858 0 model.layers.5.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0353 -0.2891 0.2656 0.0000 -0.00 0.0014 1.5872 0 model.layers.5.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0333 -0.4961 0.4395 0.0000 -0.00 0.0024 1.4987 0 model.layers.5.self_attn.q_norm.weight [128] 1.5458 1.0519 -0.0007 6.2812 0.0000 -0.00 0.0312 nan N/A model.layers.5.self_attn.k_norm.weight [128] 2.4593 3.5832 0.0013 31.1250 0.0000 0.00 0.0234 nan N/A model.layers.5.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0400 -0.7227 0.7539 0.0000 -0.00 0.0009 1.7845 0 model.layers.5.mlp.up_proj.weight [6144, 2048] 0.0000 0.0331 -0.6602 0.5352 0.0000 -0.00 0.0004 1.4943 0 model.layers.5.mlp.down_proj.weight [2048, 6144] -0.0000 0.0329 -0.6211 0.6758 0.0000 -0.00 0.0006 2.5677 0 model.layers.5.input_layernorm.weight [2048] 0.3916 0.0926 -0.9727 1.4219 0.0000 0.00 0.0210 nan N/A model.layers.5.post_attention_layernorm.weight [2048] 0.5236 0.0931 -0.3320 0.9688 0.0000 -0.00 0.0029 nan N/A model.layers.6.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0349 -0.4180 0.5859 0.0000 -0.00 0.0049 1.4738 0 model.layers.6.self_attn.k_proj.weight [1024, 2048] 0.0001 0.0331 -0.5781 0.6094 0.0000 -0.00 0.0032 1.3997 0 model.layers.6.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0360 -0.2168 0.2207 0.0000 -0.00 0.0007 1.6123 0 model.layers.6.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0320 -0.5039 0.7539 0.0000 -0.00 0.0015 1.4365 0 model.layers.6.self_attn.q_norm.weight [128] 1.7113 1.0610 -0.0048 7.0312 0.0000 -0.00 0.0156 nan N/A model.layers.6.self_attn.k_norm.weight [128] 2.4032 2.1293 -0.0004 19.2500 0.0000 0.00 0.0156 nan N/A model.layers.6.mlp.gate_proj.weight [6144, 2048] -0.0001 0.0394 -0.7031 0.6484 0.0000 -0.00 0.0010 1.7528 0 model.layers.6.mlp.up_proj.weight [6144, 2048] -0.0000 0.0333 -0.4863 0.3652 0.0000 -0.00 0.0005 1.4986 0 model.layers.6.mlp.down_proj.weight [2048, 6144] -0.0000 0.0324 -0.8672 0.8359 0.0000 -0.00 0.0008 2.5322 0 model.layers.6.input_layernorm.weight [2048] 0.3556 0.1117 -0.5938 1.3750 0.0000 0.00 0.0254 nan N/A model.layers.6.post_attention_layernorm.weight [2048] 0.5847 0.1243 -0.6328 1.2656 0.0000 0.00 0.0034 nan N/A model.layers.7.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0338 -0.4238 0.3887 0.0000 -0.00 0.0025 1.4765 0 model.layers.7.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0319 -0.3340 0.2617 0.0000 -0.00 0.0019 1.3969 0 model.layers.7.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0350 -0.2578 0.3203 0.0000 -0.00 0.0007 1.5745 0 model.layers.7.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0322 -0.5742 0.5039 0.0000 -0.00 0.0012 1.4427 0 model.layers.7.self_attn.q_norm.weight [128] 1.5685 0.9296 0.0188 6.5312 0.0000 0.00 0.0156 nan N/A model.layers.7.self_attn.k_norm.weight [128] 1.8889 2.0411 -0.0031 19.2500 0.0000 -0.00 0.0078 nan N/A model.layers.7.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0406 -0.4727 0.5078 0.0000 -0.00 0.0008 1.7994 0 model.layers.7.mlp.up_proj.weight [6144, 2048] 0.0000 0.0328 -0.5430 0.4570 0.0000 -0.00 0.0006 1.4743 0 model.layers.7.mlp.down_proj.weight [2048, 6144] 0.0000 0.0319 -0.9805 0.6133 0.0000 -0.00 0.0007 2.4845 0 model.layers.7.input_layernorm.weight [2048] 0.5572 0.1534 -0.5820 3.5469 0.0000 0.00 0.0107 nan N/A model.layers.7.post_attention_layernorm.weight [2048] 0.6644 0.1445 -0.6914 1.5234 0.0000 -0.00 0.0029 nan N/A model.layers.8.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0347 -0.4434 0.4473 0.0000 -0.00 0.0042 1.4826 0 model.layers.8.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0326 -0.8008 0.8750 0.0000 -0.00 0.0031 1.4049 0 model.layers.8.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0354 -0.1914 0.2285 0.0000 -0.00 0.0004 1.5972 0 model.layers.8.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0316 -0.5039 0.4746 0.0000 -0.00 0.0010 1.4136 0 model.layers.8.self_attn.q_norm.weight [128] 1.4452 0.5422 -0.0425 4.6562 0.0000 -0.00 0.0078 nan N/A model.layers.8.self_attn.k_norm.weight [128] 1.6409 0.8953 -0.0139 7.6875 0.0000 0.00 0.0078 nan N/A model.layers.8.mlp.gate_proj.weight [6144, 2048] -0.0001 0.0388 -0.8125 0.8789 0.0000 -0.00 0.0010 1.7171 0 model.layers.8.mlp.up_proj.weight [6144, 2048] -0.0000 0.0332 -0.5000 0.4355 0.0000 -0.00 0.0005 1.4933 0 model.layers.8.mlp.down_proj.weight [2048, 6144] 0.0000 0.0328 -0.6406 0.5391 0.0000 -0.00 0.0008 2.5532 0 model.layers.8.input_layernorm.weight [2048] 0.6002 0.1623 0.0991 1.9766 0.0000 0.00 0.0200 nan N/A model.layers.8.post_attention_layernorm.weight [2048] 0.6617 0.1351 -0.3613 1.3203 0.0000 0.00 0.0010 nan N/A model.layers.9.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0351 -0.4961 0.4453 0.0000 -0.00 0.0042 1.5042 0 model.layers.9.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0324 -0.5508 0.5547 0.0000 -0.00 0.0024 1.4137 0 model.layers.9.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0347 -0.3086 0.3320 0.0000 -0.00 0.0012 1.5503 0 model.layers.9.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0327 -0.5547 0.5078 0.0000 -0.00 0.0025 1.4583 0 model.layers.9.self_attn.q_norm.weight [128] 1.6585 1.1838 -0.0251 6.7812 0.0000 0.00 0.0234 nan N/A model.layers.9.self_attn.k_norm.weight [128] 1.9812 1.8305 -0.0009 12.3750 0.0000 -0.00 0.0312 nan N/A model.layers.9.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0386 -0.5586 0.5000 0.0000 -0.00 0.0011 1.7093 0 model.layers.9.mlp.up_proj.weight [6144, 2048] -0.0000 0.0346 -0.4746 0.3633 0.0000 -0.00 0.0006 1.5537 0 model.layers.9.mlp.down_proj.weight [2048, 6144] -0.0000 0.0330 -0.8047 0.5430 0.0000 -0.00 0.0009 2.5660 0 model.layers.9.input_layernorm.weight [2048] 0.7933 0.2096 0.0057 3.0312 0.0000 0.00 0.0098 nan N/A model.layers.9.post_attention_layernorm.weight [2048] 0.6711 0.1426 -0.0014 1.3750 0.0000 -0.00 0.0010 nan N/A model.layers.10.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0346 -0.4316 0.3926 0.0000 -0.00 0.0026 1.5106 0 model.layers.10.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0332 -0.2432 0.2676 0.0000 -0.00 0.0018 1.4593 0 model.layers.10.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0345 -0.2217 0.1992 0.0000 -0.00 0.0006 1.5555 0 model.layers.10.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0319 -0.4688 0.5625 0.0000 -0.00 0.0016 1.4223 0 model.layers.10.self_attn.q_norm.weight [128] 1.7754 1.3316 0.0095 8.4375 0.0000 0.00 0.0312 nan N/A model.layers.10.self_attn.k_norm.weight [128] 1.8918 2.0176 -0.0007 20.7500 0.0000 0.00 0.0156 nan N/A model.layers.10.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0381 -0.7266 0.6289 0.0000 -0.00 0.0016 1.6789 0 model.layers.10.mlp.up_proj.weight [6144, 2048] -0.0000 0.0347 -0.6133 0.6055 0.0000 -0.00 0.0008 1.5587 0 model.layers.10.mlp.down_proj.weight [2048, 6144] 0.0000 0.0328 -0.7148 0.6406 0.0000 -0.00 0.0012 2.5447 0 model.layers.10.input_layernorm.weight [2048] 0.7780 0.2307 -0.1816 3.1406 0.0000 0.00 0.0073 nan N/A model.layers.10.post_attention_layernorm.weight [2048] 0.6835 0.1354 -0.0013 1.2891 0.0000 -0.00 0.0015 nan N/A model.layers.11.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0367 -0.5273 0.4961 0.0000 -0.00 0.0055 1.5396 0 model.layers.11.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0335 -0.4043 0.4062 0.0000 -0.00 0.0028 1.4380 0 model.layers.11.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0328 -0.2578 0.3027 0.0000 -0.00 0.0012 1.4683 0 model.layers.11.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0303 -0.6797 0.6797 0.0000 -0.00 0.0028 1.3430 0 model.layers.11.self_attn.q_norm.weight [128] 1.7712 0.6162 -0.0052 3.1250 0.0000 0.00 0.0000 nan N/A model.layers.11.self_attn.k_norm.weight [128] 2.0542 1.0209 -0.0045 5.8125 0.0000 -0.00 0.0156 nan N/A model.layers.11.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0362 -0.5703 0.5586 0.0000 -0.00 0.0013 1.6086 0 model.layers.11.mlp.up_proj.weight [6144, 2048] 0.0000 0.0355 -0.4160 0.4160 0.0000 -0.00 0.0007 1.5934 0 model.layers.11.mlp.down_proj.weight [2048, 6144] 0.0000 0.0339 -0.7695 0.8281 0.0000 -0.00 0.0010 2.6350 0 model.layers.11.input_layernorm.weight [2048] 1.0137 0.4236 0.0009 6.5000 0.0000 0.00 0.0117 nan N/A model.layers.11.post_attention_layernorm.weight [2048] 0.7500 0.2023 -0.6133 2.4375 0.0000 0.00 0.0015 nan N/A model.layers.12.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0351 -0.4121 0.4121 0.0000 -0.00 0.0038 1.5119 0 model.layers.12.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0326 -0.4414 0.4531 0.0000 -0.00 0.0027 1.4140 0 model.layers.12.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0337 -0.2451 0.2676 0.0000 -0.00 0.0007 1.5215 0 model.layers.12.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0303 -0.3926 0.5508 0.0000 -0.00 0.0012 1.3480 0 model.layers.12.self_attn.q_norm.weight [128] 1.7989 0.7431 -0.0101 3.8594 0.0000 0.00 0.0000 nan N/A model.layers.12.self_attn.k_norm.weight [128] 1.7795 1.8132 -0.0039 20.1250 0.0000 0.00 0.0078 nan N/A model.layers.12.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0353 -0.5117 0.4824 0.0000 -0.00 0.0013 1.5627 0 model.layers.12.mlp.up_proj.weight [6144, 2048] 0.0000 0.0354 -0.5273 0.4805 0.0000 -0.00 0.0008 1.5855 0 model.layers.12.mlp.down_proj.weight [2048, 6144] -0.0000 0.0339 -0.8164 0.6250 0.0000 -0.00 0.0010 2.6305 0 model.layers.12.input_layernorm.weight [2048] 1.2048 0.5195 -0.7461 6.4062 0.0000 -0.00 0.0103 nan N/A model.layers.12.post_attention_layernorm.weight [2048] 0.7572 0.2071 -0.6055 1.6484 0.0000 -0.00 0.0020 nan N/A model.layers.13.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0355 -0.4121 0.4414 0.0000 -0.00 0.0057 1.4901 0 model.layers.13.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0311 -0.2969 0.3027 0.0000 -0.00 0.0024 1.3504 0 model.layers.13.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0336 -0.2197 0.2100 0.0000 -0.00 0.0005 1.5133 0 model.layers.13.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0294 -0.4746 0.3711 0.0000 -0.00 0.0011 1.3055 0 model.layers.13.self_attn.q_norm.weight [128] 1.8504 0.9921 -0.1807 6.1875 0.0000 0.00 0.0234 nan N/A model.layers.13.self_attn.k_norm.weight [128] 1.7791 0.8772 -0.0124 4.9062 0.0000 0.00 0.0078 nan N/A model.layers.13.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0341 -0.6523 0.6484 0.0000 -0.00 0.0017 1.5077 0 model.layers.13.mlp.up_proj.weight [6144, 2048] -0.0000 0.0344 -0.4180 0.5859 0.0000 -0.00 0.0008 1.5437 0 model.layers.13.mlp.down_proj.weight [2048, 6144] 0.0000 0.0330 -0.9844 0.6992 0.0000 -0.00 0.0011 2.5615 0 model.layers.13.input_layernorm.weight [2048] 1.3235 0.6034 -0.8984 8.0625 0.0000 0.00 0.0112 nan N/A model.layers.13.post_attention_layernorm.weight [2048] 0.7902 0.2272 -0.0023 1.7266 0.0000 0.00 0.0049 nan N/A model.layers.14.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0344 -0.4062 0.3945 0.0000 -0.00 0.0051 1.4512 0 model.layers.14.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0308 -0.4531 0.5156 0.0000 -0.00 0.0032 1.3290 0 model.layers.14.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0327 -0.1914 0.1934 0.0000 -0.00 0.0006 1.4757 0 model.layers.14.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0294 -0.5195 0.4395 0.0000 -0.00 0.0012 1.3063 0 model.layers.14.self_attn.q_norm.weight [128] 1.7915 1.0480 -0.0293 6.3750 0.0000 0.00 0.0312 nan N/A model.layers.14.self_attn.k_norm.weight [128] 1.8330 1.2059 -0.0398 12.1875 0.0000 -0.00 0.0078 nan N/A model.layers.14.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0326 -0.7891 0.6875 0.0000 -0.00 0.0018 1.4436 0 model.layers.14.mlp.up_proj.weight [6144, 2048] 0.0000 0.0334 -0.5586 0.4570 0.0000 -0.00 0.0009 1.4960 0 model.layers.14.mlp.down_proj.weight [2048, 6144] -0.0000 0.0328 -0.6719 0.6484 0.0000 -0.00 0.0011 2.5438 0 model.layers.14.input_layernorm.weight [2048] 1.6904 0.7990 -0.9961 10.3750 0.0000 -0.00 0.0107 nan N/A model.layers.14.post_attention_layernorm.weight [2048] 0.8251 0.2290 -0.0019 1.9688 0.0000 -0.00 0.0054 nan N/A model.layers.15.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0352 -0.3848 0.4277 0.0000 -0.00 0.0054 1.4810 0 model.layers.15.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0295 -0.2891 0.2969 0.0000 -0.00 0.0024 1.2812 0 model.layers.15.self_attn.v_proj.weight [1024, 2048] 0.0001 0.0326 -0.4355 0.4141 0.0000 -0.00 0.0012 1.4678 0 model.layers.15.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0301 -0.4727 0.4473 0.0000 -0.00 0.0013 1.3374 0 model.layers.15.self_attn.q_norm.weight [128] 1.4976 0.6046 -0.0320 2.9062 0.0000 -0.00 0.0000 nan N/A model.layers.15.self_attn.k_norm.weight [128] 2.2923 2.5574 -0.0043 22.0000 0.0000 -0.00 0.0234 nan N/A model.layers.15.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0328 -0.7383 0.6914 0.0000 -0.00 0.0019 1.4522 0 model.layers.15.mlp.up_proj.weight [6144, 2048] 0.0000 0.0341 -0.4570 0.5547 0.0000 -0.00 0.0010 1.5281 0 model.layers.15.mlp.down_proj.weight [2048, 6144] 0.0000 0.0329 -0.7461 0.7148 0.0000 -0.00 0.0011 2.5531 0 model.layers.15.input_layernorm.weight [2048] 2.3836 0.9875 -1.6250 14.4375 0.0000 0.00 0.0088 nan N/A model.layers.15.post_attention_layernorm.weight [2048] 0.8280 0.1941 -0.0019 1.9141 0.0000 -0.00 0.0078 nan N/A model.layers.16.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0376 -0.4414 0.5039 0.0000 -0.00 0.0076 1.5383 0 model.layers.16.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0306 -1.0312 0.8477 0.0000 -0.00 0.0041 1.2903 0 model.layers.16.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0328 -0.1973 0.2305 0.0000 -0.00 0.0008 1.4732 0 model.layers.16.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0301 -0.5000 0.4258 0.0000 -0.00 0.0016 1.3393 0 model.layers.16.self_attn.q_norm.weight [128] 1.6256 0.6673 -0.1147 3.5000 0.0000 -0.00 0.0000 nan N/A model.layers.16.self_attn.k_norm.weight [128] 2.0342 1.1452 -0.1045 7.8750 0.0000 -0.00 0.0078 nan N/A model.layers.16.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0326 -0.4863 0.5273 0.0000 -0.00 0.0018 1.4504 0 model.layers.16.mlp.up_proj.weight [6144, 2048] -0.0000 0.0350 -0.5742 0.5820 0.0000 -0.00 0.0011 1.5721 0 model.layers.16.mlp.down_proj.weight [2048, 6144] 0.0000 0.0332 -0.7578 0.6016 0.0000 -0.00 0.0014 2.5963 0 model.layers.16.input_layernorm.weight [2048] 2.2985 0.8804 0.0021 16.2500 0.0000 0.00 0.0103 nan N/A model.layers.16.post_attention_layernorm.weight [2048] 0.8429 0.1426 -0.0032 2.1719 0.0000 0.00 0.0122 nan N/A model.layers.17.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0359 -0.3926 0.3906 0.0000 -0.00 0.0052 1.5199 0 model.layers.17.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0305 -0.4199 0.4023 0.0000 -0.00 0.0020 1.3363 0 model.layers.17.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0340 -0.2480 0.2539 0.0000 -0.00 0.0012 1.5262 0 model.layers.17.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0305 -0.4102 0.3633 0.0000 -0.00 0.0020 1.3648 0 model.layers.17.self_attn.q_norm.weight [128] 1.7546 1.1692 -0.0264 9.4375 0.0000 0.00 0.0156 nan N/A model.layers.17.self_attn.k_norm.weight [128] 1.9915 1.5669 -0.0845 16.0000 0.0000 0.00 0.0078 nan N/A model.layers.17.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0325 -0.6172 0.6172 0.0000 -0.00 0.0017 1.4493 0 model.layers.17.mlp.up_proj.weight [6144, 2048] -0.0000 0.0364 -0.3984 0.3887 0.0000 -0.00 0.0010 1.6309 0 model.layers.17.mlp.down_proj.weight [2048, 6144] -0.0000 0.0343 -0.6797 0.5547 0.0000 -0.00 0.0009 2.6680 0 model.layers.17.input_layernorm.weight [2048] 4.0729 1.4421 -2.2188 33.7500 0.0000 -0.00 0.0098 nan N/A model.layers.17.post_attention_layernorm.weight [2048] 1.0375 0.2185 -0.0027 2.7656 0.0000 -0.00 0.0117 nan N/A model.layers.18.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0370 -0.4414 0.4707 0.0000 -0.00 0.0061 1.5392 0 model.layers.18.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0297 -0.4922 0.4395 0.0000 -0.00 0.0026 1.2818 0 model.layers.18.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0321 -0.2461 0.3145 0.0000 -0.00 0.0008 1.4507 0 model.layers.18.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0287 -0.3672 0.4844 0.0000 -0.00 0.0013 1.2860 0 model.layers.18.self_attn.q_norm.weight [128] 1.5350 0.5170 -0.0645 2.6562 0.0000 -0.00 0.0000 nan N/A model.layers.18.self_attn.k_norm.weight [128] 1.8658 0.9406 -0.1133 6.3125 0.0000 -0.00 0.0078 nan N/A model.layers.18.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0329 -0.5273 0.5625 0.0000 -0.00 0.0017 1.4636 0 model.layers.18.mlp.up_proj.weight [6144, 2048] 0.0000 0.0365 -0.3809 0.4062 0.0000 -0.00 0.0010 1.6349 0 model.layers.18.mlp.down_proj.weight [2048, 6144] -0.0000 0.0352 -0.6484 0.7305 0.0000 -0.00 0.0009 2.7382 0 model.layers.18.input_layernorm.weight [2048] 4.1920 1.5139 -2.2500 27.1250 0.0000 0.00 0.0161 nan N/A model.layers.18.post_attention_layernorm.weight [2048] 1.1746 0.2241 -0.0031 3.2031 0.0000 0.00 0.0127 nan N/A model.layers.19.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0382 -0.6016 0.4648 0.0000 -0.00 0.0079 1.5478 0 model.layers.19.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0299 -0.5156 0.4922 0.0000 -0.00 0.0025 1.2970 0 model.layers.19.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0320 -0.2812 0.3906 0.0000 -0.00 0.0021 1.4382 0 model.layers.19.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0298 -0.3457 0.4219 0.0000 -0.00 0.0014 1.3383 0 model.layers.19.self_attn.q_norm.weight [128] 1.6152 0.6862 -0.0435 3.8906 0.0000 0.00 0.0156 nan N/A model.layers.19.self_attn.k_norm.weight [128] 1.8694 1.0279 -0.0339 7.4062 0.0000 -0.00 0.0156 nan N/A model.layers.19.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0334 -0.6719 0.6289 0.0000 -0.00 0.0020 1.4815 0 model.layers.19.mlp.up_proj.weight [6144, 2048] -0.0000 0.0371 -0.9062 0.6562 0.0000 -0.00 0.0011 1.6638 0 model.layers.19.mlp.down_proj.weight [2048, 6144] 0.0000 0.0363 -0.7070 0.6719 0.0000 -0.00 0.0010 2.8286 0 model.layers.19.input_layernorm.weight [2048] 5.4223 2.2384 -0.0017 70.0000 0.0000 -0.00 0.0142 nan N/A model.layers.19.post_attention_layernorm.weight [2048] 1.2960 0.2268 -0.0038 3.7969 0.0000 -0.00 0.0142 nan N/A model.layers.20.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0387 -0.4453 0.5703 0.0000 -0.00 0.0075 1.5911 0 model.layers.20.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0313 -0.3730 0.3574 0.0000 -0.00 0.0024 1.3673 0 model.layers.20.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0333 -0.2812 0.3594 0.0000 -0.00 0.0022 1.4952 0 model.layers.20.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0311 -0.5586 0.6758 0.0000 -0.00 0.0016 1.3934 0 model.layers.20.self_attn.q_norm.weight [128] 1.6370 0.6676 -0.0091 4.4062 0.0000 0.00 0.0156 nan N/A model.layers.20.self_attn.k_norm.weight [128] 1.7924 1.0205 -0.0199 8.2500 0.0000 -0.00 0.0156 nan N/A model.layers.20.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0347 -0.6328 0.4922 0.0000 -0.00 0.0017 1.5405 0 model.layers.20.mlp.up_proj.weight [6144, 2048] -0.0000 0.0381 -0.5312 0.6797 0.0000 -0.00 0.0009 1.7075 0 model.layers.20.mlp.down_proj.weight [2048, 6144] 0.0000 0.0372 -0.6250 0.8047 0.0000 -0.00 0.0008 2.8967 0 model.layers.20.input_layernorm.weight [2048] 6.3076 2.1956 0.0019 29.2500 0.0000 0.00 0.0229 nan N/A model.layers.20.post_attention_layernorm.weight [2048] 1.4658 0.2796 -0.4805 4.4375 0.0000 0.00 0.0122 nan N/A model.layers.21.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0388 -0.4688 0.4258 0.0000 -0.00 0.0064 1.6170 0 model.layers.21.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0320 -0.6484 0.8789 0.0000 -0.00 0.0027 1.3681 0 model.layers.21.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0360 -0.3535 0.3496 0.0000 -0.00 0.0037 1.5851 0 model.layers.21.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0327 -0.3242 0.3066 0.0000 -0.00 0.0021 1.4665 0 model.layers.21.self_attn.q_norm.weight [128] 1.4818 0.6233 -0.0381 2.7344 0.0000 0.00 0.0000 nan N/A model.layers.21.self_attn.k_norm.weight [128] 1.6387 1.1450 -0.0481 9.3750 0.0000 0.00 0.0078 nan N/A model.layers.21.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0355 -0.6133 0.6250 0.0000 -0.00 0.0011 1.5840 0 model.layers.21.mlp.up_proj.weight [6144, 2048] -0.0000 0.0387 -0.6992 0.8242 0.0000 -0.00 0.0007 1.7390 0 model.layers.21.mlp.down_proj.weight [2048, 6144] 0.0000 0.0379 -0.7812 0.8398 0.0000 -0.00 0.0006 2.9408 0 model.layers.21.input_layernorm.weight [2048] 7.8696 3.1144 0.0018 44.2500 0.0000 0.00 0.0229 nan N/A model.layers.21.post_attention_layernorm.weight [2048] 1.6247 0.2874 -0.0024 4.4375 0.0000 -0.00 0.0098 nan N/A model.layers.22.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0378 -0.4062 0.4102 0.0000 -0.00 0.0056 1.5827 0 model.layers.22.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0317 -0.3711 0.4043 0.0000 -0.00 0.0027 1.3675 0 model.layers.22.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0355 -0.3086 0.3047 0.0000 -0.00 0.0014 1.5926 0 model.layers.22.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0324 -0.3984 0.5234 0.0000 -0.00 0.0011 1.4517 0 model.layers.22.self_attn.q_norm.weight [128] 1.4677 0.6570 -0.0283 6.2812 0.0000 0.00 0.0078 nan N/A model.layers.22.self_attn.k_norm.weight [128] 1.5249 0.8786 -0.0417 5.0938 0.0000 -0.00 0.0234 nan N/A model.layers.22.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0361 -0.7266 0.7188 0.0000 -0.00 0.0008 1.6176 0 model.layers.22.mlp.up_proj.weight [6144, 2048] 0.0000 0.0391 -0.5977 0.9453 0.0000 -0.00 0.0005 1.7588 0 model.layers.22.mlp.down_proj.weight [2048, 6144] -0.0000 0.0386 -1.2031 1.1406 0.0000 -0.00 0.0005 2.9804 0 model.layers.22.input_layernorm.weight [2048] 9.1258 4.4278 0.8594 55.0000 0.0000 0.00 0.0259 nan N/A model.layers.22.post_attention_layernorm.weight [2048] 1.8063 0.2904 -0.0029 4.8438 0.0000 0.00 0.0103 nan N/A model.layers.23.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0353 -0.4531 0.4492 0.0000 -0.00 0.0042 1.5161 0 model.layers.23.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0313 -0.3457 0.3906 0.0000 -0.00 0.0022 1.3741 0 model.layers.23.self_attn.v_proj.weight [1024, 2048] 0.0001 0.0348 -0.3438 0.2988 0.0000 -0.00 0.0015 1.5687 0 model.layers.23.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0326 -0.2969 0.4004 0.0000 -0.00 0.0009 1.4652 0 model.layers.23.self_attn.q_norm.weight [128] 1.4275 0.5005 0.0052 2.5625 0.0000 0.00 0.0000 nan N/A model.layers.23.self_attn.k_norm.weight [128] 2.0273 1.2693 -0.0050 9.0000 0.0000 0.00 0.0078 nan N/A model.layers.23.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0366 -0.7305 0.8555 0.0000 -0.00 0.0006 1.6415 0 model.layers.23.mlp.up_proj.weight [6144, 2048] -0.0000 0.0392 -0.7852 0.6836 0.0000 -0.00 0.0004 1.7657 0 model.layers.23.mlp.down_proj.weight [2048, 6144] -0.0000 0.0390 -0.9844 1.1797 0.0000 -0.00 0.0005 3.0050 0 model.layers.23.input_layernorm.weight [2048] 10.4963 5.4583 0.0018 74.5000 0.0000 0.00 0.0220 nan N/A model.layers.23.post_attention_layernorm.weight [2048] 2.0123 0.3104 -0.0019 5.1875 0.0000 -0.00 0.0112 nan N/A model.layers.24.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0369 -0.3828 0.4375 0.0000 -0.00 0.0057 1.5583 0 model.layers.24.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0312 -0.3535 0.3047 0.0000 -0.00 0.0030 1.3521 0 model.layers.24.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0350 -0.2539 0.3438 0.0000 -0.00 0.0010 1.5803 0 model.layers.24.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0333 -0.3066 0.2812 0.0000 -0.00 0.0007 1.4859 0 model.layers.24.self_attn.q_norm.weight [128] 1.3816 0.5937 -0.0349 2.8125 0.0000 0.00 0.0000 nan N/A model.layers.24.self_attn.k_norm.weight [128] 2.0919 1.4716 -0.0908 8.0625 0.0000 -0.00 0.0156 nan N/A model.layers.24.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0362 -0.8164 0.8711 0.0000 -0.00 0.0005 1.6259 0 model.layers.24.mlp.up_proj.weight [6144, 2048] 0.0000 0.0394 -0.5664 0.5742 0.0000 -0.00 0.0004 1.7753 0 model.layers.24.mlp.down_proj.weight [2048, 6144] 0.0000 0.0391 -0.8789 1.0859 0.0000 -0.00 0.0005 3.0185 0 model.layers.24.input_layernorm.weight [2048] 13.5919 9.3049 -4.0938 95.5000 0.0000 0.00 0.0220 nan N/A model.layers.24.post_attention_layernorm.weight [2048] 2.0885 0.3191 -0.0020 5.4375 0.0000 0.00 0.0137 nan N/A model.layers.25.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0359 -0.4219 0.5078 0.0000 -0.00 0.0050 1.5183 0 model.layers.25.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0302 -0.3594 0.3223 0.0000 -0.00 0.0026 1.3119 0 model.layers.25.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0388 -0.3223 0.2988 0.0000 -0.00 0.0010 1.7480 0 model.layers.25.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0350 -0.2949 0.2988 0.0000 -0.00 0.0007 1.5653 0 model.layers.25.self_attn.q_norm.weight [128] 1.4685 0.6075 0.0018 3.1562 0.0000 -0.00 0.0000 nan N/A model.layers.25.self_attn.k_norm.weight [128] 1.8986 1.2900 -0.0150 5.7188 0.0000 0.00 0.0000 nan N/A model.layers.25.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0357 -0.7695 0.6953 0.0000 -0.00 0.0006 1.6021 0 model.layers.25.mlp.up_proj.weight [6144, 2048] 0.0000 0.0395 -0.7109 0.8867 0.0000 -0.00 0.0004 1.7789 0 model.layers.25.mlp.down_proj.weight [2048, 6144] -0.0000 0.0388 -0.8516 0.9336 0.0000 -0.00 0.0004 3.0089 0 model.layers.25.input_layernorm.weight [2048] 15.9484 13.3387 -4.4688 121.5000 0.0000 0.00 0.0317 nan N/A model.layers.25.post_attention_layernorm.weight [2048] 2.2062 0.4000 -0.0029 9.3125 0.0000 -0.00 0.0151 nan N/A model.layers.26.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0389 -0.3672 0.4121 0.0000 -0.00 0.0054 1.6131 0 model.layers.26.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0293 -0.2432 0.3145 0.0000 -0.00 0.0025 1.2736 0 model.layers.26.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0377 -0.3262 0.2432 0.0000 -0.00 0.0007 1.6897 0 model.layers.26.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0348 -0.6719 0.6094 0.0000 -0.00 0.0020 1.5532 0 model.layers.26.self_attn.q_norm.weight [128] 1.2684 0.7311 -0.0376 3.5000 0.0000 0.00 0.0078 nan N/A model.layers.26.self_attn.k_norm.weight [128] 1.5801 1.0784 -0.0698 4.6250 0.0000 0.00 0.0000 nan N/A model.layers.26.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0354 -0.5664 0.6797 0.0000 -0.00 0.0006 1.5891 0 model.layers.26.mlp.up_proj.weight [6144, 2048] 0.0000 0.0392 -0.5547 0.6094 0.0000 -0.00 0.0004 1.7659 0 model.layers.26.mlp.down_proj.weight [2048, 6144] -0.0000 0.0381 -1.2109 0.8555 0.0000 -0.00 0.0006 2.9697 0 model.layers.26.input_layernorm.weight [2048] 17.9357 16.5038 -5.9688 136.0000 0.0000 0.00 0.0488 nan N/A model.layers.26.post_attention_layernorm.weight [2048] 2.4546 0.7710 -0.0022 25.5000 0.0000 -0.01 0.0117 nan N/A model.layers.27.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0355 -0.4297 0.5508 0.0000 -0.00 0.0024 1.5582 0 model.layers.27.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0339 -1.6250 0.9023 0.0000 -0.00 0.0010 1.5101 0 model.layers.27.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0376 -0.2373 0.2852 0.0000 -0.00 0.0004 1.6919 0 model.layers.27.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0351 -0.5664 0.5273 0.0000 -0.00 0.0007 1.5805 0 model.layers.27.self_attn.q_norm.weight [128] 1.5298 0.6679 -0.0913 4.3125 0.0000 0.00 0.0234 nan N/A model.layers.27.self_attn.k_norm.weight [128] 1.7309 1.1485 -1.1172 10.1875 0.0000 0.00 0.0078 nan N/A model.layers.27.mlp.gate_proj.weight [6144, 2048] -0.0001 0.0374 -0.7109 0.6719 0.0000 -0.00 0.0008 1.6732 0 model.layers.27.mlp.up_proj.weight [6144, 2048] -0.0000 0.0387 -0.4609 1.0156 0.0000 -0.00 0.0006 1.7331 0 model.layers.27.mlp.down_proj.weight [2048, 6144] -0.0000 0.0354 -1.0234 1.0234 0.0000 -0.00 0.0009 2.7653 0 model.layers.27.input_layernorm.weight [2048] 17.2849 4.6932 1.5312 77.5000 0.0000 0.00 0.0215 nan N/A model.layers.27.post_attention_layernorm.weight [2048] 3.0747 3.4869 -0.0024 153.0000 0.0000 0.00 0.0020 nan N/A model.norm.weight [2048] 2.2139 0.5169 -0.0461 12.0625 0.0000 0.00 0.0142 nan N/A lm_head.weight [151936, 2048] -0.0001 0.0354 -0.3613 0.2988 0.0000 -0.00 0.0003 1.5789 0 [06] Layer-type aggregated statistics (distilled) Type Count Params AvgMean AvgStd AvgSparse AvgKurtD ---------------------------------------------------------------------------------- attn_k 28 58,720,256 -0.00000 0.03208 0.00003 -0.000 attn_knorm 28 3,584 2.01822 1.85875 0.00000 0.000 attn_o 28 117,440,512 -0.00000 0.03193 0.00003 -0.000 attn_q 28 117,440,512 -0.00000 0.03597 0.00003 -0.000 attn_qnorm 28 3,584 1.57232 0.78864 0.00000 0.000 attn_v 28 58,720,256 0.00001 0.03476 0.00003 -0.000 embedding 1 311,164,928 -0.00011 0.03539 0.00001 -0.001 final_norm 1 2,048 2.21386 0.51687 0.00000 0.000 layernorm 56 114,688 2.81740 1.41555 0.00000 0.002 lm_head 1 311,164,928 -0.00011 0.03539 0.00001 -0.001 mlp_down 28 352,321,536 0.00000 0.03432 0.00002 -0.000 mlp_gate 28 352,321,536 0.00002 0.03742 0.00002 -0.000 mlp_up 28 352,321,536 -0.00000 0.03512 0.00002 -0.000 [07] Per-transformer-block breakdown (distilled) Blk Sublayer Shape L2 AbsMn Std Sparse RowL2 ------------------------------------------------------------------------------------------------------------------- 0 input_layernorm [2048] 4.685 0.08947 0.05210 0.00000 nan 0 self_attn.q_proj [2048, 2048] 72.901 0.02744 0.03559 0.00003 1.57298 0 self_attn.k_proj [1024, 2048] 52.450 0.02796 0.03622 0.00003 1.59720 0 self_attn.v_proj [1024, 2048] 49.252 0.02678 0.03401 0.00002 1.53317 0 self_attn.o_proj [2048, 2048] 67.170 0.02499 0.03280 0.00003 1.46516 0 self_attn.q_norm [128] 20.866 1.64822 0.84563 0.00000 nan 0 self_attn.k_norm [128] 81.200 3.03745 6.52823 0.00000 nan 0 post_attention_layernorm [2048] 11.116 0.24129 0.14591 0.00000 nan 0 mlp.gate_proj [6144, 2048] 138.356 0.03043 0.03902 0.00002 1.74270 0 mlp.up_proj [6144, 2048] 112.786 0.02509 0.03180 0.00002 1.43743 0 mlp.down_proj [2048, 6144] 119.234 0.02647 0.03362 0.00002 2.62472 1 input_layernorm [2048] 4.442 0.09140 0.09403 0.00000 nan 1 self_attn.q_proj [2048, 2048] 71.518 0.02654 0.03492 0.00003 1.51872 1 self_attn.k_proj [1024, 2048] 49.452 0.02613 0.03415 0.00003 1.49522 1 self_attn.v_proj [1024, 2048] 50.624 0.02751 0.03496 0.00003 1.57671 1 self_attn.o_proj [2048, 2048] 67.294 0.02560 0.03286 0.00003 1.47633 1 self_attn.q_norm [128] 18.303 1.45519 0.79937 0.00000 nan 1 self_attn.k_norm [128] 40.221 2.25925 2.81085 0.00000 nan 1 post_attention_layernorm [2048] 24.810 0.52890 0.16376 0.00000 nan 1 mlp.gate_proj [6144, 2048] 160.751 0.03478 0.04534 0.00002 1.98843 1 mlp.up_proj [6144, 2048] 102.435 0.02192 0.02889 0.00004 1.25396 1 mlp.down_proj [2048, 6144] 104.364 0.02248 0.02943 0.00003 2.29928 2 input_layernorm [2048] 6.071 0.12772 0.04693 0.00000 nan 2 self_attn.q_proj [2048, 2048] 71.066 0.02627 0.03470 0.00003 1.50912 2 self_attn.k_proj [1024, 2048] 49.005 0.02568 0.03384 0.00004 1.47036 2 self_attn.v_proj [1024, 2048] 50.551 0.02718 0.03491 0.00003 1.55884 2 self_attn.o_proj [2048, 2048] 65.435 0.02473 0.03195 0.00003 1.43501 2 self_attn.q_norm [128] 19.083 1.49875 0.78076 0.00000 nan 2 self_attn.k_norm [128] 44.698 2.31981 3.21067 0.00000 nan 2 post_attention_layernorm [2048] 23.675 0.50428 0.14753 0.00000 nan 2 mlp.gate_proj [6144, 2048] 166.168 0.03672 0.04686 0.00002 2.08946 2 mlp.up_proj [6144, 2048] 105.972 0.02350 0.02988 0.00003 1.34295 2 mlp.down_proj [2048, 6144] 109.577 0.02421 0.03090 0.00003 2.41458 3 input_layernorm [2048] 15.178 0.32260 0.10331 0.00000 nan 3 self_attn.q_proj [2048, 2048] 72.017 0.02663 0.03516 0.00003 1.52801 3 self_attn.k_proj [1024, 2048] 50.006 0.02596 0.03453 0.00002 1.49155 3 self_attn.v_proj [1024, 2048] 53.659 0.02896 0.03705 0.00003 1.66516 3 self_attn.o_proj [2048, 2048] 69.315 0.02641 0.03384 0.00002 1.52010 3 self_attn.q_norm [128] 22.712 1.81032 0.87110 0.00000 nan 3 self_attn.k_norm [128] 29.281 1.76197 1.90323 0.00000 nan 3 post_attention_layernorm [2048] 26.474 0.57677 0.11189 0.00000 nan 3 mlp.gate_proj [6144, 2048] 164.227 0.03641 0.04631 0.00002 2.07124 3 mlp.up_proj [6144, 2048] 108.473 0.02410 0.03059 0.00003 1.37582 3 mlp.down_proj [2048, 6144] 109.132 0.02419 0.03077 0.00003 2.40517 4 input_layernorm [2048] 15.345 0.33078 0.08625 0.00000 nan 4 self_attn.q_proj [2048, 2048] 70.937 0.02631 0.03464 0.00003 1.50560 4 self_attn.k_proj [1024, 2048] 48.394 0.02525 0.03342 0.00003 1.44879 4 self_attn.v_proj [1024, 2048] 52.777 0.02844 0.03644 0.00002 1.63425 4 self_attn.o_proj [2048, 2048] 67.797 0.02570 0.03310 0.00003 1.48844 4 self_attn.q_norm [128] 13.751 1.06239 0.59266 0.00000 nan 4 self_attn.k_norm [128] 54.329 3.15128 3.63764 0.00000 nan 4 post_attention_layernorm [2048] 28.826 0.62692 0.12249 0.00000 nan 4 mlp.gate_proj [6144, 2048] 159.325 0.03529 0.04493 0.00002 2.00817 4 mlp.up_proj [6144, 2048] 109.481 0.02430 0.03087 0.00003 1.38822 4 mlp.down_proj [2048, 6144] 109.280 0.02420 0.03082 0.00003 2.40825 5 input_layernorm [2048] 18.209 0.39405 0.09262 0.00000 nan 5 self_attn.q_proj [2048, 2048] 73.221 0.02723 0.03575 0.00003 1.56245 5 self_attn.k_proj [1024, 2048] 49.084 0.02599 0.03389 0.00002 1.48577 5 self_attn.v_proj [1024, 2048] 51.053 0.02729 0.03525 0.00003 1.58715 5 self_attn.o_proj [2048, 2048] 68.198 0.02546 0.03330 0.00003 1.49868 5 self_attn.q_norm [128] 21.127 1.54580 1.05185 0.00000 nan 5 self_attn.k_norm [128] 49.039 2.45930 3.58324 0.00000 nan 5 post_attention_layernorm [2048] 24.066 0.52422 0.09310 0.00000 nan 5 mlp.gate_proj [6144, 2048] 141.749 0.03130 0.03997 0.00002 1.78453 5 mlp.up_proj [6144, 2048] 117.417 0.02614 0.03311 0.00002 1.49425 5 mlp.down_proj [2048, 6144] 116.503 0.02585 0.03285 0.00003 2.56775 6 input_layernorm [2048] 16.867 0.35778 0.11165 0.00000 nan 6 self_attn.q_proj [2048, 2048] 71.503 0.02570 0.03491 0.00003 1.47382 6 self_attn.k_proj [1024, 2048] 47.997 0.02440 0.03314 0.00004 1.39967 6 self_attn.v_proj [1024, 2048] 52.187 0.02799 0.03604 0.00002 1.61226 6 self_attn.o_proj [2048, 2048] 65.590 0.02470 0.03202 0.00003 1.43655 6 self_attn.q_norm [128] 22.756 1.71137 1.06104 0.00000 nan 6 self_attn.k_norm [128] 36.264 2.40319 2.12933 0.00000 nan 6 post_attention_layernorm [2048] 27.050 0.58757 0.12428 0.00000 nan 6 mlp.gate_proj [6144, 2048] 139.719 0.03072 0.03940 0.00002 1.75281 6 mlp.up_proj [6144, 2048] 117.968 0.02621 0.03326 0.00002 1.49864 6 mlp.down_proj [2048, 6144] 115.044 0.02540 0.03244 0.00003 2.53218 7 input_layernorm [2048] 26.151 0.55813 0.15338 0.00000 nan 7 self_attn.q_proj [2048, 2048] 69.230 0.02582 0.03380 0.00003 1.47648 7 self_attn.k_proj [1024, 2048] 46.204 0.02446 0.03190 0.00003 1.39686 7 self_attn.v_proj [1024, 2048] 50.629 0.02729 0.03496 0.00002 1.57454 7 self_attn.o_proj [2048, 2048] 65.907 0.02504 0.03218 0.00003 1.44267 7 self_attn.q_norm [128] 20.607 1.56852 0.92958 0.00000 nan 7 self_attn.k_norm [128] 31.397 1.88898 2.04107 0.00000 nan 7 post_attention_layernorm [2048] 30.772 0.66610 0.14451 0.00000 nan 7 mlp.gate_proj [6144, 2048] 144.020 0.03162 0.04061 0.00002 1.79936 7 mlp.up_proj [6144, 2048] 116.378 0.02579 0.03282 0.00003 1.47432 7 mlp.down_proj [2048, 6144] 113.238 0.02495 0.03193 0.00003 2.48453 8 input_layernorm [2048] 28.138 0.60021 0.16233 0.00000 nan 8 self_attn.q_proj [2048, 2048] 70.971 0.02579 0.03465 0.00003 1.48264 8 self_attn.k_proj [1024, 2048] 47.226 0.02438 0.03261 0.00003 1.40493 8 self_attn.v_proj [1024, 2048] 51.273 0.02780 0.03540 0.00002 1.59720 8 self_attn.o_proj [2048, 2048] 64.765 0.02465 0.03162 0.00003 1.41357 8 self_attn.q_norm [128] 17.455 1.44596 0.54216 0.00000 nan 8 self_attn.k_norm [128] 21.129 1.64117 0.89530 0.00000 nan 8 post_attention_layernorm [2048] 30.563 0.66207 0.13507 0.00000 nan 8 mlp.gate_proj [6144, 2048] 137.524 0.03013 0.03878 0.00002 1.71706 8 mlp.up_proj [6144, 2048] 117.760 0.02611 0.03321 0.00002 1.49330 8 mlp.down_proj [2048, 6144] 116.430 0.02562 0.03283 0.00002 2.55323 9 input_layernorm [2048] 37.133 0.79334 0.20959 0.00000 nan 9 self_attn.q_proj [2048, 2048] 71.985 0.02616 0.03515 0.00003 1.50425 9 self_attn.k_proj [1024, 2048] 46.960 0.02450 0.03243 0.00003 1.41366 9 self_attn.v_proj [1024, 2048] 50.199 0.02671 0.03466 0.00003 1.55031 9 self_attn.o_proj [2048, 2048] 66.940 0.02478 0.03268 0.00003 1.45834 9 self_attn.q_norm [128] 23.023 1.65887 1.18381 0.00000 nan 9 self_attn.k_norm [128] 30.462 1.98125 1.83048 0.00000 nan 9 post_attention_layernorm [2048] 31.048 0.67109 0.14262 0.00000 nan 9 mlp.gate_proj [6144, 2048] 136.793 0.02997 0.03858 0.00002 1.70927 9 mlp.up_proj [6144, 2048] 122.606 0.02717 0.03457 0.00002 1.55372 9 mlp.down_proj [2048, 6144] 117.158 0.02569 0.03304 0.00003 2.56601 10 input_layernorm [2048] 36.724 0.77820 0.23073 0.00000 nan 10 self_attn.q_proj [2048, 2048] 70.798 0.02641 0.03457 0.00002 1.51064 10 self_attn.k_proj [1024, 2048] 48.128 0.02556 0.03323 0.00002 1.45925 10 self_attn.v_proj [1024, 2048] 49.962 0.02700 0.03450 0.00003 1.55550 10 self_attn.o_proj [2048, 2048] 65.289 0.02457 0.03188 0.00003 1.42228 10 self_attn.q_norm [128] 25.073 1.77536 1.33162 0.00000 nan 10 self_attn.k_norm [128] 31.226 1.89182 2.01759 0.00000 nan 10 post_attention_layernorm [2048] 31.534 0.68355 0.13537 0.00000 nan 10 mlp.gate_proj [6144, 2048] 135.160 0.02929 0.03812 0.00002 1.67895 10 mlp.up_proj [6144, 2048] 123.157 0.02714 0.03473 0.00002 1.55874 10 mlp.down_proj [2048, 6144] 116.162 0.02527 0.03276 0.00003 2.54469 11 input_layernorm [2048] 49.718 1.01369 0.42364 0.00000 nan 11 self_attn.q_proj [2048, 2048] 75.117 0.02664 0.03668 0.00002 1.53962 11 self_attn.k_proj [1024, 2048] 48.444 0.02482 0.03345 0.00003 1.43803 11 self_attn.v_proj [1024, 2048] 47.471 0.02513 0.03278 0.00003 1.46825 11 self_attn.o_proj [2048, 2048] 61.987 0.02241 0.03027 0.00003 1.34302 11 self_attn.q_norm [128] 21.207 1.77125 0.61618 0.00000 nan 11 self_attn.k_norm [128] 25.933 2.05436 1.02094 0.00000 nan 11 post_attention_layernorm [2048] 35.154 0.75061 0.20229 0.00000 nan 11 mlp.gate_proj [6144, 2048] 128.540 0.02813 0.03624 0.00002 1.60858 11 mlp.up_proj [6144, 2048] 125.940 0.02782 0.03551 0.00003 1.59340 11 mlp.down_proj [2048, 6144] 120.142 0.02632 0.03388 0.00002 2.63503 12 input_layernorm [2048] 59.373 1.20554 0.51949 0.00000 nan 12 self_attn.q_proj [2048, 2048] 71.912 0.02639 0.03511 0.00003 1.51193 12 self_attn.k_proj [1024, 2048] 47.209 0.02461 0.03260 0.00004 1.41397 12 self_attn.v_proj [1024, 2048] 48.870 0.02639 0.03374 0.00003 1.52150 12 self_attn.o_proj [2048, 2048] 62.022 0.02345 0.03028 0.00003 1.34798 12 self_attn.q_norm [128] 22.008 1.79909 0.74309 0.00000 nan 12 self_attn.k_norm [128] 28.686 1.77965 1.81323 0.00000 nan 12 post_attention_layernorm [2048] 35.525 0.75779 0.20713 0.00000 nan 12 mlp.gate_proj [6144, 2048] 125.090 0.02734 0.03527 0.00002 1.56271 12 mlp.up_proj [6144, 2048] 125.435 0.02766 0.03537 0.00002 1.58552 12 mlp.down_proj [2048, 6144] 120.075 0.02627 0.03386 0.00003 2.63047 13 input_layernorm [2048] 65.825 1.32442 0.60343 0.00000 nan 13 self_attn.q_proj [2048, 2048] 72.686 0.02604 0.03549 0.00002 1.49013 13 self_attn.k_proj [1024, 2048] 44.996 0.02354 0.03107 0.00003 1.35037 13 self_attn.v_proj [1024, 2048] 48.618 0.02629 0.03357 0.00003 1.51328 13 self_attn.o_proj [2048, 2048] 60.200 0.02273 0.02939 0.00003 1.30548 13 self_attn.q_norm [128] 23.733 1.85438 0.99210 0.00000 nan 13 self_attn.k_norm [128] 22.425 1.77937 0.87723 0.00000 nan 13 post_attention_layernorm [2048] 37.208 0.79018 0.22723 0.00000 nan 13 mlp.gate_proj [6144, 2048] 120.812 0.02631 0.03407 0.00003 1.50769 13 mlp.up_proj [6144, 2048] 122.147 0.02692 0.03444 0.00003 1.54372 13 mlp.down_proj [2048, 6144] 117.098 0.02556 0.03302 0.00003 2.56153 14 input_layernorm [2048] 84.611 1.69142 0.79895 0.00000 nan 14 self_attn.q_proj [2048, 2048] 70.351 0.02527 0.03435 0.00003 1.45125 14 self_attn.k_proj [1024, 2048] 44.671 0.02308 0.03085 0.00003 1.32900 14 self_attn.v_proj [1024, 2048] 47.417 0.02557 0.03274 0.00003 1.47567 14 self_attn.o_proj [2048, 2048] 60.245 0.02268 0.02941 0.00003 1.30631 14 self_attn.q_norm [128] 23.459 1.79195 1.04803 0.00000 nan 14 self_attn.k_norm [128] 24.794 1.83374 1.20587 0.00000 nan 14 post_attention_layernorm [2048] 38.752 0.82513 0.22901 0.00000 nan 14 mlp.gate_proj [6144, 2048] 115.589 0.02511 0.03260 0.00003 1.44356 14 mlp.up_proj [6144, 2048] 118.312 0.02602 0.03336 0.00002 1.49604 14 mlp.down_proj [2048, 6144] 116.246 0.02533 0.03278 0.00003 2.54382 15 input_layernorm [2048] 116.758 2.38523 0.98750 0.00000 nan 15 self_attn.q_proj [2048, 2048] 72.109 0.02581 0.03521 0.00003 1.48097 15 self_attn.k_proj [1024, 2048] 42.654 0.02237 0.02945 0.00003 1.28119 15 self_attn.v_proj [1024, 2048] 47.263 0.02518 0.03264 0.00003 1.46780 15 self_attn.o_proj [2048, 2048] 61.616 0.02314 0.03008 0.00003 1.33738 15 self_attn.q_norm [128] 18.262 1.49814 0.60459 0.00000 nan 15 self_attn.k_norm [128] 38.772 2.29244 2.55740 0.00000 nan 15 post_attention_layernorm [2048] 38.488 0.82803 0.19409 0.00000 nan 15 mlp.gate_proj [6144, 2048] 116.356 0.02523 0.03281 0.00003 1.45222 15 mlp.up_proj [6144, 2048] 120.975 0.02656 0.03411 0.00003 1.52807 15 mlp.down_proj [2048, 6144] 116.761 0.02546 0.03293 0.00003 2.55310 16 input_layernorm [2048] 111.385 2.29854 0.88037 0.00000 nan 16 self_attn.q_proj [2048, 2048] 77.028 0.02664 0.03761 0.00003 1.53832 16 self_attn.k_proj [1024, 2048] 44.274 0.02238 0.03057 0.00003 1.29034 16 self_attn.v_proj [1024, 2048] 47.469 0.02541 0.03278 0.00002 1.47323 16 self_attn.o_proj [2048, 2048] 61.723 0.02308 0.03014 0.00003 1.33927 16 self_attn.q_norm [128] 19.869 1.62765 0.66726 0.00000 nan 16 self_attn.k_norm [128] 26.386 2.03656 1.14519 0.00000 nan 16 post_attention_layernorm [2048] 38.689 0.84295 0.14264 0.00000 nan 16 mlp.gate_proj [6144, 2048] 115.659 0.02505 0.03262 0.00003 1.45039 16 mlp.up_proj [6144, 2048] 124.277 0.02710 0.03504 0.00002 1.57209 16 mlp.down_proj [2048, 6144] 117.831 0.02567 0.03323 0.00003 2.59632 17 input_layernorm [2048] 195.528 4.07512 1.44214 0.00000 nan 17 self_attn.q_proj [2048, 2048] 73.600 0.02634 0.03594 0.00002 1.51986 17 self_attn.k_proj [1024, 2048] 44.185 0.02334 0.03051 0.00002 1.33630 17 self_attn.v_proj [1024, 2048] 49.308 0.02630 0.03405 0.00004 1.52620 17 self_attn.o_proj [2048, 2048] 62.506 0.02339 0.03052 0.00003 1.36478 17 self_attn.q_norm [128] 23.826 1.75517 1.16920 0.00000 nan 17 self_attn.k_norm [128] 28.626 1.99286 1.56686 0.00000 nan 17 post_attention_layernorm [2048] 47.981 1.03749 0.21853 0.00000 nan 17 mlp.gate_proj [6144, 2048] 115.404 0.02511 0.03254 0.00003 1.44933 17 mlp.up_proj [6144, 2048] 129.050 0.02835 0.03639 0.00002 1.63090 17 mlp.down_proj [2048, 6144] 121.534 0.02669 0.03427 0.00003 2.66801 18 input_layernorm [2048] 201.693 4.19416 1.51388 0.00000 nan 18 self_attn.q_proj [2048, 2048] 75.780 0.02675 0.03700 0.00003 1.53918 18 self_attn.k_proj [1024, 2048] 43.076 0.02232 0.02974 0.00003 1.28183 18 self_attn.v_proj [1024, 2048] 46.493 0.02501 0.03210 0.00003 1.45073 18 self_attn.o_proj [2048, 2048] 58.817 0.02227 0.02872 0.00003 1.28603 18 self_attn.q_norm [128] 18.318 1.53600 0.51702 0.00000 nan 18 self_attn.k_norm [128] 23.621 1.86847 0.94057 0.00000 nan 18 post_attention_layernorm [2048] 54.117 1.17464 0.22414 0.00000 nan 18 mlp.gate_proj [6144, 2048] 116.670 0.02534 0.03290 0.00003 1.46360 18 mlp.up_proj [6144, 2048] 129.370 0.02840 0.03648 0.00003 1.63489 18 mlp.down_proj [2048, 6144] 124.809 0.02738 0.03519 0.00002 2.73824 19 input_layernorm [2048] 265.464 5.42234 2.23838 0.00000 nan 19 self_attn.q_proj [2048, 2048] 78.144 0.02687 0.03816 0.00003 1.54781 19 self_attn.k_proj [1024, 2048] 43.280 0.02255 0.02989 0.00003 1.29700 19 self_attn.v_proj [1024, 2048] 46.370 0.02445 0.03202 0.00002 1.43815 19 self_attn.o_proj [2048, 2048] 61.106 0.02311 0.02984 0.00003 1.33827 19 self_attn.q_norm [128] 19.842 1.61686 0.68621 0.00000 nan 19 self_attn.k_norm [128] 24.114 1.87106 1.02789 0.00000 nan 19 post_attention_layernorm [2048] 59.540 1.29598 0.22682 0.00000 nan 19 mlp.gate_proj [6144, 2048] 118.280 0.02563 0.03335 0.00003 1.48151 19 mlp.up_proj [6144, 2048] 131.654 0.02883 0.03712 0.00002 1.66379 19 mlp.down_proj [2048, 6144] 128.726 0.02814 0.03630 0.00002 2.82859 20 input_layernorm [2048] 302.241 6.30764 2.19555 0.00000 nan 20 self_attn.q_proj [2048, 2048] 79.257 0.02737 0.03870 0.00003 1.59106 20 self_attn.k_proj [1024, 2048] 45.357 0.02364 0.03132 0.00003 1.36726 20 self_attn.v_proj [1024, 2048] 48.235 0.02536 0.03331 0.00002 1.49516 20 self_attn.o_proj [2048, 2048] 63.751 0.02390 0.03113 0.00002 1.39336 20 self_attn.q_norm [128] 19.990 1.63712 0.66759 0.00000 nan 20 self_attn.k_norm [128] 23.313 1.79316 1.02053 0.00000 nan 20 post_attention_layernorm [2048] 67.529 1.46625 0.27958 0.00000 nan 20 mlp.gate_proj [6144, 2048] 122.886 0.02678 0.03465 0.00003 1.54046 20 mlp.up_proj [6144, 2048] 134.977 0.02972 0.03806 0.00002 1.70752 20 mlp.down_proj [2048, 6144] 132.033 0.02897 0.03723 0.00002 2.89669 21 input_layernorm [2048] 382.999 7.86958 3.11439 0.00000 nan 21 self_attn.q_proj [2048, 2048] 79.468 0.02798 0.03880 0.00003 1.61697 21 self_attn.k_proj [1024, 2048] 46.374 0.02374 0.03202 0.00003 1.36808 21 self_attn.v_proj [1024, 2048] 52.175 0.02694 0.03603 0.00003 1.58513 21 self_attn.o_proj [2048, 2048] 67.043 0.02514 0.03273 0.00003 1.46654 21 self_attn.q_norm [128] 18.176 1.48291 0.62333 0.00000 nan 21 self_attn.k_norm [128] 22.588 1.64089 1.14500 0.00000 nan 21 post_attention_layernorm [2048] 74.667 1.62471 0.28739 0.00000 nan 21 mlp.gate_proj [6144, 2048] 125.949 0.02768 0.03551 0.00002 1.58401 21 mlp.up_proj [6144, 2048] 137.228 0.03036 0.03870 0.00002 1.73898 21 mlp.down_proj [2048, 6144] 134.445 0.02958 0.03791 0.00002 2.94081 22 input_layernorm [2048] 459.010 9.12582 4.42780 0.00000 nan 22 self_attn.q_proj [2048, 2048] 77.429 0.02735 0.03781 0.00003 1.58268 22 self_attn.k_proj [1024, 2048] 45.910 0.02373 0.03170 0.00004 1.36746 22 self_attn.v_proj [1024, 2048] 51.373 0.02718 0.03547 0.00002 1.59256 22 self_attn.o_proj [2048, 2048] 66.308 0.02509 0.03238 0.00002 1.45173 22 self_attn.q_norm [128] 18.181 1.46818 0.65705 0.00000 nan 22 self_attn.k_norm [128] 19.892 1.52586 0.87863 0.00000 nan 22 post_attention_layernorm [2048] 82.794 1.80634 0.29039 0.00000 nan 22 mlp.gate_proj [6144, 2048] 128.172 0.02834 0.03614 0.00002 1.61757 22 mlp.up_proj [6144, 2048] 138.566 0.03079 0.03907 0.00002 1.75877 22 mlp.down_proj [2048, 6144] 136.816 0.03010 0.03858 0.00002 2.98038 23 input_layernorm [2048] 535.369 10.49632 5.45829 0.00000 nan 23 self_attn.q_proj [2048, 2048] 72.309 0.02620 0.03531 0.00003 1.51609 23 self_attn.k_proj [1024, 2048] 45.286 0.02374 0.03127 0.00004 1.37414 23 self_attn.v_proj [1024, 2048] 50.443 0.02670 0.03483 0.00003 1.56868 23 self_attn.o_proj [2048, 2048] 66.829 0.02537 0.03263 0.00002 1.46522 23 self_attn.q_norm [128] 17.107 1.42749 0.50050 0.00000 nan 23 self_attn.k_norm [128] 27.031 2.02744 1.26929 0.00000 nan 23 post_attention_layernorm [2048] 92.141 2.01226 0.31043 0.00000 nan 23 mlp.gate_proj [6144, 2048] 129.759 0.02883 0.03659 0.00002 1.64149 23 mlp.up_proj [6144, 2048] 139.033 0.03096 0.03921 0.00002 1.76572 23 mlp.down_proj [2048, 6144] 138.182 0.03038 0.03897 0.00002 3.00505 24 input_layernorm [2048] 745.373 13.59591 9.30494 0.00000 nan 24 self_attn.q_proj [2048, 2048] 75.594 0.02681 0.03691 0.00003 1.55826 24 self_attn.k_proj [1024, 2048] 45.236 0.02332 0.03124 0.00003 1.35214 24 self_attn.v_proj [1024, 2048] 50.747 0.02695 0.03504 0.00003 1.58034 24 self_attn.o_proj [2048, 2048] 68.236 0.02596 0.03332 0.00002 1.48595 24 self_attn.q_norm [128] 17.003 1.38267 0.59374 0.00000 nan 24 self_attn.k_norm [128] 28.900 2.09434 1.47164 0.00000 nan 24 post_attention_layernorm [2048] 95.613 2.08855 0.31907 0.00000 nan 24 mlp.gate_proj [6144, 2048] 128.384 0.02856 0.03620 0.00002 1.62588 24 mlp.up_proj [6144, 2048] 139.729 0.03114 0.03940 0.00002 1.77526 24 mlp.down_proj [2048, 6144] 138.658 0.03053 0.03910 0.00002 3.01847 25 input_layernorm [2048] 940.804 15.95273 13.33869 0.00000 nan 25 self_attn.q_proj [2048, 2048] 73.422 0.02611 0.03585 0.00003 1.51835 25 self_attn.k_proj [1024, 2048] 43.686 0.02266 0.03017 0.00003 1.31192 25 self_attn.v_proj [1024, 2048] 56.192 0.02986 0.03880 0.00002 1.74803 25 self_attn.o_proj [2048, 2048] 71.640 0.02723 0.03498 0.00002 1.56534 25 self_attn.q_norm [128] 17.970 1.46855 0.60751 0.00000 nan 25 self_attn.k_norm [128] 25.937 1.89888 1.29004 0.00000 nan 25 post_attention_layernorm [2048] 101.466 2.20616 0.40003 0.00000 nan 25 mlp.gate_proj [6144, 2048] 126.516 0.02814 0.03567 0.00002 1.60211 25 mlp.up_proj [6144, 2048] 140.019 0.03120 0.03948 0.00002 1.77892 25 mlp.down_proj [2048, 6144] 137.531 0.03044 0.03878 0.00002 3.00895 26 input_layernorm [2048] 1102.892 17.94151 16.50383 0.00000 nan 26 self_attn.q_proj [2048, 2048] 79.566 0.02773 0.03885 0.00003 1.61306 26 self_attn.k_proj [1024, 2048] 42.406 0.02208 0.02928 0.00004 1.27361 26 self_attn.v_proj [1024, 2048] 54.631 0.02919 0.03772 0.00002 1.68969 26 self_attn.o_proj [2048, 2048] 71.338 0.02655 0.03483 0.00002 1.55324 26 self_attn.q_norm [128] 16.547 1.26933 0.73112 0.00000 nan 26 self_attn.k_norm [128] 21.617 1.58323 1.07843 0.00000 nan 26 post_attention_layernorm [2048] 116.430 2.45460 0.77095 0.00000 nan 26 mlp.gate_proj [6144, 2048] 125.452 0.02789 0.03537 0.00002 1.58906 26 mlp.up_proj [6144, 2048] 139.144 0.03096 0.03924 0.00002 1.76593 26 mlp.down_proj [2048, 6144] 135.120 0.02999 0.03810 0.00002 2.96974 27 input_layernorm [2048] 810.531 17.28486 4.69322 0.00000 nan 27 self_attn.q_proj [2048, 2048] 72.699 0.02733 0.03550 0.00002 1.55820 27 self_attn.k_proj [1024, 2048] 49.078 0.02647 0.03389 0.00002 1.51007 27 self_attn.v_proj [1024, 2048] 54.406 0.02942 0.03757 0.00003 1.69188 27 self_attn.o_proj [2048, 2048] 71.960 0.02762 0.03514 0.00002 1.58045 27 self_attn.q_norm [128] 18.873 1.53224 0.66789 0.00000 nan 27 self_attn.k_norm [128] 23.474 1.75737 1.14853 0.00000 nan 27 post_attention_layernorm [2048] 210.356 3.07466 3.48695 0.00000 nan 27 mlp.gate_proj [6144, 2048] 132.785 0.02938 0.03744 0.00002 1.67322 27 mlp.up_proj [6144, 2048] 137.233 0.03040 0.03870 0.00002 1.73311 27 mlp.down_proj [2048, 6144] 125.503 0.02759 0.03539 0.00002 2.76534 [08] Isotropy analysis (distilled, 2D tensors only) Sampling up to 2048 rows per layer. Score near 0 = isotropic (healthy). Score near 1 = representation collapse. Layer Shape Score ------------------------------------------------------------------------------------------------------ model.embed_tokens.weight [151936, 2048] 0.053880 model.layers.0.self_attn.q_proj.weight [2048, 2048] -0.000024 model.layers.0.self_attn.k_proj.weight [1024, 2048] -0.000105 model.layers.0.self_attn.v_proj.weight [1024, 2048] 0.000003 model.layers.0.self_attn.o_proj.weight [2048, 2048] 0.000003 model.layers.0.mlp.gate_proj.weight [6144, 2048] 0.004988 model.layers.0.mlp.up_proj.weight [6144, 2048] -0.000027 model.layers.0.mlp.down_proj.weight [2048, 6144] -0.000010 model.layers.1.self_attn.q_proj.weight [2048, 2048] 0.000003 model.layers.1.self_attn.k_proj.weight [1024, 2048] 0.000002 model.layers.1.self_attn.v_proj.weight [1024, 2048] -0.000032 model.layers.1.self_attn.o_proj.weight [2048, 2048] -0.000009 model.layers.1.mlp.gate_proj.weight [6144, 2048] 0.022451 model.layers.1.mlp.up_proj.weight [6144, 2048] 0.000007 model.layers.1.mlp.down_proj.weight [2048, 6144] -0.000011 model.layers.2.self_attn.q_proj.weight [2048, 2048] 0.000066 model.layers.2.self_attn.k_proj.weight [1024, 2048] -0.000001 model.layers.2.self_attn.v_proj.weight [1024, 2048] 0.000026 model.layers.2.self_attn.o_proj.weight [2048, 2048] -0.000055 model.layers.2.mlp.gate_proj.weight [6144, 2048] 0.006968 model.layers.2.mlp.up_proj.weight [6144, 2048] 0.000021 model.layers.2.mlp.down_proj.weight [2048, 6144] 0.000004 model.layers.3.self_attn.q_proj.weight [2048, 2048] 0.000014 model.layers.3.self_attn.k_proj.weight [1024, 2048] -0.000067 model.layers.3.self_attn.v_proj.weight [1024, 2048] -0.000037 model.layers.3.self_attn.o_proj.weight [2048, 2048] 0.000007 model.layers.3.mlp.gate_proj.weight [6144, 2048] 0.007741 model.layers.3.mlp.up_proj.weight [6144, 2048] 0.000003 model.layers.3.mlp.down_proj.weight [2048, 6144] 0.000014 model.layers.4.self_attn.q_proj.weight [2048, 2048] 0.000045 model.layers.4.self_attn.k_proj.weight [1024, 2048] -0.000034 model.layers.4.self_attn.v_proj.weight [1024, 2048] 0.000003 model.layers.4.self_attn.o_proj.weight [2048, 2048] -0.000006 model.layers.4.mlp.gate_proj.weight [6144, 2048] 0.011344 model.layers.4.mlp.up_proj.weight [6144, 2048] 0.000008 model.layers.4.mlp.down_proj.weight [2048, 6144] 0.000012 model.layers.5.self_attn.q_proj.weight [2048, 2048] 0.000046 model.layers.5.self_attn.k_proj.weight [1024, 2048] -0.000004 model.layers.5.self_attn.v_proj.weight [1024, 2048] 0.000016 model.layers.5.self_attn.o_proj.weight [2048, 2048] 0.000011 model.layers.5.mlp.gate_proj.weight [6144, 2048] 0.006708 model.layers.5.mlp.up_proj.weight [6144, 2048] -0.000022 model.layers.5.mlp.down_proj.weight [2048, 6144] 0.000015 model.layers.6.self_attn.q_proj.weight [2048, 2048] 0.000093 model.layers.6.self_attn.k_proj.weight [1024, 2048] -0.000075 model.layers.6.self_attn.v_proj.weight [1024, 2048] -0.000043 model.layers.6.self_attn.o_proj.weight [2048, 2048] 0.000007 model.layers.6.mlp.gate_proj.weight [6144, 2048] 0.007193 model.layers.6.mlp.up_proj.weight [6144, 2048] -0.000010 model.layers.6.mlp.down_proj.weight [2048, 6144] 0.000017 model.layers.7.self_attn.q_proj.weight [2048, 2048] 0.000059 model.layers.7.self_attn.k_proj.weight [1024, 2048] -0.000034 model.layers.7.self_attn.v_proj.weight [1024, 2048] -0.000037 model.layers.7.self_attn.o_proj.weight [2048, 2048] 0.000010 model.layers.7.mlp.gate_proj.weight [6144, 2048] 0.009189 model.layers.7.mlp.up_proj.weight [6144, 2048] 0.000011 model.layers.7.mlp.down_proj.weight [2048, 6144] -0.000001 model.layers.8.self_attn.q_proj.weight [2048, 2048] 0.000037 model.layers.8.self_attn.k_proj.weight [1024, 2048] -0.000017 model.layers.8.self_attn.v_proj.weight [1024, 2048] -0.000040 model.layers.8.self_attn.o_proj.weight [2048, 2048] 0.000012 model.layers.8.mlp.gate_proj.weight [6144, 2048] 0.006754 model.layers.8.mlp.up_proj.weight [6144, 2048] 0.000022 model.layers.8.mlp.down_proj.weight [2048, 6144] 0.000009 model.layers.9.self_attn.q_proj.weight [2048, 2048] 0.000025 model.layers.9.self_attn.k_proj.weight [1024, 2048] 0.000040 model.layers.9.self_attn.v_proj.weight [1024, 2048] 0.000018 model.layers.9.self_attn.o_proj.weight [2048, 2048] 0.000007 model.layers.9.mlp.gate_proj.weight [6144, 2048] 0.005668 model.layers.9.mlp.up_proj.weight [6144, 2048] 0.000008 model.layers.9.mlp.down_proj.weight [2048, 6144] 0.000004 model.layers.10.self_attn.q_proj.weight [2048, 2048] 0.000032 model.layers.10.self_attn.k_proj.weight [1024, 2048] -0.000022 model.layers.10.self_attn.v_proj.weight [1024, 2048] 0.000004 model.layers.10.self_attn.o_proj.weight [2048, 2048] 0.000008 model.layers.10.mlp.gate_proj.weight [6144, 2048] 0.004282 model.layers.10.mlp.up_proj.weight [6144, 2048] -0.000008 model.layers.10.mlp.down_proj.weight [2048, 6144] 0.000002 model.layers.11.self_attn.q_proj.weight [2048, 2048] 0.000071 model.layers.11.self_attn.k_proj.weight [1024, 2048] -0.000073 model.layers.11.self_attn.v_proj.weight [1024, 2048] 0.000048 model.layers.11.self_attn.o_proj.weight [2048, 2048] 0.000011 model.layers.11.mlp.gate_proj.weight [6144, 2048] 0.003255 model.layers.11.mlp.up_proj.weight [6144, 2048] -0.000026 model.layers.11.mlp.down_proj.weight [2048, 6144] -0.000023 model.layers.12.self_attn.q_proj.weight [2048, 2048] 0.000031 model.layers.12.self_attn.k_proj.weight [1024, 2048] -0.000078 model.layers.12.self_attn.v_proj.weight [1024, 2048] 0.000001 model.layers.12.self_attn.o_proj.weight [2048, 2048] -0.000010 model.layers.12.mlp.gate_proj.weight [6144, 2048] 0.002464 model.layers.12.mlp.up_proj.weight [6144, 2048] 0.000018 model.layers.12.mlp.down_proj.weight [2048, 6144] -0.000003 model.layers.13.self_attn.q_proj.weight [2048, 2048] 0.000075 model.layers.13.self_attn.k_proj.weight [1024, 2048] -0.000093 model.layers.13.self_attn.v_proj.weight [1024, 2048] 0.000004 model.layers.13.self_attn.o_proj.weight [2048, 2048] 0.000006 model.layers.13.mlp.gate_proj.weight [6144, 2048] 0.002376 model.layers.13.mlp.up_proj.weight [6144, 2048] 0.000014 model.layers.13.mlp.down_proj.weight [2048, 6144] 0.000006 model.layers.14.self_attn.q_proj.weight [2048, 2048] 0.000035 model.layers.14.self_attn.k_proj.weight [1024, 2048] -0.000052 model.layers.14.self_attn.v_proj.weight [1024, 2048] 0.000018 model.layers.14.self_attn.o_proj.weight [2048, 2048] 0.000013 model.layers.14.mlp.gate_proj.weight [6144, 2048] 0.002415 model.layers.14.mlp.up_proj.weight [6144, 2048] 0.000037 model.layers.14.mlp.down_proj.weight [2048, 6144] 0.000017 model.layers.15.self_attn.q_proj.weight [2048, 2048] 0.000125 model.layers.15.self_attn.k_proj.weight [1024, 2048] 0.000015 model.layers.15.self_attn.v_proj.weight [1024, 2048] -0.000009 model.layers.15.self_attn.o_proj.weight [2048, 2048] 0.000011 model.layers.15.mlp.gate_proj.weight [6144, 2048] 0.002226 model.layers.15.mlp.up_proj.weight [6144, 2048] -0.000013 model.layers.15.mlp.down_proj.weight [2048, 6144] -0.000011 model.layers.16.self_attn.q_proj.weight [2048, 2048] 0.000119 model.layers.16.self_attn.k_proj.weight [1024, 2048] 0.000000 model.layers.16.self_attn.v_proj.weight [1024, 2048] -0.000033 model.layers.16.self_attn.o_proj.weight [2048, 2048] 0.000011 model.layers.16.mlp.gate_proj.weight [6144, 2048] 0.002481 model.layers.16.mlp.up_proj.weight [6144, 2048] -0.000019 model.layers.16.mlp.down_proj.weight [2048, 6144] -0.000020 model.layers.17.self_attn.q_proj.weight [2048, 2048] 0.000154 model.layers.17.self_attn.k_proj.weight [1024, 2048] 0.000096 model.layers.17.self_attn.v_proj.weight [1024, 2048] -0.000025 model.layers.17.self_attn.o_proj.weight [2048, 2048] 0.000005 model.layers.17.mlp.gate_proj.weight [6144, 2048] 0.003676 model.layers.17.mlp.up_proj.weight [6144, 2048] 0.000009 model.layers.17.mlp.down_proj.weight [2048, 6144] 0.000008 model.layers.18.self_attn.q_proj.weight [2048, 2048] 0.000152 model.layers.18.self_attn.k_proj.weight [1024, 2048] -0.000039 model.layers.18.self_attn.v_proj.weight [1024, 2048] -0.000016 model.layers.18.self_attn.o_proj.weight [2048, 2048] -0.000004 model.layers.18.mlp.gate_proj.weight [6144, 2048] 0.004275 model.layers.18.mlp.up_proj.weight [6144, 2048] -0.000034 model.layers.18.mlp.down_proj.weight [2048, 6144] 0.000024 model.layers.19.self_attn.q_proj.weight [2048, 2048] 0.000093 model.layers.19.self_attn.k_proj.weight [1024, 2048] -0.000042 model.layers.19.self_attn.v_proj.weight [1024, 2048] -0.000006 model.layers.19.self_attn.o_proj.weight [2048, 2048] -0.000030 model.layers.19.mlp.gate_proj.weight [6144, 2048] 0.005028 model.layers.19.mlp.up_proj.weight [6144, 2048] 0.000002 model.layers.19.mlp.down_proj.weight [2048, 6144] -0.000000 model.layers.20.self_attn.q_proj.weight [2048, 2048] 0.000107 model.layers.20.self_attn.k_proj.weight [1024, 2048] 0.000033 model.layers.20.self_attn.v_proj.weight [1024, 2048] 0.000032 model.layers.20.self_attn.o_proj.weight [2048, 2048] 0.000024 model.layers.20.mlp.gate_proj.weight [6144, 2048] 0.006397 model.layers.20.mlp.up_proj.weight [6144, 2048] 0.000010 model.layers.20.mlp.down_proj.weight [2048, 6144] 0.000013 model.layers.21.self_attn.q_proj.weight [2048, 2048] 0.000059 model.layers.21.self_attn.k_proj.weight [1024, 2048] -0.000016 model.layers.21.self_attn.v_proj.weight [1024, 2048] 0.000006 model.layers.21.self_attn.o_proj.weight [2048, 2048] 0.000017 model.layers.21.mlp.gate_proj.weight [6144, 2048] 0.007248 model.layers.21.mlp.up_proj.weight [6144, 2048] -0.000014 model.layers.21.mlp.down_proj.weight [2048, 6144] -0.000005 model.layers.22.self_attn.q_proj.weight [2048, 2048] 0.000028 model.layers.22.self_attn.k_proj.weight [1024, 2048] -0.000029 model.layers.22.self_attn.v_proj.weight [1024, 2048] -0.000045 model.layers.22.self_attn.o_proj.weight [2048, 2048] 0.000050 model.layers.22.mlp.gate_proj.weight [6144, 2048] 0.008393 model.layers.22.mlp.up_proj.weight [6144, 2048] 0.000002 model.layers.22.mlp.down_proj.weight [2048, 6144] 0.000002 model.layers.23.self_attn.q_proj.weight [2048, 2048] 0.000058 model.layers.23.self_attn.k_proj.weight [1024, 2048] -0.000055 model.layers.23.self_attn.v_proj.weight [1024, 2048] 0.000044 model.layers.23.self_attn.o_proj.weight [2048, 2048] -0.000004 model.layers.23.mlp.gate_proj.weight [6144, 2048] 0.010053 model.layers.23.mlp.up_proj.weight [6144, 2048] -0.000030 model.layers.23.mlp.down_proj.weight [2048, 6144] 0.000003 model.layers.24.self_attn.q_proj.weight [2048, 2048] 0.000135 model.layers.24.self_attn.k_proj.weight [1024, 2048] -0.000025 model.layers.24.self_attn.v_proj.weight [1024, 2048] 0.000060 model.layers.24.self_attn.o_proj.weight [2048, 2048] 0.000004 model.layers.24.mlp.gate_proj.weight [6144, 2048] 0.008813 model.layers.24.mlp.up_proj.weight [6144, 2048] 0.000034 model.layers.24.mlp.down_proj.weight [2048, 6144] -0.000001 model.layers.25.self_attn.q_proj.weight [2048, 2048] 0.000192 model.layers.25.self_attn.k_proj.weight [1024, 2048] 0.000218 model.layers.25.self_attn.v_proj.weight [1024, 2048] -0.000024 model.layers.25.self_attn.o_proj.weight [2048, 2048] -0.000006 model.layers.25.mlp.gate_proj.weight [6144, 2048] 0.006651 model.layers.25.mlp.up_proj.weight [6144, 2048] -0.000004 model.layers.25.mlp.down_proj.weight [2048, 6144] 0.000001 model.layers.26.self_attn.q_proj.weight [2048, 2048] 0.000148 model.layers.26.self_attn.k_proj.weight [1024, 2048] -0.000014 model.layers.26.self_attn.v_proj.weight [1024, 2048] 0.000003 model.layers.26.self_attn.o_proj.weight [2048, 2048] 0.000006 model.layers.26.mlp.gate_proj.weight [6144, 2048] 0.006363 model.layers.26.mlp.up_proj.weight [6144, 2048] 0.000014 model.layers.26.mlp.down_proj.weight [2048, 6144] 0.000014 model.layers.27.self_attn.q_proj.weight [2048, 2048] 0.000033 model.layers.27.self_attn.k_proj.weight [1024, 2048] -0.000059 model.layers.27.self_attn.v_proj.weight [1024, 2048] -0.000051 model.layers.27.self_attn.o_proj.weight [2048, 2048] 0.000006 model.layers.27.mlp.gate_proj.weight [6144, 2048] 0.009203 model.layers.27.mlp.up_proj.weight [6144, 2048] -0.000008 model.layers.27.mlp.down_proj.weight [2048, 6144] 0.000004 lm_head.weight [151936, 2048] 0.053880 Global (across 198 2D layers) mean : 0.001484 min : -0.000105 max : 0.053880 [09] Base vs distilled divergence (all shared layers) Shared tensors: 311 Layer MaxDelta MeanDelta L2Delta CosSim RelErr SNR_dB Chg --------------------------------------------------------------------------------------------------------------------------------------- lm_head.weight 0.00835 0.000098 8.2285 0.99991 0.00353 37.60 Y model.embed_tokens.weight 0.00835 0.000098 8.2285 0.99991 0.00353 37.60 Y model.layers.0.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 253.41 N model.layers.0.mlp.down_proj.weight 0.00291 0.000030 0.4447 0.99999 0.00112 48.57 Y model.layers.0.mlp.gate_proj.weight 0.00278 0.000026 0.4056 1.00000 0.00084 50.66 Y model.layers.0.mlp.up_proj.weight 0.00301 0.000031 0.4568 0.99999 0.00125 47.85 Y model.layers.0.post_attention_layernorm.weight 0.00154 0.000006 0.0033 1.00000 0.00003 70.61 Y model.layers.0.self_attn.k_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 278.19 N model.layers.0.self_attn.k_proj.weight 0.00240 0.000026 0.1657 1.00000 0.00094 50.01 Y model.layers.0.self_attn.o_proj.weight 0.00251 0.000031 0.2599 0.99999 0.00125 48.25 Y model.layers.0.self_attn.q_norm.weight 0.00018 0.000001 0.0002 1.00000 0.00000 101.13 Y model.layers.0.self_attn.q_proj.weight 0.00270 0.000027 0.2340 0.99999 0.00099 49.87 Y model.layers.0.self_attn.v_proj.weight 0.00266 0.000029 0.1828 0.99999 0.00110 48.61 Y model.layers.1.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 252.95 N model.layers.1.mlp.down_proj.weight 0.00307 0.000039 0.5086 0.99999 0.00175 46.25 Y model.layers.1.mlp.gate_proj.weight 0.00272 0.000026 0.4024 1.00000 0.00076 52.03 Y model.layers.1.mlp.up_proj.weight 0.00310 0.000043 0.5255 0.99999 0.00198 45.80 Y model.layers.1.post_attention_layernorm.weight 0.00060 0.000001 0.0008 1.00000 0.00000 89.51 Y model.layers.1.self_attn.k_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 272.09 N model.layers.1.self_attn.k_proj.weight 0.00246 0.000030 0.1827 0.99999 0.00116 48.65 Y model.layers.1.self_attn.o_proj.weight 0.00259 0.000031 0.2623 0.99999 0.00121 48.18 Y model.layers.1.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.25 N model.layers.1.self_attn.q_proj.weight 0.00296 0.000029 0.2489 0.99999 0.00111 49.17 Y model.layers.1.self_attn.v_proj.weight 0.00258 0.000029 0.1793 0.99999 0.00104 49.01 Y model.layers.10.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 271.30 N model.layers.10.mlp.down_proj.weight 0.00322 0.000044 0.5900 0.99999 0.00173 45.89 Y model.layers.10.mlp.gate_proj.weight 0.00347 0.000036 0.5295 0.99999 0.00124 48.14 Y model.layers.10.mlp.up_proj.weight 0.00333 0.000039 0.5501 0.99999 0.00143 47.00 Y model.layers.10.post_attention_layernorm.weight 0.00214 0.000002 0.0028 1.00000 0.00000 80.97 Y model.layers.10.self_attn.k_norm.weight 0.00071 0.000013 0.0009 1.00000 0.00001 90.93 Y model.layers.10.self_attn.k_proj.weight 0.00366 0.000039 0.2194 0.99999 0.00151 46.82 Y model.layers.10.self_attn.o_proj.weight 0.00312 0.000044 0.3382 0.99999 0.00179 45.71 Y model.layers.10.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.98 N model.layers.10.self_attn.q_proj.weight 0.00278 0.000038 0.3137 0.99999 0.00146 47.07 Y model.layers.10.self_attn.v_proj.weight 0.00298 0.000039 0.2219 0.99999 0.00144 47.05 Y model.layers.11.input_layernorm.weight 0.00099 0.000000 0.0010 1.00000 0.00000 93.98 Y model.layers.11.mlp.down_proj.weight 0.00330 0.000042 0.5863 0.99999 0.00160 46.23 Y model.layers.11.mlp.gate_proj.weight 0.00365 0.000038 0.5469 0.99999 0.00136 47.43 Y model.layers.11.mlp.up_proj.weight 0.00340 0.000039 0.5542 0.99999 0.00139 47.13 Y model.layers.11.post_attention_layernorm.weight 0.00206 0.000002 0.0028 1.00000 0.00000 82.01 Y model.layers.11.self_attn.k_norm.weight 0.00061 0.000006 0.0006 1.00000 0.00000 92.39 Y model.layers.11.self_attn.k_proj.weight 0.00314 0.000044 0.2314 0.99999 0.00175 46.42 Y model.layers.11.self_attn.o_proj.weight 0.00332 0.000050 0.3651 0.99998 0.00224 44.60 Y model.layers.11.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.53 N model.layers.11.self_attn.q_proj.weight 0.00308 0.000042 0.3231 0.99999 0.00157 47.33 Y model.layers.11.self_attn.v_proj.weight 0.00307 0.000047 0.2479 0.99999 0.00187 45.64 Y model.layers.12.input_layernorm.weight 0.00122 0.000001 0.0012 1.00000 0.00000 93.71 Y model.layers.12.mlp.down_proj.weight 0.00352 0.000044 0.6074 0.99999 0.00168 45.92 Y model.layers.12.mlp.gate_proj.weight 0.00385 0.000041 0.5746 0.99999 0.00149 46.76 Y model.layers.12.mlp.up_proj.weight 0.00341 0.000040 0.5733 0.99999 0.00145 46.80 Y model.layers.12.post_attention_layernorm.weight 0.00207 0.000003 0.0035 1.00000 0.00000 80.16 Y model.layers.12.self_attn.k_norm.weight 0.00067 0.000020 0.0012 1.00000 0.00001 87.90 Y model.layers.12.self_attn.k_proj.weight 0.00305 0.000043 0.2318 0.99999 0.00173 46.18 Y model.layers.12.self_attn.o_proj.weight 0.00328 0.000049 0.3622 0.99998 0.00207 44.67 Y model.layers.12.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.85 N model.layers.12.self_attn.q_proj.weight 0.00286 0.000040 0.3211 0.99999 0.00153 47.00 Y model.layers.12.self_attn.v_proj.weight 0.00271 0.000042 0.2334 0.99999 0.00158 46.42 Y model.layers.13.input_layernorm.weight 0.00095 0.000000 0.0009 1.00000 0.00000 96.85 Y model.layers.13.mlp.down_proj.weight 0.00358 0.000047 0.6369 0.99999 0.00184 45.29 Y model.layers.13.mlp.gate_proj.weight 0.00328 0.000043 0.5999 0.99999 0.00165 46.08 Y model.layers.13.mlp.up_proj.weight 0.00400 0.000042 0.5943 0.99999 0.00157 46.26 Y model.layers.13.post_attention_layernorm.weight 0.00226 0.000002 0.0030 1.00000 0.00000 81.90 Y model.layers.13.self_attn.k_norm.weight 0.00053 0.000014 0.0008 1.00000 0.00001 89.00 Y model.layers.13.self_attn.k_proj.weight 0.00278 0.000044 0.2374 0.99999 0.00188 45.55 Y model.layers.13.self_attn.o_proj.weight 0.00347 0.000051 0.3762 0.99998 0.00225 44.08 Y model.layers.13.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.51 N model.layers.13.self_attn.q_proj.weight 0.00315 0.000042 0.3262 0.99999 0.00160 46.96 Y model.layers.13.self_attn.v_proj.weight 0.00307 0.000042 0.2380 0.99999 0.00161 46.20 Y model.layers.14.input_layernorm.weight 0.00164 0.000001 0.0016 1.00000 0.00000 94.24 Y model.layers.14.mlp.down_proj.weight 0.00346 0.000049 0.6642 0.99998 0.00195 44.86 Y model.layers.14.mlp.gate_proj.weight 0.00356 0.000047 0.6357 0.99998 0.00189 45.20 Y model.layers.14.mlp.up_proj.weight 0.00351 0.000045 0.6247 0.99999 0.00175 45.55 Y model.layers.14.post_attention_layernorm.weight 0.00279 0.000003 0.0041 1.00000 0.00000 79.52 Y model.layers.14.self_attn.k_norm.weight 0.00095 0.000017 0.0012 1.00000 0.00001 86.20 Y model.layers.14.self_attn.k_proj.weight 0.00294 0.000047 0.2449 0.99998 0.00202 45.22 Y model.layers.14.self_attn.o_proj.weight 0.00346 0.000053 0.3908 0.99998 0.00234 43.76 Y model.layers.14.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.41 N model.layers.14.self_attn.q_proj.weight 0.00294 0.000044 0.3352 0.99999 0.00173 46.44 Y model.layers.14.self_attn.v_proj.weight 0.00388 0.000045 0.2499 0.99999 0.00177 45.56 Y model.layers.15.input_layernorm.weight 0.00200 0.000001 0.0020 1.00000 0.00000 95.34 Y model.layers.15.mlp.down_proj.weight 0.00398 0.000051 0.6876 0.99998 0.00201 44.60 Y model.layers.15.mlp.gate_proj.weight 0.00379 0.000049 0.6628 0.99998 0.00195 44.89 Y model.layers.15.mlp.up_proj.weight 0.00357 0.000047 0.6463 0.99999 0.00176 45.45 Y model.layers.15.post_attention_layernorm.weight 0.00395 0.000004 0.0054 1.00000 0.00001 77.13 Y model.layers.15.self_attn.k_norm.weight 0.00094 0.000012 0.0011 1.00000 0.00001 90.67 Y model.layers.15.self_attn.k_proj.weight 0.00307 0.000052 0.2690 0.99998 0.00233 44.00 Y model.layers.15.self_attn.o_proj.weight 0.00349 0.000056 0.4069 0.99998 0.00240 43.60 Y model.layers.15.self_attn.q_norm.weight 0.00089 0.000009 0.0009 1.00000 0.00001 86.00 Y model.layers.15.self_attn.q_proj.weight 0.00331 0.000050 0.3798 0.99999 0.00193 45.57 Y model.layers.15.self_attn.v_proj.weight 0.00330 0.000051 0.2726 0.99998 0.00202 44.78 Y model.layers.16.input_layernorm.weight 0.00205 0.000001 0.0020 1.00000 0.00000 94.71 Y model.layers.16.mlp.down_proj.weight 0.00372 0.000051 0.6956 0.99998 0.00198 44.58 Y model.layers.16.mlp.gate_proj.weight 0.00354 0.000052 0.6944 0.99998 0.00209 44.43 Y model.layers.16.mlp.up_proj.weight 0.00415 0.000049 0.6728 0.99999 0.00180 45.33 Y model.layers.16.post_attention_layernorm.weight 0.00370 0.000004 0.0052 1.00000 0.00001 77.43 Y model.layers.16.self_attn.k_norm.weight 0.00192 0.000021 0.0020 1.00000 0.00001 82.55 Y model.layers.16.self_attn.k_proj.weight 0.00351 0.000053 0.2705 0.99998 0.00236 44.28 Y model.layers.16.self_attn.o_proj.weight 0.00360 0.000056 0.4125 0.99998 0.00244 43.50 Y model.layers.16.self_attn.q_norm.weight 0.00018 0.000001 0.0002 1.00000 0.00000 100.71 Y model.layers.16.self_attn.q_proj.weight 0.00307 0.000048 0.3639 0.99999 0.00178 46.51 Y model.layers.16.self_attn.v_proj.weight 0.00346 0.000049 0.2702 0.99998 0.00194 44.89 Y model.layers.17.input_layernorm.weight 0.00184 0.000001 0.0018 1.00000 0.00000 100.52 Y model.layers.17.mlp.down_proj.weight 0.00383 0.000050 0.6924 0.99998 0.00186 44.89 Y model.layers.17.mlp.gate_proj.weight 0.00432 0.000052 0.6950 0.99998 0.00207 44.41 Y model.layers.17.mlp.up_proj.weight 0.00419 0.000046 0.6594 0.99999 0.00163 45.83 Y model.layers.17.post_attention_layernorm.weight 0.00394 0.000007 0.0073 1.00000 0.00001 76.41 Y model.layers.17.self_attn.k_norm.weight 0.00046 0.000007 0.0006 1.00000 0.00000 93.71 Y model.layers.17.self_attn.k_proj.weight 0.00357 0.000053 0.2760 0.99998 0.00227 44.09 Y model.layers.17.self_attn.o_proj.weight 0.00359 0.000059 0.4249 0.99998 0.00250 43.35 Y model.layers.17.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.54 N model.layers.17.self_attn.q_proj.weight 0.00327 0.000051 0.3870 0.99999 0.00195 45.58 Y model.layers.17.self_attn.v_proj.weight 0.00369 0.000054 0.2911 0.99998 0.00205 44.58 Y model.layers.18.input_layernorm.weight 0.00153 0.000001 0.0015 1.00000 0.00000 102.37 Y model.layers.18.mlp.down_proj.weight 0.00391 0.000049 0.6975 0.99998 0.00180 45.06 Y model.layers.18.mlp.gate_proj.weight 0.00538 0.000052 0.7063 0.99998 0.00207 44.36 Y model.layers.18.mlp.up_proj.weight 0.00451 0.000047 0.6702 0.99999 0.00165 45.71 Y model.layers.18.post_attention_layernorm.weight 0.00314 0.000004 0.0046 1.00000 0.00000 81.37 Y model.layers.18.self_attn.k_norm.weight 0.00085 0.000010 0.0010 1.00000 0.00001 87.85 Y model.layers.18.self_attn.k_proj.weight 0.00306 0.000053 0.2766 0.99998 0.00240 43.85 Y model.layers.18.self_attn.o_proj.weight 0.00364 0.000059 0.4308 0.99997 0.00263 42.70 Y model.layers.18.self_attn.q_norm.weight 0.00012 0.000001 0.0001 1.00000 0.00000 103.53 Y model.layers.18.self_attn.q_proj.weight 0.00320 0.000048 0.3734 0.99999 0.00179 46.15 Y model.layers.18.self_attn.v_proj.weight 0.00336 0.000053 0.2861 0.99998 0.00211 44.22 Y model.layers.19.input_layernorm.weight 0.00160 0.000001 0.0016 1.00000 0.00000 104.39 Y model.layers.19.mlp.down_proj.weight 0.00453 0.000050 0.7104 0.99998 0.00177 45.17 Y model.layers.19.mlp.gate_proj.weight 0.00410 0.000053 0.7214 0.99998 0.00208 44.30 Y model.layers.19.mlp.up_proj.weight 0.00522 0.000048 0.6880 0.99999 0.00165 45.64 Y model.layers.19.post_attention_layernorm.weight 0.00383 0.000004 0.0054 1.00000 0.00000 80.83 Y model.layers.19.self_attn.k_norm.weight 0.00066 0.000010 0.0008 1.00000 0.00001 89.56 Y model.layers.19.self_attn.k_proj.weight 0.00372 0.000056 0.2909 0.99998 0.00248 43.45 Y model.layers.19.self_attn.o_proj.weight 0.00391 0.000060 0.4523 0.99997 0.00262 42.61 Y model.layers.19.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.95 N model.layers.19.self_attn.q_proj.weight 0.00365 0.000054 0.4036 0.99999 0.00199 45.74 Y model.layers.19.self_attn.v_proj.weight 0.00397 0.000061 0.3253 0.99998 0.00251 43.08 Y model.layers.2.input_layernorm.weight 0.00003 0.000000 0.0000 1.00000 0.00000 105.97 Y model.layers.2.mlp.down_proj.weight 0.00309 0.000034 0.4665 0.99999 0.00140 47.42 Y model.layers.2.mlp.gate_proj.weight 0.00284 0.000022 0.3744 1.00000 0.00061 52.95 Y model.layers.2.mlp.up_proj.weight 0.00284 0.000035 0.4758 0.99999 0.00148 46.96 Y model.layers.2.post_attention_layernorm.weight 0.00164 0.000002 0.0024 1.00000 0.00000 79.71 Y model.layers.2.self_attn.k_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 273.01 N model.layers.2.self_attn.k_proj.weight 0.00269 0.000032 0.1865 0.99999 0.00123 48.39 Y model.layers.2.self_attn.o_proj.weight 0.00289 0.000032 0.2681 0.99999 0.00131 47.75 Y model.layers.2.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.61 N model.layers.2.self_attn.q_proj.weight 0.00246 0.000030 0.2516 0.99999 0.00114 49.02 Y model.layers.2.self_attn.v_proj.weight 0.00323 0.000029 0.1817 0.99999 0.00108 48.89 Y model.layers.20.input_layernorm.weight 0.00168 0.000001 0.0017 1.00000 0.00000 105.09 Y model.layers.20.mlp.down_proj.weight 0.00505 0.000047 0.6766 0.99999 0.00161 45.81 Y model.layers.20.mlp.gate_proj.weight 0.00436 0.000050 0.6897 0.99998 0.00185 45.02 Y model.layers.20.mlp.up_proj.weight 0.00443 0.000045 0.6601 0.99999 0.00151 46.22 Y model.layers.20.post_attention_layernorm.weight 0.00268 0.000002 0.0033 1.00000 0.00000 86.18 Y model.layers.20.self_attn.k_norm.weight 0.00015 0.000002 0.0002 1.00000 0.00000 103.31 Y model.layers.20.self_attn.k_proj.weight 0.00351 0.000054 0.2880 0.99998 0.00231 43.95 Y model.layers.20.self_attn.o_proj.weight 0.00394 0.000060 0.4510 0.99997 0.00250 43.01 Y model.layers.20.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.02 N model.layers.20.self_attn.q_proj.weight 0.00348 0.000052 0.3948 0.99999 0.00189 46.05 Y model.layers.20.self_attn.v_proj.weight 0.00371 0.000060 0.3226 0.99998 0.00235 43.49 Y model.layers.21.input_layernorm.weight 0.00166 0.000001 0.0017 1.00000 0.00000 107.25 Y model.layers.21.mlp.down_proj.weight 0.00422 0.000045 0.6490 0.99999 0.00151 46.33 Y model.layers.21.mlp.gate_proj.weight 0.00478 0.000045 0.6403 0.99999 0.00163 45.88 Y model.layers.21.mlp.up_proj.weight 0.00461 0.000041 0.6157 0.99999 0.00136 46.96 Y model.layers.21.post_attention_layernorm.weight 0.00370 0.000004 0.0047 1.00000 0.00000 84.07 Y model.layers.21.self_attn.k_norm.weight 0.00037 0.000012 0.0006 1.00000 0.00001 90.93 Y model.layers.21.self_attn.k_proj.weight 0.00367 0.000055 0.2872 0.99998 0.00234 44.16 Y model.layers.21.self_attn.o_proj.weight 0.00447 0.000059 0.4621 0.99998 0.00236 43.23 Y model.layers.21.self_attn.q_norm.weight 0.00053 0.000011 0.0008 1.00000 0.00001 87.66 Y model.layers.21.self_attn.q_proj.weight 0.00397 0.000048 0.3821 0.99999 0.00172 46.36 Y model.layers.21.self_attn.v_proj.weight 0.00406 0.000061 0.3313 0.99998 0.00226 43.94 Y model.layers.22.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 293.24 N model.layers.22.mlp.down_proj.weight 0.00450 0.000042 0.6191 0.99999 0.00141 46.89 Y model.layers.22.mlp.gate_proj.weight 0.00479 0.000042 0.5993 0.99999 0.00148 46.60 Y model.layers.22.mlp.up_proj.weight 0.00500 0.000039 0.5853 0.99999 0.00127 47.49 Y model.layers.22.post_attention_layernorm.weight 0.00310 0.000004 0.0049 1.00000 0.00000 84.63 Y model.layers.22.self_attn.k_norm.weight 0.00101 0.000026 0.0015 1.00000 0.00002 82.39 Y model.layers.22.self_attn.k_proj.weight 0.00382 0.000058 0.3024 0.99998 0.00243 43.63 Y model.layers.22.self_attn.o_proj.weight 0.00444 0.000066 0.4993 0.99997 0.00263 42.46 Y model.layers.22.self_attn.q_norm.weight 0.00073 0.000012 0.0010 1.00000 0.00001 85.04 Y model.layers.22.self_attn.q_proj.weight 0.00376 0.000055 0.4232 0.99999 0.00202 45.25 Y model.layers.22.self_attn.v_proj.weight 0.00386 0.000064 0.3426 0.99998 0.00237 43.52 Y model.layers.23.input_layernorm.weight 0.00183 0.000001 0.0018 1.00000 0.00000 109.30 Y model.layers.23.mlp.down_proj.weight 0.00555 0.000042 0.6098 0.99999 0.00138 47.11 Y model.layers.23.mlp.gate_proj.weight 0.00572 0.000040 0.5813 0.99999 0.00140 46.98 Y model.layers.23.mlp.up_proj.weight 0.00586 0.000038 0.5674 0.99999 0.00123 47.79 Y model.layers.23.post_attention_layernorm.weight 0.00265 0.000003 0.0038 1.00000 0.00000 87.79 Y model.layers.23.self_attn.k_norm.weight 0.00066 0.000015 0.0009 1.00000 0.00001 89.57 Y model.layers.23.self_attn.k_proj.weight 0.00394 0.000060 0.3171 0.99998 0.00254 43.10 Y model.layers.23.self_attn.o_proj.weight 0.00522 0.000063 0.4985 0.99997 0.00247 42.55 Y model.layers.23.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 264.66 N model.layers.23.self_attn.q_proj.weight 0.00385 0.000056 0.4306 0.99998 0.00214 44.50 Y model.layers.23.self_attn.v_proj.weight 0.00435 0.000066 0.3622 0.99997 0.00248 42.88 Y model.layers.24.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 297.45 N model.layers.24.mlp.down_proj.weight 0.00490 0.000043 0.6213 0.99999 0.00139 46.98 Y model.layers.24.mlp.gate_proj.weight 0.00485 0.000041 0.5920 0.99999 0.00144 46.73 Y model.layers.24.mlp.up_proj.weight 0.00537 0.000039 0.5803 0.99999 0.00125 47.64 Y model.layers.24.post_attention_layernorm.weight 0.00253 0.000003 0.0038 1.00000 0.00000 87.93 Y model.layers.24.self_attn.k_norm.weight 0.00043 0.000005 0.0004 1.00000 0.00000 96.20 Y model.layers.24.self_attn.k_proj.weight 0.00393 0.000061 0.3163 0.99998 0.00260 43.11 Y model.layers.24.self_attn.o_proj.weight 0.00444 0.000064 0.5073 0.99997 0.00248 42.58 Y model.layers.24.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 264.61 N model.layers.24.self_attn.q_proj.weight 0.00375 0.000054 0.4165 0.99998 0.00200 45.18 Y model.layers.24.self_attn.v_proj.weight 0.00462 0.000071 0.3789 0.99997 0.00262 42.54 Y model.layers.25.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 299.47 N model.layers.25.mlp.down_proj.weight 0.00560 0.000043 0.6433 0.99999 0.00143 46.60 Y model.layers.25.mlp.gate_proj.weight 0.00534 0.000044 0.6358 0.99999 0.00157 45.98 Y model.layers.25.mlp.up_proj.weight 0.00577 0.000040 0.6062 0.99999 0.00129 47.27 Y model.layers.25.post_attention_layernorm.weight 0.00295 0.000004 0.0044 1.00000 0.00000 87.35 Y model.layers.25.self_attn.k_norm.weight 0.00090 0.000018 0.0013 1.00000 0.00001 86.33 Y model.layers.25.self_attn.k_proj.weight 0.00451 0.000069 0.3509 0.99997 0.00306 41.90 Y model.layers.25.self_attn.o_proj.weight 0.00551 0.000070 0.5722 0.99997 0.00257 41.95 Y model.layers.25.self_attn.q_norm.weight 0.00082 0.000007 0.0008 1.00000 0.00000 86.70 Y model.layers.25.self_attn.q_proj.weight 0.00470 0.000063 0.4760 0.99998 0.00241 43.76 Y model.layers.25.self_attn.v_proj.weight 0.00507 0.000072 0.4023 0.99997 0.00242 42.90 Y model.layers.26.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 300.85 N model.layers.26.mlp.down_proj.weight 0.00528 0.000045 0.6640 0.99999 0.00150 46.17 Y model.layers.26.mlp.gate_proj.weight 0.00610 0.000047 0.6667 0.99999 0.00168 45.49 Y model.layers.26.mlp.up_proj.weight 0.00630 0.000042 0.6272 0.99999 0.00136 46.92 Y model.layers.26.post_attention_layernorm.weight 0.00246 0.000003 0.0040 1.00000 0.00000 89.19 Y model.layers.26.self_attn.k_norm.weight 0.00059 0.000009 0.0008 1.00000 0.00001 88.71 Y model.layers.26.self_attn.k_proj.weight 0.00468 0.000078 0.3770 0.99996 0.00355 41.02 Y model.layers.26.self_attn.o_proj.weight 0.00565 0.000078 0.5811 0.99997 0.00294 41.78 Y model.layers.26.self_attn.q_norm.weight 0.00061 0.000012 0.0008 1.00000 0.00001 85.86 Y model.layers.26.self_attn.q_proj.weight 0.00470 0.000069 0.4942 0.99998 0.00247 44.14 Y model.layers.26.self_attn.v_proj.weight 0.00506 0.000076 0.4092 0.99997 0.00259 42.51 Y model.layers.27.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 298.18 N model.layers.27.mlp.down_proj.weight 0.00674 0.000048 0.6742 0.99999 0.00173 45.40 Y model.layers.27.mlp.gate_proj.weight 0.00656 0.000044 0.6610 0.99999 0.00150 46.06 Y model.layers.27.mlp.up_proj.weight 0.00679 0.000041 0.6234 0.99999 0.00137 46.86 Y model.layers.27.post_attention_layernorm.weight 0.00257 0.000002 0.0035 1.00000 0.00000 95.48 Y model.layers.27.self_attn.k_norm.weight 0.00079 0.000015 0.0010 1.00000 0.00001 87.28 Y model.layers.27.self_attn.k_proj.weight 0.00404 0.000056 0.3102 0.99998 0.00212 43.99 Y model.layers.27.self_attn.o_proj.weight 0.00525 0.000063 0.4910 0.99998 0.00229 43.32 Y model.layers.27.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.52 N model.layers.27.self_attn.q_proj.weight 0.00464 0.000056 0.4464 0.99998 0.00207 44.24 Y model.layers.27.self_attn.v_proj.weight 0.00400 0.000062 0.3477 0.99998 0.00210 43.89 Y model.layers.3.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 263.62 N model.layers.3.mlp.down_proj.weight 0.00308 0.000035 0.4759 0.99999 0.00144 47.21 Y model.layers.3.mlp.gate_proj.weight 0.00297 0.000023 0.3753 1.00000 0.00062 52.82 Y model.layers.3.mlp.up_proj.weight 0.00294 0.000034 0.4731 0.99999 0.00143 47.21 Y model.layers.3.post_attention_layernorm.weight 0.00071 0.000001 0.0010 1.00000 0.00000 88.33 Y model.layers.3.self_attn.k_norm.weight 0.00082 0.000031 0.0016 1.00000 0.00002 85.17 Y model.layers.3.self_attn.k_proj.weight 0.00264 0.000031 0.1822 0.99999 0.00120 48.77 Y model.layers.3.self_attn.o_proj.weight 0.00283 0.000030 0.2614 0.99999 0.00114 48.47 Y model.layers.3.self_attn.q_norm.weight 0.00144 0.000011 0.0014 1.00000 0.00001 83.93 Y model.layers.3.self_attn.q_proj.weight 0.00288 0.000030 0.2514 0.99999 0.00111 49.14 Y model.layers.3.self_attn.v_proj.weight 0.00283 0.000028 0.1784 0.99999 0.00096 49.56 Y model.layers.4.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 263.72 N model.layers.4.mlp.down_proj.weight 0.00302 0.000036 0.4926 0.99999 0.00148 46.92 Y model.layers.4.mlp.gate_proj.weight 0.00314 0.000024 0.3943 1.00000 0.00068 52.13 Y model.layers.4.mlp.up_proj.weight 0.00315 0.000035 0.4877 0.99999 0.00145 47.03 Y model.layers.4.post_attention_layernorm.weight 0.00110 0.000001 0.0013 1.00000 0.00000 86.80 Y model.layers.4.self_attn.k_norm.weight 0.00041 0.000004 0.0004 1.00000 0.00000 102.18 Y model.layers.4.self_attn.k_proj.weight 0.00231 0.000034 0.1930 0.99999 0.00135 47.99 Y model.layers.4.self_attn.o_proj.weight 0.00297 0.000033 0.2751 0.99999 0.00128 47.83 Y model.layers.4.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 262.77 N model.layers.4.self_attn.q_proj.weight 0.00261 0.000032 0.2630 0.99999 0.00121 48.62 Y model.layers.4.self_attn.v_proj.weight 0.00288 0.000030 0.1861 0.99999 0.00106 49.05 Y model.layers.5.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.21 N model.layers.5.mlp.down_proj.weight 0.00332 0.000034 0.4893 0.99999 0.00131 47.54 Y model.layers.5.mlp.gate_proj.weight 0.00289 0.000028 0.4340 1.00000 0.00089 50.28 Y model.layers.5.mlp.up_proj.weight 0.00301 0.000033 0.4778 0.99999 0.00126 47.81 Y model.layers.5.post_attention_layernorm.weight 0.00134 0.000001 0.0015 1.00000 0.00000 83.84 Y model.layers.5.self_attn.k_norm.weight 0.00109 0.000009 0.0011 1.00000 0.00000 93.01 Y model.layers.5.self_attn.k_proj.weight 0.00241 0.000033 0.1896 0.99999 0.00126 48.26 Y model.layers.5.self_attn.o_proj.weight 0.00307 0.000035 0.2836 0.99999 0.00136 47.62 Y model.layers.5.self_attn.q_norm.weight 0.00098 0.000008 0.0010 1.00000 0.00000 86.69 Y model.layers.5.self_attn.q_proj.weight 0.00267 0.000032 0.2687 0.99999 0.00117 48.71 Y model.layers.5.self_attn.v_proj.weight 0.00282 0.000033 0.1941 0.99999 0.00120 48.40 Y model.layers.6.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 264.54 N model.layers.6.mlp.down_proj.weight 0.00294 0.000036 0.5101 0.99999 0.00142 47.07 Y model.layers.6.mlp.gate_proj.weight 0.00287 0.000029 0.4471 0.99999 0.00094 49.90 Y model.layers.6.mlp.up_proj.weight 0.00273 0.000034 0.4868 0.99999 0.00129 47.69 Y model.layers.6.post_attention_layernorm.weight 0.00194 0.000002 0.0025 1.00000 0.00000 80.77 Y model.layers.6.self_attn.k_norm.weight 0.00061 0.000012 0.0008 1.00000 0.00000 93.05 Y model.layers.6.self_attn.k_proj.weight 0.00246 0.000039 0.2090 0.99999 0.00160 47.22 Y model.layers.6.self_attn.o_proj.weight 0.00284 0.000037 0.2948 0.99999 0.00149 46.95 Y model.layers.6.self_attn.q_norm.weight 0.00023 0.000002 0.0002 1.00000 0.00000 99.95 Y model.layers.6.self_attn.q_proj.weight 0.00280 0.000035 0.2785 0.99999 0.00135 48.19 Y model.layers.6.self_attn.v_proj.weight 0.00285 0.000032 0.1911 0.99999 0.00114 48.73 Y model.layers.7.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 268.35 N model.layers.7.mlp.down_proj.weight 0.00320 0.000038 0.5276 0.99999 0.00153 46.64 Y model.layers.7.mlp.gate_proj.weight 0.00307 0.000029 0.4485 1.00000 0.00091 50.14 Y model.layers.7.mlp.up_proj.weight 0.00301 0.000035 0.5010 0.99999 0.00136 47.32 Y model.layers.7.post_attention_layernorm.weight 0.00055 0.000000 0.0006 1.00000 0.00000 94.82 Y model.layers.7.self_attn.k_norm.weight 0.00106 0.000019 0.0012 1.00000 0.00001 88.06 Y model.layers.7.self_attn.k_proj.weight 0.00264 0.000036 0.2023 0.99999 0.00148 47.17 Y model.layers.7.self_attn.o_proj.weight 0.00292 0.000037 0.2979 0.99999 0.00146 46.90 Y model.layers.7.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.28 N model.layers.7.self_attn.q_proj.weight 0.00261 0.000035 0.2821 0.99999 0.00134 47.80 Y model.layers.7.self_attn.v_proj.weight 0.00237 0.000033 0.1980 0.99999 0.00123 48.16 Y model.layers.8.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 268.99 N model.layers.8.mlp.down_proj.weight 0.00306 0.000039 0.5414 0.99999 0.00153 46.65 Y model.layers.8.mlp.gate_proj.weight 0.00285 0.000032 0.4766 0.99999 0.00105 49.21 Y model.layers.8.mlp.up_proj.weight 0.00293 0.000037 0.5160 0.99999 0.00140 47.17 Y model.layers.8.post_attention_layernorm.weight 0.00121 0.000002 0.0019 1.00000 0.00000 84.34 Y model.layers.8.self_attn.k_norm.weight 0.00036 0.000013 0.0006 1.00000 0.00001 90.26 Y model.layers.8.self_attn.k_proj.weight 0.00277 0.000036 0.2029 0.99999 0.00148 47.34 Y model.layers.8.self_attn.o_proj.weight 0.00294 0.000038 0.3006 0.99999 0.00153 46.67 Y model.layers.8.self_attn.q_norm.weight 0.00035 0.000003 0.0004 1.00000 0.00000 93.93 Y model.layers.8.self_attn.q_proj.weight 0.00281 0.000034 0.2798 0.99999 0.00133 48.09 Y model.layers.8.self_attn.v_proj.weight 0.00269 0.000033 0.1968 0.99999 0.00117 48.32 Y model.layers.9.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 271.40 N model.layers.9.mlp.down_proj.weight 0.00362 0.000041 0.5626 0.99999 0.00159 46.37 Y model.layers.9.mlp.gate_proj.weight 0.00291 0.000033 0.4943 0.99999 0.00110 48.84 Y model.layers.9.mlp.up_proj.weight 0.00317 0.000037 0.5275 0.99999 0.00135 47.33 Y model.layers.9.post_attention_layernorm.weight 0.00178 0.000002 0.0025 1.00000 0.00000 82.01 Y model.layers.9.self_attn.k_norm.weight 0.00100 0.000015 0.0011 1.00000 0.00001 88.79 Y model.layers.9.self_attn.k_proj.weight 0.00293 0.000038 0.2147 0.99999 0.00157 46.80 Y model.layers.9.self_attn.o_proj.weight 0.00284 0.000042 0.3233 0.99999 0.00169 46.32 Y model.layers.9.self_attn.q_norm.weight 0.00060 0.000005 0.0006 1.00000 0.00000 91.63 Y model.layers.9.self_attn.q_proj.weight 0.00291 0.000037 0.2997 0.99999 0.00142 47.61 Y model.layers.9.self_attn.v_proj.weight 0.00263 0.000038 0.2159 0.99999 0.00144 47.33 Y model.norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 280.25 N Changed : 277 / 311 Unchanged: 34 / 311 Unchanged (first 10): ['model.layers.0.input_layernorm.weight', 'model.layers.0.self_attn.k_norm.weight', 'model.layers.1.input_layernorm.weight', 'model.layers.1.self_attn.k_norm.weight', 'model.layers.1.self_attn.q_norm.weight', 'model.layers.10.input_layernorm.weight', 'model.layers.10.self_attn.q_norm.weight', 'model.layers.11.self_attn.q_norm.weight', 'model.layers.12.self_attn.q_norm.weight', 'model.layers.13.self_attn.q_norm.weight'] Note: Unchanged tensors are primarily normalization layers (input_layernorm, q_norm, k_norm, model.norm). This demonstrates that the SFT/KD process modified the primary semantic projection weights (attention and MLP projections) while preserving basic layer scaling characteristics. [10] Cosine similarity distribution histogram Range Count Histogram [ -inf, 0.90000) 0 [ 0.90000, 0.99000) 0 [ 0.99000, 0.99900) 0 [ 0.99900, 0.99990) 0 [ 0.99990, 0.99999) 127 ############################ [ 0.99999, 1.00001) 184 ######################################## [ 1.00001, 1.00100) 0 [ 1.00100, inf) 0 [11] Attention geometry per transformer block Query heads: 16 | KV heads: 8 | head_dim: 128 | GQA: 2:1 Blk Q shape K shape V shape O shape Q L2 K L2 V L2 O L2 ---------------------------------------------------------------------------------------------------------------------------------- 0 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.901 52.450 49.252 67.170 1 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.518 49.452 50.624 67.294 2 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.066 49.005 50.551 65.435 3 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.017 50.006 53.659 69.315 4 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.937 48.394 52.777 67.797 5 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 73.221 49.084 51.053 68.198 6 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.503 47.997 52.187 65.590 7 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 69.230 46.204 50.629 65.907 8 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.971 47.226 51.273 64.765 9 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.985 46.960 50.199 66.940 10 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.798 48.128 49.962 65.289 11 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 75.117 48.444 47.471 61.987 12 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.912 47.209 48.870 62.022 13 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.686 44.996 48.618 60.200 14 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.351 44.671 47.417 60.245 15 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.109 42.654 47.263 61.616 16 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 77.028 44.274 47.469 61.723 17 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 73.600 44.185 49.308 62.506 18 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 75.780 43.076 46.493 58.817 19 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 78.144 43.280 46.370 61.106 20 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 79.257 45.357 48.235 63.751 21 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 79.468 46.374 52.175 67.043 22 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 77.429 45.910 51.373 66.308 23 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.309 45.286 50.443 66.829 24 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 75.594 45.236 50.747 68.236 25 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 73.422 43.686 56.192 71.640 26 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 79.566 42.406 54.631 71.338 27 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.699 49.078 54.406 71.960 [12] MLP feed-forward geometry per transformer block intermediate_size: 6144 | activation: silu Blk Gate shape Up shape Down shape Gate L2 Up L2 Down L2 GateSp UpSp DnSp --------------------------------------------------------------------------------------------------------------------------------------- 0 [6144, 2048] [6144, 2048] [2048, 6144] 138.356 112.786 119.234 0.00002 0.00002 0.00002 1 [6144, 2048] [6144, 2048] [2048, 6144] 160.751 102.435 104.364 0.00002 0.00004 0.00003 2 [6144, 2048] [6144, 2048] [2048, 6144] 166.168 105.972 109.577 0.00002 0.00003 0.00003 3 [6144, 2048] [6144, 2048] [2048, 6144] 164.227 108.473 109.132 0.00002 0.00003 0.00003 4 [6144, 2048] [6144, 2048] [2048, 6144] 159.325 109.481 109.280 0.00002 0.00003 0.00003 5 [6144, 2048] [6144, 2048] [2048, 6144] 141.749 117.417 116.503 0.00002 0.00002 0.00003 6 [6144, 2048] [6144, 2048] [2048, 6144] 139.719 117.968 115.044 0.00002 0.00002 0.00003 7 [6144, 2048] [6144, 2048] [2048, 6144] 144.020 116.378 113.238 0.00002 0.00003 0.00003 8 [6144, 2048] [6144, 2048] [2048, 6144] 137.524 117.760 116.430 0.00002 0.00002 0.00002 9 [6144, 2048] [6144, 2048] [2048, 6144] 136.793 122.606 117.158 0.00002 0.00002 0.00003 10 [6144, 2048] [6144, 2048] [2048, 6144] 135.160 123.157 116.162 0.00002 0.00002 0.00003 11 [6144, 2048] [6144, 2048] [2048, 6144] 128.540 125.940 120.142 0.00002 0.00003 0.00002 12 [6144, 2048] [6144, 2048] [2048, 6144] 125.090 125.435 120.075 0.00002 0.00002 0.00003 13 [6144, 2048] [6144, 2048] [2048, 6144] 120.812 122.147 117.098 0.00003 0.00003 0.00003 14 [6144, 2048] [6144, 2048] [2048, 6144] 115.589 118.312 116.246 0.00003 0.00002 0.00003 15 [6144, 2048] [6144, 2048] [2048, 6144] 116.356 120.975 116.761 0.00003 0.00003 0.00003 16 [6144, 2048] [6144, 2048] [2048, 6144] 115.659 124.277 117.831 0.00003 0.00002 0.00003 17 [6144, 2048] [6144, 2048] [2048, 6144] 115.404 129.050 121.534 0.00003 0.00002 0.00003 18 [6144, 2048] [6144, 2048] [2048, 6144] 116.670 129.370 124.809 0.00003 0.00003 0.00002 19 [6144, 2048] [6144, 2048] [2048, 6144] 118.280 131.654 128.726 0.00003 0.00002 0.00002 20 [6144, 2048] [6144, 2048] [2048, 6144] 122.886 134.977 132.033 0.00003 0.00002 0.00002 21 [6144, 2048] [6144, 2048] [2048, 6144] 125.949 137.228 134.445 0.00002 0.00002 0.00002 22 [6144, 2048] [6144, 2048] [2048, 6144] 128.172 138.566 136.816 0.00002 0.00002 0.00002 23 [6144, 2048] [6144, 2048] [2048, 6144] 129.759 139.033 138.182 0.00002 0.00002 0.00002 24 [6144, 2048] [6144, 2048] [2048, 6144] 128.384 139.729 138.658 0.00002 0.00002 0.00002 25 [6144, 2048] [6144, 2048] [2048, 6144] 126.516 140.019 137.531 0.00002 0.00002 0.00002 26 [6144, 2048] [6144, 2048] [2048, 6144] 125.452 139.144 135.120 0.00002 0.00002 0.00002 27 [6144, 2048] [6144, 2048] [2048, 6144] 132.785 137.233 125.503 0.00002 0.00002 0.00002 [13] Weight health diagnostics Sparsity > 10% none |Kurtosis Delta| > 5.0 none Outlier ratio > 1% model.layers.0.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.0.self_attn.k_norm.weight outlier_ratio=0.02344 model.layers.0.input_layernorm.weight outlier_ratio=0.01416 model.layers.2.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.2.input_layernorm.weight outlier_ratio=0.02100 model.layers.3.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.3.input_layernorm.weight outlier_ratio=0.02295 model.layers.4.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.4.input_layernorm.weight outlier_ratio=0.01855 model.layers.5.self_attn.q_norm.weight outlier_ratio=0.03125 model.layers.5.self_attn.k_norm.weight outlier_ratio=0.02344 model.layers.5.input_layernorm.weight outlier_ratio=0.02100 model.layers.6.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.6.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.6.input_layernorm.weight outlier_ratio=0.02539 model.layers.7.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.7.input_layernorm.weight outlier_ratio=0.01074 model.layers.8.input_layernorm.weight outlier_ratio=0.02002 model.layers.9.self_attn.q_norm.weight outlier_ratio=0.02344 model.layers.9.self_attn.k_norm.weight outlier_ratio=0.03125 model.layers.10.self_attn.q_norm.weight outlier_ratio=0.03125 model.layers.10.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.11.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.11.input_layernorm.weight outlier_ratio=0.01172 model.layers.12.input_layernorm.weight outlier_ratio=0.01025 model.layers.13.self_attn.q_norm.weight outlier_ratio=0.02344 model.layers.13.input_layernorm.weight outlier_ratio=0.01123 model.layers.14.self_attn.q_norm.weight outlier_ratio=0.03125 model.layers.14.input_layernorm.weight outlier_ratio=0.01074 model.layers.15.self_attn.k_norm.weight outlier_ratio=0.02344 model.layers.16.input_layernorm.weight outlier_ratio=0.01025 model.layers.16.post_attention_layernorm.weight outlier_ratio=0.01221 model.layers.17.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.17.post_attention_layernorm.weight outlier_ratio=0.01172 model.layers.18.input_layernorm.weight outlier_ratio=0.01611 model.layers.18.post_attention_layernorm.weight outlier_ratio=0.01270 model.layers.19.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.19.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.19.input_layernorm.weight outlier_ratio=0.01416 model.layers.19.post_attention_layernorm.weight outlier_ratio=0.01416 model.layers.20.self_attn.q_norm.weight outlier_ratio=0.01562 model.layers.20.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.20.input_layernorm.weight outlier_ratio=0.02295 model.layers.20.post_attention_layernorm.weight outlier_ratio=0.01221 model.layers.21.input_layernorm.weight outlier_ratio=0.02295 model.layers.22.self_attn.k_norm.weight outlier_ratio=0.02344 model.layers.22.input_layernorm.weight outlier_ratio=0.02588 model.layers.22.post_attention_layernorm.weight outlier_ratio=0.01025 model.layers.23.input_layernorm.weight outlier_ratio=0.02197 model.layers.23.post_attention_layernorm.weight outlier_ratio=0.01123 model.layers.24.self_attn.k_norm.weight outlier_ratio=0.01562 model.layers.24.input_layernorm.weight outlier_ratio=0.02197 model.layers.24.post_attention_layernorm.weight outlier_ratio=0.01367 model.layers.25.input_layernorm.weight outlier_ratio=0.03174 model.layers.25.post_attention_layernorm.weight outlier_ratio=0.01514 model.layers.26.input_layernorm.weight outlier_ratio=0.04883 model.layers.26.post_attention_layernorm.weight outlier_ratio=0.01172 model.layers.27.self_attn.q_norm.weight outlier_ratio=0.02344 model.layers.27.input_layernorm.weight outlier_ratio=0.02148 model.norm.weight outlier_ratio=0.01416 Dead rows (L2 < 1e-6) none Low cosine sim vs base (<0.95) none Low SNR vs base (< 20 dB) none Note on kurtosis delta: Kurtosis values are reported as the difference (delta) compared to the base model. A high kurtosis delta on tiny vectors (like norm/q-k-norm vectors of size 128) is statistically expected due to small sample sizes and does not indicate a model health or representation collapse issue. [14] Executive summary shared tensors : 311 tensors changed vs base : 277 / 311 cosine similarity : mean = 0.999991 | median = 0.999992 | p10 = 0.999980 | p90 = 1.000000 relative error : mean = 0.001093 | median = 0.001293 | p10 = 0.000000 | p90 = 0.002338 SNR dB : mean = 81.86 | median = 47.79 | p10 = 43.89 | p90 = 262.77 high-sparsity layers (>10%) : 0 heavy-tail layers (|kurt_d|>5.0) : 0 dead-row layers : 0 low-cos layers (<0.95) : 0 low-SNR layers (<20 dB) : 0 distillation alpha : 0.3 checkpoint size on disk : 3.218 GiB base weights in memory : 3.784 GiB distilled weights in memory : 3.784 GiB ======================================================================================== END OF REPORT ========================================================================================