1545 lines
171 KiB
Plaintext
1545 lines
171 KiB
Plaintext
========================================================================================
|
|
QUINTUS WEIGHT AUDIT
|
|
========================================================================================
|
|
<REDACTED_ON_PURPOSE> <REDACTED_ON_PURPOSE>
|
|
base model : Qwen/Qwen3-1.7B-Base
|
|
distilled model : iamrahulreddy/Quintus
|
|
alpha : 0.3
|
|
device : cuda | dtype: torch.bfloat16
|
|
python : 3.12.13 | torch: 2.11.0+cu128
|
|
========================================================================================
|
|
|
|
[01] Resolve checkpoints
|
|
Downloading 'iamrahulreddy/Quintus' from HuggingFace Hub...
|
|
Done in 0.2s
|
|
base model commit : <REDACTED_ON_PURPOSE>
|
|
distilled commit : <REDACTED_ON_PURPOSE>
|
|
snapshot root : <HF_CACHE_DIR>/snapshots
|
|
|
|
Filename Size Modified
|
|
.gitattributes 1.533 KiB 2026-06-14 09:19
|
|
README.md 5.388 KiB 2026-06-14 09:19
|
|
added_tokens.json 707 B 2026-06-14 09:19
|
|
chat_template.jinja 4.070 KiB 2026-06-14 09:19
|
|
config.json 1.328 KiB 2026-06-14 09:19
|
|
generation_config.json 143 B 2026-06-14 09:19
|
|
model.safetensors 3.205 GiB 2026-06-14 09:21
|
|
special_tokens_map.json 613 B 2026-06-14 09:19
|
|
tokenizer.json 10.893 MiB 2026-06-14 09:19
|
|
tokenizer_config.json 5.277 KiB 2026-06-14 09:19
|
|
vocab.json 2.648 MiB 2026-06-14 09:19
|
|
total 3.218 GiB
|
|
|
|
[02] Architecture configuration
|
|
Loading base config...
|
|
Loading distilled config...
|
|
|
|
-- Base
|
|
label : base (Qwen/Qwen3-1.7B-Base)
|
|
model_type : qwen3
|
|
architecture : Qwen3ForCausalLM
|
|
|
|
Vocabulary
|
|
vocab_size : 151,936
|
|
bos / eos / pad : 151643 / 151643 / None
|
|
|
|
Positional encoding
|
|
max_position_embeddings: 32,768
|
|
rope_theta : N/A
|
|
rope_scaling : None
|
|
|
|
Transformer dimensions
|
|
hidden_size : 2048
|
|
num_hidden_layers : 28
|
|
intermediate_size : 6144
|
|
|
|
Attention
|
|
num_attention_heads : 16
|
|
num_key_value_heads : 8
|
|
head_dim : 128
|
|
GQA ratio : 2:1
|
|
attention_bias : False
|
|
use_qk_norm : True
|
|
sliding_window : None
|
|
|
|
Feed-forward
|
|
hidden_act : silu
|
|
mlp_bias : False
|
|
|
|
Misc
|
|
rms_norm_eps : 1e-06
|
|
tie_word_embeddings : True
|
|
use_cache : True
|
|
torch_dtype : float32
|
|
initializer_range : 0.02
|
|
|
|
-- Distilled
|
|
label : distilled (iamrahulreddy/Quintus)
|
|
model_type : qwen3
|
|
architecture : Qwen3ForCausalLM
|
|
|
|
Vocabulary
|
|
vocab_size : 151,936
|
|
bos / eos / pad : 151643 / 151643 / None
|
|
|
|
Positional encoding
|
|
max_position_embeddings: 32,768
|
|
rope_theta : N/A
|
|
rope_scaling : None
|
|
|
|
Transformer dimensions
|
|
hidden_size : 2048
|
|
num_hidden_layers : 28
|
|
intermediate_size : 6144
|
|
|
|
Attention
|
|
num_attention_heads : 16
|
|
num_key_value_heads : 8
|
|
head_dim : 128
|
|
GQA ratio : 2:1
|
|
attention_bias : False
|
|
use_qk_norm : True
|
|
sliding_window : None
|
|
|
|
Feed-forward
|
|
hidden_act : silu
|
|
mlp_bias : False
|
|
|
|
Misc
|
|
rms_norm_eps : 1e-06
|
|
tie_word_embeddings : True
|
|
use_cache : True
|
|
torch_dtype : float32
|
|
initializer_range : 0.02
|
|
|
|
-- Config diff (ignoring: _name_or_path, transformers_version)
|
|
No differences — configs identical (expected for same-architecture KD).
|
|
|
|
[03] Parameter accounting
|
|
|
|
-- Base
|
|
base
|
|
raw (all named) : 2,031,739,904 (2.031740 B)
|
|
embedding : 311,164,928 (311.164928 M)
|
|
lm_head : 311,164,928 (311.164928 M)
|
|
tied : True
|
|
unique (deduped) : 1,720,574,976 (1.720575 B)
|
|
non-embedding : 1,409,410,048 (1.409410 B)
|
|
per layer (approx) : 50,336,000
|
|
|
|
-- Distilled
|
|
distilled
|
|
raw (all named) : 2,031,739,904 (2.031740 B)
|
|
embedding : 311,164,928 (311.164928 M)
|
|
lm_head : 311,164,928 (311.164928 M)
|
|
tied : True
|
|
unique (deduped) : 1,720,574,976 (1.720575 B)
|
|
non-embedding : 1,409,410,048 (1.409410 B)
|
|
per layer (approx) : 50,336,000
|
|
|
|
-- Delta
|
|
unique param delta : +0 (+0.0000 %)
|
|
non-embed param delta : +0
|
|
|
|
[04] Load weights
|
|
device: cuda | dtype: torch.bfloat16
|
|
Loading base model : Qwen/Qwen3-1.7B-Base
|
|
Done in 15.4s
|
|
Loading distilled : iamrahulreddy/Quintus
|
|
Done in 13.1s
|
|
base tensors : 311
|
|
distilled tensors : 311
|
|
weight tying confirmed (embed == lm_head): True
|
|
base weight memory : 3.784 GiB
|
|
distilled memory : 3.784 GiB
|
|
|
|
[05] Per-tensor weight statistics (distilled)
|
|
Layer Shape Mean Std Min Max Sparse KurtD OutlR RowL2 DeadR
|
|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
|
model.embed_tokens.weight [151936, 2048] -0.0001 0.0354 -0.3613 0.2988 0.0000 -0.00 0.0003 1.5789 0
|
|
model.layers.0.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0356 -0.6641 0.6680 0.0000 -0.00 0.0017 1.5730 0
|
|
model.layers.0.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0362 -0.8320 0.6172 0.0000 -0.00 0.0019 1.5972 0
|
|
model.layers.0.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0340 -0.2275 0.2227 0.0000 -0.00 0.0005 1.5332 0
|
|
model.layers.0.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0328 -0.4512 0.5156 0.0000 -0.00 0.0031 1.4652 0
|
|
model.layers.0.self_attn.q_norm.weight [128] 1.6407 0.8456 -0.4355 6.9688 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.0.self_attn.k_norm.weight [128] 3.0374 6.5282 0.0679 68.0000 0.0000 0.00 0.0234 nan N/A
|
|
model.layers.0.mlp.gate_proj.weight [6144, 2048] 0.0002 0.0390 -0.6484 0.6367 0.0000 -0.00 0.0012 1.7427 0
|
|
model.layers.0.mlp.up_proj.weight [6144, 2048] -0.0000 0.0318 -0.4160 0.3945 0.0000 -0.00 0.0004 1.4374 0
|
|
model.layers.0.mlp.down_proj.weight [2048, 6144] 0.0000 0.0336 -0.6367 0.5156 0.0000 -0.00 0.0007 2.6247 0
|
|
model.layers.0.input_layernorm.weight [2048] 0.0895 0.0521 0.0576 1.0078 0.0000 0.00 0.0142 nan N/A
|
|
model.layers.0.post_attention_layernorm.weight [2048] 0.1976 0.1459 -0.6406 0.8047 0.0000 0.00 0.0005 nan N/A
|
|
model.layers.1.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0349 -0.5039 0.6094 0.0000 -0.00 0.0027 1.5187 0
|
|
model.layers.1.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0341 -0.5078 0.4082 0.0000 -0.00 0.0022 1.4952 0
|
|
model.layers.1.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0350 -0.2305 0.2617 0.0000 -0.00 0.0005 1.5767 0
|
|
model.layers.1.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0329 -0.3398 0.3711 0.0000 -0.00 0.0011 1.4763 0
|
|
model.layers.1.self_attn.q_norm.weight [128] 1.4082 0.7994 -2.1875 4.5938 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.1.self_attn.k_norm.weight [128] 2.1907 2.8109 -2.9688 30.6250 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.1.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0453 -0.4043 0.5273 0.0000 -0.00 0.0007 1.9884 0
|
|
model.layers.1.mlp.up_proj.weight [6144, 2048] -0.0000 0.0289 -0.4688 0.5234 0.0000 -0.00 0.0014 1.2540 0
|
|
model.layers.1.mlp.down_proj.weight [2048, 6144] -0.0000 0.0294 -0.6289 0.5312 0.0000 -0.00 0.0014 2.2993 0
|
|
model.layers.1.input_layernorm.weight [2048] 0.0283 0.0940 -0.3125 0.5703 0.0000 0.00 0.0029 nan N/A
|
|
model.layers.1.post_attention_layernorm.weight [2048] 0.5232 0.1638 -1.1953 5.2500 0.0000 -0.00 0.0068 nan N/A
|
|
model.layers.2.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0347 -0.5156 0.6406 0.0000 -0.00 0.0029 1.5091 0
|
|
model.layers.2.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0338 -0.2910 0.3613 0.0000 -0.00 0.0022 1.4704 0
|
|
model.layers.2.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0349 -0.2119 0.2422 0.0000 -0.00 0.0007 1.5588 0
|
|
model.layers.2.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0319 -0.4121 0.4414 0.0000 -0.00 0.0018 1.4350 0
|
|
model.layers.2.self_attn.q_norm.weight [128] 1.4967 0.7808 -0.1245 4.9688 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.2.self_attn.k_norm.weight [128] 2.3197 3.2107 -0.0057 35.5000 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.2.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0469 -0.4805 0.5000 0.0000 -0.00 0.0005 2.0895 0
|
|
model.layers.2.mlp.up_proj.weight [6144, 2048] 0.0000 0.0299 -0.5859 0.7031 0.0000 -0.00 0.0007 1.3430 0
|
|
model.layers.2.mlp.down_proj.weight [2048, 6144] -0.0000 0.0309 -0.8203 0.6445 0.0000 -0.00 0.0007 2.4146 0
|
|
model.layers.2.input_layernorm.weight [2048] 0.1257 0.0469 -0.2061 0.6133 0.0000 -0.00 0.0210 nan N/A
|
|
model.layers.2.post_attention_layernorm.weight [2048] 0.5019 0.1475 -0.5039 5.5000 0.0000 0.13 0.0029 nan N/A
|
|
model.layers.3.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0352 -0.6797 0.5078 0.0000 -0.00 0.0031 1.5280 0
|
|
model.layers.3.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0345 -0.6484 0.7969 0.0000 -0.00 0.0025 1.4915 0
|
|
model.layers.3.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0371 -0.3926 0.4844 0.0000 -0.00 0.0007 1.6652 0
|
|
model.layers.3.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0338 -0.4004 0.4453 0.0000 -0.00 0.0012 1.5201 0
|
|
model.layers.3.self_attn.q_norm.weight [128] 1.8103 0.8711 0.0004 7.0000 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.3.self_attn.k_norm.weight [128] 1.7619 1.9032 -0.0018 20.2500 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.3.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0463 -0.5352 0.4258 0.0000 -0.00 0.0005 2.0712 0
|
|
model.layers.3.mlp.up_proj.weight [6144, 2048] 0.0000 0.0306 -0.4414 0.4688 0.0000 -0.00 0.0005 1.3758 0
|
|
model.layers.3.mlp.down_proj.weight [2048, 6144] 0.0000 0.0308 -0.4160 0.6641 0.0000 -0.00 0.0005 2.4052 0
|
|
model.layers.3.input_layernorm.weight [2048] 0.3191 0.1033 -0.6133 1.5625 0.0000 0.00 0.0229 nan N/A
|
|
model.layers.3.post_attention_layernorm.weight [2048] 0.5742 0.1119 -0.5938 2.0000 0.0000 0.00 0.0068 nan N/A
|
|
model.layers.4.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0346 -0.4961 0.4805 0.0000 -0.00 0.0028 1.5056 0
|
|
model.layers.4.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0334 -0.5742 0.4141 0.0000 -0.00 0.0023 1.4488 0
|
|
model.layers.4.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0364 -0.2402 0.2441 0.0000 -0.00 0.0006 1.6342 0
|
|
model.layers.4.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0331 -0.6250 0.6641 0.0000 -0.00 0.0013 1.4884 0
|
|
model.layers.4.self_attn.q_norm.weight [128] 1.0624 0.5927 0.0052 3.7500 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.4.self_attn.k_norm.weight [128] 3.1512 3.6376 -0.0025 36.5000 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.4.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0449 -0.6172 0.6328 0.0000 -0.00 0.0005 2.0082 0
|
|
model.layers.4.mlp.up_proj.weight [6144, 2048] 0.0000 0.0309 -0.4668 0.4395 0.0000 -0.00 0.0005 1.3882 0
|
|
model.layers.4.mlp.down_proj.weight [2048, 6144] -0.0000 0.0308 -0.5898 0.6328 0.0000 -0.00 0.0006 2.4082 0
|
|
model.layers.4.input_layernorm.weight [2048] 0.3279 0.0863 -0.5820 1.2969 0.0000 0.00 0.0186 nan N/A
|
|
model.layers.4.post_attention_layernorm.weight [2048] 0.6251 0.1225 -0.8164 1.6094 0.0000 -0.00 0.0068 nan N/A
|
|
model.layers.5.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0358 -0.4375 0.5195 0.0000 -0.00 0.0026 1.5625 0
|
|
model.layers.5.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0339 -0.5469 0.5781 0.0000 -0.00 0.0016 1.4858 0
|
|
model.layers.5.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0353 -0.2891 0.2656 0.0000 -0.00 0.0014 1.5872 0
|
|
model.layers.5.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0333 -0.4961 0.4395 0.0000 -0.00 0.0024 1.4987 0
|
|
model.layers.5.self_attn.q_norm.weight [128] 1.5458 1.0519 -0.0007 6.2812 0.0000 -0.00 0.0312 nan N/A
|
|
model.layers.5.self_attn.k_norm.weight [128] 2.4593 3.5832 0.0013 31.1250 0.0000 0.00 0.0234 nan N/A
|
|
model.layers.5.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0400 -0.7227 0.7539 0.0000 -0.00 0.0009 1.7845 0
|
|
model.layers.5.mlp.up_proj.weight [6144, 2048] 0.0000 0.0331 -0.6602 0.5352 0.0000 -0.00 0.0004 1.4943 0
|
|
model.layers.5.mlp.down_proj.weight [2048, 6144] -0.0000 0.0329 -0.6211 0.6758 0.0000 -0.00 0.0006 2.5677 0
|
|
model.layers.5.input_layernorm.weight [2048] 0.3916 0.0926 -0.9727 1.4219 0.0000 0.00 0.0210 nan N/A
|
|
model.layers.5.post_attention_layernorm.weight [2048] 0.5236 0.0931 -0.3320 0.9688 0.0000 -0.00 0.0029 nan N/A
|
|
model.layers.6.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0349 -0.4180 0.5859 0.0000 -0.00 0.0049 1.4738 0
|
|
model.layers.6.self_attn.k_proj.weight [1024, 2048] 0.0001 0.0331 -0.5781 0.6094 0.0000 -0.00 0.0032 1.3997 0
|
|
model.layers.6.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0360 -0.2168 0.2207 0.0000 -0.00 0.0007 1.6123 0
|
|
model.layers.6.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0320 -0.5039 0.7539 0.0000 -0.00 0.0015 1.4365 0
|
|
model.layers.6.self_attn.q_norm.weight [128] 1.7113 1.0610 -0.0048 7.0312 0.0000 -0.00 0.0156 nan N/A
|
|
model.layers.6.self_attn.k_norm.weight [128] 2.4032 2.1293 -0.0004 19.2500 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.6.mlp.gate_proj.weight [6144, 2048] -0.0001 0.0394 -0.7031 0.6484 0.0000 -0.00 0.0010 1.7528 0
|
|
model.layers.6.mlp.up_proj.weight [6144, 2048] -0.0000 0.0333 -0.4863 0.3652 0.0000 -0.00 0.0005 1.4986 0
|
|
model.layers.6.mlp.down_proj.weight [2048, 6144] -0.0000 0.0324 -0.8672 0.8359 0.0000 -0.00 0.0008 2.5322 0
|
|
model.layers.6.input_layernorm.weight [2048] 0.3556 0.1117 -0.5938 1.3750 0.0000 0.00 0.0254 nan N/A
|
|
model.layers.6.post_attention_layernorm.weight [2048] 0.5847 0.1243 -0.6328 1.2656 0.0000 0.00 0.0034 nan N/A
|
|
model.layers.7.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0338 -0.4238 0.3887 0.0000 -0.00 0.0025 1.4765 0
|
|
model.layers.7.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0319 -0.3340 0.2617 0.0000 -0.00 0.0019 1.3969 0
|
|
model.layers.7.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0350 -0.2578 0.3203 0.0000 -0.00 0.0007 1.5745 0
|
|
model.layers.7.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0322 -0.5742 0.5039 0.0000 -0.00 0.0012 1.4427 0
|
|
model.layers.7.self_attn.q_norm.weight [128] 1.5685 0.9296 0.0188 6.5312 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.7.self_attn.k_norm.weight [128] 1.8889 2.0411 -0.0031 19.2500 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.7.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0406 -0.4727 0.5078 0.0000 -0.00 0.0008 1.7994 0
|
|
model.layers.7.mlp.up_proj.weight [6144, 2048] 0.0000 0.0328 -0.5430 0.4570 0.0000 -0.00 0.0006 1.4743 0
|
|
model.layers.7.mlp.down_proj.weight [2048, 6144] 0.0000 0.0319 -0.9805 0.6133 0.0000 -0.00 0.0007 2.4845 0
|
|
model.layers.7.input_layernorm.weight [2048] 0.5572 0.1534 -0.5820 3.5469 0.0000 0.00 0.0107 nan N/A
|
|
model.layers.7.post_attention_layernorm.weight [2048] 0.6644 0.1445 -0.6914 1.5234 0.0000 -0.00 0.0029 nan N/A
|
|
model.layers.8.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0347 -0.4434 0.4473 0.0000 -0.00 0.0042 1.4826 0
|
|
model.layers.8.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0326 -0.8008 0.8750 0.0000 -0.00 0.0031 1.4049 0
|
|
model.layers.8.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0354 -0.1914 0.2285 0.0000 -0.00 0.0004 1.5972 0
|
|
model.layers.8.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0316 -0.5039 0.4746 0.0000 -0.00 0.0010 1.4136 0
|
|
model.layers.8.self_attn.q_norm.weight [128] 1.4452 0.5422 -0.0425 4.6562 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.8.self_attn.k_norm.weight [128] 1.6409 0.8953 -0.0139 7.6875 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.8.mlp.gate_proj.weight [6144, 2048] -0.0001 0.0388 -0.8125 0.8789 0.0000 -0.00 0.0010 1.7171 0
|
|
model.layers.8.mlp.up_proj.weight [6144, 2048] -0.0000 0.0332 -0.5000 0.4355 0.0000 -0.00 0.0005 1.4933 0
|
|
model.layers.8.mlp.down_proj.weight [2048, 6144] 0.0000 0.0328 -0.6406 0.5391 0.0000 -0.00 0.0008 2.5532 0
|
|
model.layers.8.input_layernorm.weight [2048] 0.6002 0.1623 0.0991 1.9766 0.0000 0.00 0.0200 nan N/A
|
|
model.layers.8.post_attention_layernorm.weight [2048] 0.6617 0.1351 -0.3613 1.3203 0.0000 0.00 0.0010 nan N/A
|
|
model.layers.9.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0351 -0.4961 0.4453 0.0000 -0.00 0.0042 1.5042 0
|
|
model.layers.9.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0324 -0.5508 0.5547 0.0000 -0.00 0.0024 1.4137 0
|
|
model.layers.9.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0347 -0.3086 0.3320 0.0000 -0.00 0.0012 1.5503 0
|
|
model.layers.9.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0327 -0.5547 0.5078 0.0000 -0.00 0.0025 1.4583 0
|
|
model.layers.9.self_attn.q_norm.weight [128] 1.6585 1.1838 -0.0251 6.7812 0.0000 0.00 0.0234 nan N/A
|
|
model.layers.9.self_attn.k_norm.weight [128] 1.9812 1.8305 -0.0009 12.3750 0.0000 -0.00 0.0312 nan N/A
|
|
model.layers.9.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0386 -0.5586 0.5000 0.0000 -0.00 0.0011 1.7093 0
|
|
model.layers.9.mlp.up_proj.weight [6144, 2048] -0.0000 0.0346 -0.4746 0.3633 0.0000 -0.00 0.0006 1.5537 0
|
|
model.layers.9.mlp.down_proj.weight [2048, 6144] -0.0000 0.0330 -0.8047 0.5430 0.0000 -0.00 0.0009 2.5660 0
|
|
model.layers.9.input_layernorm.weight [2048] 0.7933 0.2096 0.0057 3.0312 0.0000 0.00 0.0098 nan N/A
|
|
model.layers.9.post_attention_layernorm.weight [2048] 0.6711 0.1426 -0.0014 1.3750 0.0000 -0.00 0.0010 nan N/A
|
|
model.layers.10.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0346 -0.4316 0.3926 0.0000 -0.00 0.0026 1.5106 0
|
|
model.layers.10.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0332 -0.2432 0.2676 0.0000 -0.00 0.0018 1.4593 0
|
|
model.layers.10.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0345 -0.2217 0.1992 0.0000 -0.00 0.0006 1.5555 0
|
|
model.layers.10.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0319 -0.4688 0.5625 0.0000 -0.00 0.0016 1.4223 0
|
|
model.layers.10.self_attn.q_norm.weight [128] 1.7754 1.3316 0.0095 8.4375 0.0000 0.00 0.0312 nan N/A
|
|
model.layers.10.self_attn.k_norm.weight [128] 1.8918 2.0176 -0.0007 20.7500 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.10.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0381 -0.7266 0.6289 0.0000 -0.00 0.0016 1.6789 0
|
|
model.layers.10.mlp.up_proj.weight [6144, 2048] -0.0000 0.0347 -0.6133 0.6055 0.0000 -0.00 0.0008 1.5587 0
|
|
model.layers.10.mlp.down_proj.weight [2048, 6144] 0.0000 0.0328 -0.7148 0.6406 0.0000 -0.00 0.0012 2.5447 0
|
|
model.layers.10.input_layernorm.weight [2048] 0.7780 0.2307 -0.1816 3.1406 0.0000 0.00 0.0073 nan N/A
|
|
model.layers.10.post_attention_layernorm.weight [2048] 0.6835 0.1354 -0.0013 1.2891 0.0000 -0.00 0.0015 nan N/A
|
|
model.layers.11.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0367 -0.5273 0.4961 0.0000 -0.00 0.0055 1.5396 0
|
|
model.layers.11.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0335 -0.4043 0.4062 0.0000 -0.00 0.0028 1.4380 0
|
|
model.layers.11.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0328 -0.2578 0.3027 0.0000 -0.00 0.0012 1.4683 0
|
|
model.layers.11.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0303 -0.6797 0.6797 0.0000 -0.00 0.0028 1.3430 0
|
|
model.layers.11.self_attn.q_norm.weight [128] 1.7712 0.6162 -0.0052 3.1250 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.11.self_attn.k_norm.weight [128] 2.0542 1.0209 -0.0045 5.8125 0.0000 -0.00 0.0156 nan N/A
|
|
model.layers.11.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0362 -0.5703 0.5586 0.0000 -0.00 0.0013 1.6086 0
|
|
model.layers.11.mlp.up_proj.weight [6144, 2048] 0.0000 0.0355 -0.4160 0.4160 0.0000 -0.00 0.0007 1.5934 0
|
|
model.layers.11.mlp.down_proj.weight [2048, 6144] 0.0000 0.0339 -0.7695 0.8281 0.0000 -0.00 0.0010 2.6350 0
|
|
model.layers.11.input_layernorm.weight [2048] 1.0137 0.4236 0.0009 6.5000 0.0000 0.00 0.0117 nan N/A
|
|
model.layers.11.post_attention_layernorm.weight [2048] 0.7500 0.2023 -0.6133 2.4375 0.0000 0.00 0.0015 nan N/A
|
|
model.layers.12.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0351 -0.4121 0.4121 0.0000 -0.00 0.0038 1.5119 0
|
|
model.layers.12.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0326 -0.4414 0.4531 0.0000 -0.00 0.0027 1.4140 0
|
|
model.layers.12.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0337 -0.2451 0.2676 0.0000 -0.00 0.0007 1.5215 0
|
|
model.layers.12.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0303 -0.3926 0.5508 0.0000 -0.00 0.0012 1.3480 0
|
|
model.layers.12.self_attn.q_norm.weight [128] 1.7989 0.7431 -0.0101 3.8594 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.12.self_attn.k_norm.weight [128] 1.7795 1.8132 -0.0039 20.1250 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.12.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0353 -0.5117 0.4824 0.0000 -0.00 0.0013 1.5627 0
|
|
model.layers.12.mlp.up_proj.weight [6144, 2048] 0.0000 0.0354 -0.5273 0.4805 0.0000 -0.00 0.0008 1.5855 0
|
|
model.layers.12.mlp.down_proj.weight [2048, 6144] -0.0000 0.0339 -0.8164 0.6250 0.0000 -0.00 0.0010 2.6305 0
|
|
model.layers.12.input_layernorm.weight [2048] 1.2048 0.5195 -0.7461 6.4062 0.0000 -0.00 0.0103 nan N/A
|
|
model.layers.12.post_attention_layernorm.weight [2048] 0.7572 0.2071 -0.6055 1.6484 0.0000 -0.00 0.0020 nan N/A
|
|
model.layers.13.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0355 -0.4121 0.4414 0.0000 -0.00 0.0057 1.4901 0
|
|
model.layers.13.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0311 -0.2969 0.3027 0.0000 -0.00 0.0024 1.3504 0
|
|
model.layers.13.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0336 -0.2197 0.2100 0.0000 -0.00 0.0005 1.5133 0
|
|
model.layers.13.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0294 -0.4746 0.3711 0.0000 -0.00 0.0011 1.3055 0
|
|
model.layers.13.self_attn.q_norm.weight [128] 1.8504 0.9921 -0.1807 6.1875 0.0000 0.00 0.0234 nan N/A
|
|
model.layers.13.self_attn.k_norm.weight [128] 1.7791 0.8772 -0.0124 4.9062 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.13.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0341 -0.6523 0.6484 0.0000 -0.00 0.0017 1.5077 0
|
|
model.layers.13.mlp.up_proj.weight [6144, 2048] -0.0000 0.0344 -0.4180 0.5859 0.0000 -0.00 0.0008 1.5437 0
|
|
model.layers.13.mlp.down_proj.weight [2048, 6144] 0.0000 0.0330 -0.9844 0.6992 0.0000 -0.00 0.0011 2.5615 0
|
|
model.layers.13.input_layernorm.weight [2048] 1.3235 0.6034 -0.8984 8.0625 0.0000 0.00 0.0112 nan N/A
|
|
model.layers.13.post_attention_layernorm.weight [2048] 0.7902 0.2272 -0.0023 1.7266 0.0000 0.00 0.0049 nan N/A
|
|
model.layers.14.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0344 -0.4062 0.3945 0.0000 -0.00 0.0051 1.4512 0
|
|
model.layers.14.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0308 -0.4531 0.5156 0.0000 -0.00 0.0032 1.3290 0
|
|
model.layers.14.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0327 -0.1914 0.1934 0.0000 -0.00 0.0006 1.4757 0
|
|
model.layers.14.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0294 -0.5195 0.4395 0.0000 -0.00 0.0012 1.3063 0
|
|
model.layers.14.self_attn.q_norm.weight [128] 1.7915 1.0480 -0.0293 6.3750 0.0000 0.00 0.0312 nan N/A
|
|
model.layers.14.self_attn.k_norm.weight [128] 1.8330 1.2059 -0.0398 12.1875 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.14.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0326 -0.7891 0.6875 0.0000 -0.00 0.0018 1.4436 0
|
|
model.layers.14.mlp.up_proj.weight [6144, 2048] 0.0000 0.0334 -0.5586 0.4570 0.0000 -0.00 0.0009 1.4960 0
|
|
model.layers.14.mlp.down_proj.weight [2048, 6144] -0.0000 0.0328 -0.6719 0.6484 0.0000 -0.00 0.0011 2.5438 0
|
|
model.layers.14.input_layernorm.weight [2048] 1.6904 0.7990 -0.9961 10.3750 0.0000 -0.00 0.0107 nan N/A
|
|
model.layers.14.post_attention_layernorm.weight [2048] 0.8251 0.2290 -0.0019 1.9688 0.0000 -0.00 0.0054 nan N/A
|
|
model.layers.15.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0352 -0.3848 0.4277 0.0000 -0.00 0.0054 1.4810 0
|
|
model.layers.15.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0295 -0.2891 0.2969 0.0000 -0.00 0.0024 1.2812 0
|
|
model.layers.15.self_attn.v_proj.weight [1024, 2048] 0.0001 0.0326 -0.4355 0.4141 0.0000 -0.00 0.0012 1.4678 0
|
|
model.layers.15.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0301 -0.4727 0.4473 0.0000 -0.00 0.0013 1.3374 0
|
|
model.layers.15.self_attn.q_norm.weight [128] 1.4976 0.6046 -0.0320 2.9062 0.0000 -0.00 0.0000 nan N/A
|
|
model.layers.15.self_attn.k_norm.weight [128] 2.2923 2.5574 -0.0043 22.0000 0.0000 -0.00 0.0234 nan N/A
|
|
model.layers.15.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0328 -0.7383 0.6914 0.0000 -0.00 0.0019 1.4522 0
|
|
model.layers.15.mlp.up_proj.weight [6144, 2048] 0.0000 0.0341 -0.4570 0.5547 0.0000 -0.00 0.0010 1.5281 0
|
|
model.layers.15.mlp.down_proj.weight [2048, 6144] 0.0000 0.0329 -0.7461 0.7148 0.0000 -0.00 0.0011 2.5531 0
|
|
model.layers.15.input_layernorm.weight [2048] 2.3836 0.9875 -1.6250 14.4375 0.0000 0.00 0.0088 nan N/A
|
|
model.layers.15.post_attention_layernorm.weight [2048] 0.8280 0.1941 -0.0019 1.9141 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.16.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0376 -0.4414 0.5039 0.0000 -0.00 0.0076 1.5383 0
|
|
model.layers.16.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0306 -1.0312 0.8477 0.0000 -0.00 0.0041 1.2903 0
|
|
model.layers.16.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0328 -0.1973 0.2305 0.0000 -0.00 0.0008 1.4732 0
|
|
model.layers.16.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0301 -0.5000 0.4258 0.0000 -0.00 0.0016 1.3393 0
|
|
model.layers.16.self_attn.q_norm.weight [128] 1.6256 0.6673 -0.1147 3.5000 0.0000 -0.00 0.0000 nan N/A
|
|
model.layers.16.self_attn.k_norm.weight [128] 2.0342 1.1452 -0.1045 7.8750 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.16.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0326 -0.4863 0.5273 0.0000 -0.00 0.0018 1.4504 0
|
|
model.layers.16.mlp.up_proj.weight [6144, 2048] -0.0000 0.0350 -0.5742 0.5820 0.0000 -0.00 0.0011 1.5721 0
|
|
model.layers.16.mlp.down_proj.weight [2048, 6144] 0.0000 0.0332 -0.7578 0.6016 0.0000 -0.00 0.0014 2.5963 0
|
|
model.layers.16.input_layernorm.weight [2048] 2.2985 0.8804 0.0021 16.2500 0.0000 0.00 0.0103 nan N/A
|
|
model.layers.16.post_attention_layernorm.weight [2048] 0.8429 0.1426 -0.0032 2.1719 0.0000 0.00 0.0122 nan N/A
|
|
model.layers.17.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0359 -0.3926 0.3906 0.0000 -0.00 0.0052 1.5199 0
|
|
model.layers.17.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0305 -0.4199 0.4023 0.0000 -0.00 0.0020 1.3363 0
|
|
model.layers.17.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0340 -0.2480 0.2539 0.0000 -0.00 0.0012 1.5262 0
|
|
model.layers.17.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0305 -0.4102 0.3633 0.0000 -0.00 0.0020 1.3648 0
|
|
model.layers.17.self_attn.q_norm.weight [128] 1.7546 1.1692 -0.0264 9.4375 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.17.self_attn.k_norm.weight [128] 1.9915 1.5669 -0.0845 16.0000 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.17.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0325 -0.6172 0.6172 0.0000 -0.00 0.0017 1.4493 0
|
|
model.layers.17.mlp.up_proj.weight [6144, 2048] -0.0000 0.0364 -0.3984 0.3887 0.0000 -0.00 0.0010 1.6309 0
|
|
model.layers.17.mlp.down_proj.weight [2048, 6144] -0.0000 0.0343 -0.6797 0.5547 0.0000 -0.00 0.0009 2.6680 0
|
|
model.layers.17.input_layernorm.weight [2048] 4.0729 1.4421 -2.2188 33.7500 0.0000 -0.00 0.0098 nan N/A
|
|
model.layers.17.post_attention_layernorm.weight [2048] 1.0375 0.2185 -0.0027 2.7656 0.0000 -0.00 0.0117 nan N/A
|
|
model.layers.18.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0370 -0.4414 0.4707 0.0000 -0.00 0.0061 1.5392 0
|
|
model.layers.18.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0297 -0.4922 0.4395 0.0000 -0.00 0.0026 1.2818 0
|
|
model.layers.18.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0321 -0.2461 0.3145 0.0000 -0.00 0.0008 1.4507 0
|
|
model.layers.18.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0287 -0.3672 0.4844 0.0000 -0.00 0.0013 1.2860 0
|
|
model.layers.18.self_attn.q_norm.weight [128] 1.5350 0.5170 -0.0645 2.6562 0.0000 -0.00 0.0000 nan N/A
|
|
model.layers.18.self_attn.k_norm.weight [128] 1.8658 0.9406 -0.1133 6.3125 0.0000 -0.00 0.0078 nan N/A
|
|
model.layers.18.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0329 -0.5273 0.5625 0.0000 -0.00 0.0017 1.4636 0
|
|
model.layers.18.mlp.up_proj.weight [6144, 2048] 0.0000 0.0365 -0.3809 0.4062 0.0000 -0.00 0.0010 1.6349 0
|
|
model.layers.18.mlp.down_proj.weight [2048, 6144] -0.0000 0.0352 -0.6484 0.7305 0.0000 -0.00 0.0009 2.7382 0
|
|
model.layers.18.input_layernorm.weight [2048] 4.1920 1.5139 -2.2500 27.1250 0.0000 0.00 0.0161 nan N/A
|
|
model.layers.18.post_attention_layernorm.weight [2048] 1.1746 0.2241 -0.0031 3.2031 0.0000 0.00 0.0127 nan N/A
|
|
model.layers.19.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0382 -0.6016 0.4648 0.0000 -0.00 0.0079 1.5478 0
|
|
model.layers.19.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0299 -0.5156 0.4922 0.0000 -0.00 0.0025 1.2970 0
|
|
model.layers.19.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0320 -0.2812 0.3906 0.0000 -0.00 0.0021 1.4382 0
|
|
model.layers.19.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0298 -0.3457 0.4219 0.0000 -0.00 0.0014 1.3383 0
|
|
model.layers.19.self_attn.q_norm.weight [128] 1.6152 0.6862 -0.0435 3.8906 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.19.self_attn.k_norm.weight [128] 1.8694 1.0279 -0.0339 7.4062 0.0000 -0.00 0.0156 nan N/A
|
|
model.layers.19.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0334 -0.6719 0.6289 0.0000 -0.00 0.0020 1.4815 0
|
|
model.layers.19.mlp.up_proj.weight [6144, 2048] -0.0000 0.0371 -0.9062 0.6562 0.0000 -0.00 0.0011 1.6638 0
|
|
model.layers.19.mlp.down_proj.weight [2048, 6144] 0.0000 0.0363 -0.7070 0.6719 0.0000 -0.00 0.0010 2.8286 0
|
|
model.layers.19.input_layernorm.weight [2048] 5.4223 2.2384 -0.0017 70.0000 0.0000 -0.00 0.0142 nan N/A
|
|
model.layers.19.post_attention_layernorm.weight [2048] 1.2960 0.2268 -0.0038 3.7969 0.0000 -0.00 0.0142 nan N/A
|
|
model.layers.20.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0387 -0.4453 0.5703 0.0000 -0.00 0.0075 1.5911 0
|
|
model.layers.20.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0313 -0.3730 0.3574 0.0000 -0.00 0.0024 1.3673 0
|
|
model.layers.20.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0333 -0.2812 0.3594 0.0000 -0.00 0.0022 1.4952 0
|
|
model.layers.20.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0311 -0.5586 0.6758 0.0000 -0.00 0.0016 1.3934 0
|
|
model.layers.20.self_attn.q_norm.weight [128] 1.6370 0.6676 -0.0091 4.4062 0.0000 0.00 0.0156 nan N/A
|
|
model.layers.20.self_attn.k_norm.weight [128] 1.7924 1.0205 -0.0199 8.2500 0.0000 -0.00 0.0156 nan N/A
|
|
model.layers.20.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0347 -0.6328 0.4922 0.0000 -0.00 0.0017 1.5405 0
|
|
model.layers.20.mlp.up_proj.weight [6144, 2048] -0.0000 0.0381 -0.5312 0.6797 0.0000 -0.00 0.0009 1.7075 0
|
|
model.layers.20.mlp.down_proj.weight [2048, 6144] 0.0000 0.0372 -0.6250 0.8047 0.0000 -0.00 0.0008 2.8967 0
|
|
model.layers.20.input_layernorm.weight [2048] 6.3076 2.1956 0.0019 29.2500 0.0000 0.00 0.0229 nan N/A
|
|
model.layers.20.post_attention_layernorm.weight [2048] 1.4658 0.2796 -0.4805 4.4375 0.0000 0.00 0.0122 nan N/A
|
|
model.layers.21.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0388 -0.4688 0.4258 0.0000 -0.00 0.0064 1.6170 0
|
|
model.layers.21.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0320 -0.6484 0.8789 0.0000 -0.00 0.0027 1.3681 0
|
|
model.layers.21.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0360 -0.3535 0.3496 0.0000 -0.00 0.0037 1.5851 0
|
|
model.layers.21.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0327 -0.3242 0.3066 0.0000 -0.00 0.0021 1.4665 0
|
|
model.layers.21.self_attn.q_norm.weight [128] 1.4818 0.6233 -0.0381 2.7344 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.21.self_attn.k_norm.weight [128] 1.6387 1.1450 -0.0481 9.3750 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.21.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0355 -0.6133 0.6250 0.0000 -0.00 0.0011 1.5840 0
|
|
model.layers.21.mlp.up_proj.weight [6144, 2048] -0.0000 0.0387 -0.6992 0.8242 0.0000 -0.00 0.0007 1.7390 0
|
|
model.layers.21.mlp.down_proj.weight [2048, 6144] 0.0000 0.0379 -0.7812 0.8398 0.0000 -0.00 0.0006 2.9408 0
|
|
model.layers.21.input_layernorm.weight [2048] 7.8696 3.1144 0.0018 44.2500 0.0000 0.00 0.0229 nan N/A
|
|
model.layers.21.post_attention_layernorm.weight [2048] 1.6247 0.2874 -0.0024 4.4375 0.0000 -0.00 0.0098 nan N/A
|
|
model.layers.22.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0378 -0.4062 0.4102 0.0000 -0.00 0.0056 1.5827 0
|
|
model.layers.22.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0317 -0.3711 0.4043 0.0000 -0.00 0.0027 1.3675 0
|
|
model.layers.22.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0355 -0.3086 0.3047 0.0000 -0.00 0.0014 1.5926 0
|
|
model.layers.22.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0324 -0.3984 0.5234 0.0000 -0.00 0.0011 1.4517 0
|
|
model.layers.22.self_attn.q_norm.weight [128] 1.4677 0.6570 -0.0283 6.2812 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.22.self_attn.k_norm.weight [128] 1.5249 0.8786 -0.0417 5.0938 0.0000 -0.00 0.0234 nan N/A
|
|
model.layers.22.mlp.gate_proj.weight [6144, 2048] 0.0001 0.0361 -0.7266 0.7188 0.0000 -0.00 0.0008 1.6176 0
|
|
model.layers.22.mlp.up_proj.weight [6144, 2048] 0.0000 0.0391 -0.5977 0.9453 0.0000 -0.00 0.0005 1.7588 0
|
|
model.layers.22.mlp.down_proj.weight [2048, 6144] -0.0000 0.0386 -1.2031 1.1406 0.0000 -0.00 0.0005 2.9804 0
|
|
model.layers.22.input_layernorm.weight [2048] 9.1258 4.4278 0.8594 55.0000 0.0000 0.00 0.0259 nan N/A
|
|
model.layers.22.post_attention_layernorm.weight [2048] 1.8063 0.2904 -0.0029 4.8438 0.0000 0.00 0.0103 nan N/A
|
|
model.layers.23.self_attn.q_proj.weight [2048, 2048] -0.0000 0.0353 -0.4531 0.4492 0.0000 -0.00 0.0042 1.5161 0
|
|
model.layers.23.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0313 -0.3457 0.3906 0.0000 -0.00 0.0022 1.3741 0
|
|
model.layers.23.self_attn.v_proj.weight [1024, 2048] 0.0001 0.0348 -0.3438 0.2988 0.0000 -0.00 0.0015 1.5687 0
|
|
model.layers.23.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0326 -0.2969 0.4004 0.0000 -0.00 0.0009 1.4652 0
|
|
model.layers.23.self_attn.q_norm.weight [128] 1.4275 0.5005 0.0052 2.5625 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.23.self_attn.k_norm.weight [128] 2.0273 1.2693 -0.0050 9.0000 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.23.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0366 -0.7305 0.8555 0.0000 -0.00 0.0006 1.6415 0
|
|
model.layers.23.mlp.up_proj.weight [6144, 2048] -0.0000 0.0392 -0.7852 0.6836 0.0000 -0.00 0.0004 1.7657 0
|
|
model.layers.23.mlp.down_proj.weight [2048, 6144] -0.0000 0.0390 -0.9844 1.1797 0.0000 -0.00 0.0005 3.0050 0
|
|
model.layers.23.input_layernorm.weight [2048] 10.4963 5.4583 0.0018 74.5000 0.0000 0.00 0.0220 nan N/A
|
|
model.layers.23.post_attention_layernorm.weight [2048] 2.0123 0.3104 -0.0019 5.1875 0.0000 -0.00 0.0112 nan N/A
|
|
model.layers.24.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0369 -0.3828 0.4375 0.0000 -0.00 0.0057 1.5583 0
|
|
model.layers.24.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0312 -0.3535 0.3047 0.0000 -0.00 0.0030 1.3521 0
|
|
model.layers.24.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0350 -0.2539 0.3438 0.0000 -0.00 0.0010 1.5803 0
|
|
model.layers.24.self_attn.o_proj.weight [2048, 2048] 0.0000 0.0333 -0.3066 0.2812 0.0000 -0.00 0.0007 1.4859 0
|
|
model.layers.24.self_attn.q_norm.weight [128] 1.3816 0.5937 -0.0349 2.8125 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.24.self_attn.k_norm.weight [128] 2.0919 1.4716 -0.0908 8.0625 0.0000 -0.00 0.0156 nan N/A
|
|
model.layers.24.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0362 -0.8164 0.8711 0.0000 -0.00 0.0005 1.6259 0
|
|
model.layers.24.mlp.up_proj.weight [6144, 2048] 0.0000 0.0394 -0.5664 0.5742 0.0000 -0.00 0.0004 1.7753 0
|
|
model.layers.24.mlp.down_proj.weight [2048, 6144] 0.0000 0.0391 -0.8789 1.0859 0.0000 -0.00 0.0005 3.0185 0
|
|
model.layers.24.input_layernorm.weight [2048] 13.5919 9.3049 -4.0938 95.5000 0.0000 0.00 0.0220 nan N/A
|
|
model.layers.24.post_attention_layernorm.weight [2048] 2.0885 0.3191 -0.0020 5.4375 0.0000 0.00 0.0137 nan N/A
|
|
model.layers.25.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0359 -0.4219 0.5078 0.0000 -0.00 0.0050 1.5183 0
|
|
model.layers.25.self_attn.k_proj.weight [1024, 2048] 0.0000 0.0302 -0.3594 0.3223 0.0000 -0.00 0.0026 1.3119 0
|
|
model.layers.25.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0388 -0.3223 0.2988 0.0000 -0.00 0.0010 1.7480 0
|
|
model.layers.25.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0350 -0.2949 0.2988 0.0000 -0.00 0.0007 1.5653 0
|
|
model.layers.25.self_attn.q_norm.weight [128] 1.4685 0.6075 0.0018 3.1562 0.0000 -0.00 0.0000 nan N/A
|
|
model.layers.25.self_attn.k_norm.weight [128] 1.8986 1.2900 -0.0150 5.7188 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.25.mlp.gate_proj.weight [6144, 2048] -0.0000 0.0357 -0.7695 0.6953 0.0000 -0.00 0.0006 1.6021 0
|
|
model.layers.25.mlp.up_proj.weight [6144, 2048] 0.0000 0.0395 -0.7109 0.8867 0.0000 -0.00 0.0004 1.7789 0
|
|
model.layers.25.mlp.down_proj.weight [2048, 6144] -0.0000 0.0388 -0.8516 0.9336 0.0000 -0.00 0.0004 3.0089 0
|
|
model.layers.25.input_layernorm.weight [2048] 15.9484 13.3387 -4.4688 121.5000 0.0000 0.00 0.0317 nan N/A
|
|
model.layers.25.post_attention_layernorm.weight [2048] 2.2062 0.4000 -0.0029 9.3125 0.0000 -0.00 0.0151 nan N/A
|
|
model.layers.26.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0389 -0.3672 0.4121 0.0000 -0.00 0.0054 1.6131 0
|
|
model.layers.26.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0293 -0.2432 0.3145 0.0000 -0.00 0.0025 1.2736 0
|
|
model.layers.26.self_attn.v_proj.weight [1024, 2048] 0.0000 0.0377 -0.3262 0.2432 0.0000 -0.00 0.0007 1.6897 0
|
|
model.layers.26.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0348 -0.6719 0.6094 0.0000 -0.00 0.0020 1.5532 0
|
|
model.layers.26.self_attn.q_norm.weight [128] 1.2684 0.7311 -0.0376 3.5000 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.26.self_attn.k_norm.weight [128] 1.5801 1.0784 -0.0698 4.6250 0.0000 0.00 0.0000 nan N/A
|
|
model.layers.26.mlp.gate_proj.weight [6144, 2048] 0.0000 0.0354 -0.5664 0.6797 0.0000 -0.00 0.0006 1.5891 0
|
|
model.layers.26.mlp.up_proj.weight [6144, 2048] 0.0000 0.0392 -0.5547 0.6094 0.0000 -0.00 0.0004 1.7659 0
|
|
model.layers.26.mlp.down_proj.weight [2048, 6144] -0.0000 0.0381 -1.2109 0.8555 0.0000 -0.00 0.0006 2.9697 0
|
|
model.layers.26.input_layernorm.weight [2048] 17.9357 16.5038 -5.9688 136.0000 0.0000 0.00 0.0488 nan N/A
|
|
model.layers.26.post_attention_layernorm.weight [2048] 2.4546 0.7710 -0.0022 25.5000 0.0000 -0.01 0.0117 nan N/A
|
|
model.layers.27.self_attn.q_proj.weight [2048, 2048] 0.0000 0.0355 -0.4297 0.5508 0.0000 -0.00 0.0024 1.5582 0
|
|
model.layers.27.self_attn.k_proj.weight [1024, 2048] -0.0000 0.0339 -1.6250 0.9023 0.0000 -0.00 0.0010 1.5101 0
|
|
model.layers.27.self_attn.v_proj.weight [1024, 2048] -0.0000 0.0376 -0.2373 0.2852 0.0000 -0.00 0.0004 1.6919 0
|
|
model.layers.27.self_attn.o_proj.weight [2048, 2048] -0.0000 0.0351 -0.5664 0.5273 0.0000 -0.00 0.0007 1.5805 0
|
|
model.layers.27.self_attn.q_norm.weight [128] 1.5298 0.6679 -0.0913 4.3125 0.0000 0.00 0.0234 nan N/A
|
|
model.layers.27.self_attn.k_norm.weight [128] 1.7309 1.1485 -1.1172 10.1875 0.0000 0.00 0.0078 nan N/A
|
|
model.layers.27.mlp.gate_proj.weight [6144, 2048] -0.0001 0.0374 -0.7109 0.6719 0.0000 -0.00 0.0008 1.6732 0
|
|
model.layers.27.mlp.up_proj.weight [6144, 2048] -0.0000 0.0387 -0.4609 1.0156 0.0000 -0.00 0.0006 1.7331 0
|
|
model.layers.27.mlp.down_proj.weight [2048, 6144] -0.0000 0.0354 -1.0234 1.0234 0.0000 -0.00 0.0009 2.7653 0
|
|
model.layers.27.input_layernorm.weight [2048] 17.2849 4.6932 1.5312 77.5000 0.0000 0.00 0.0215 nan N/A
|
|
model.layers.27.post_attention_layernorm.weight [2048] 3.0747 3.4869 -0.0024 153.0000 0.0000 0.00 0.0020 nan N/A
|
|
model.norm.weight [2048] 2.2139 0.5169 -0.0461 12.0625 0.0000 0.00 0.0142 nan N/A
|
|
lm_head.weight [151936, 2048] -0.0001 0.0354 -0.3613 0.2988 0.0000 -0.00 0.0003 1.5789 0
|
|
|
|
[06] Layer-type aggregated statistics (distilled)
|
|
Type Count Params AvgMean AvgStd AvgSparse AvgKurtD
|
|
----------------------------------------------------------------------------------
|
|
attn_k 28 58,720,256 -0.00000 0.03208 0.00003 -0.000
|
|
attn_knorm 28 3,584 2.01822 1.85875 0.00000 0.000
|
|
attn_o 28 117,440,512 -0.00000 0.03193 0.00003 -0.000
|
|
attn_q 28 117,440,512 -0.00000 0.03597 0.00003 -0.000
|
|
attn_qnorm 28 3,584 1.57232 0.78864 0.00000 0.000
|
|
attn_v 28 58,720,256 0.00001 0.03476 0.00003 -0.000
|
|
embedding 1 311,164,928 -0.00011 0.03539 0.00001 -0.001
|
|
final_norm 1 2,048 2.21386 0.51687 0.00000 0.000
|
|
layernorm 56 114,688 2.81740 1.41555 0.00000 0.002
|
|
lm_head 1 311,164,928 -0.00011 0.03539 0.00001 -0.001
|
|
mlp_down 28 352,321,536 0.00000 0.03432 0.00002 -0.000
|
|
mlp_gate 28 352,321,536 0.00002 0.03742 0.00002 -0.000
|
|
mlp_up 28 352,321,536 -0.00000 0.03512 0.00002 -0.000
|
|
|
|
[07] Per-transformer-block breakdown (distilled)
|
|
Blk Sublayer Shape L2 AbsMn Std Sparse RowL2
|
|
-------------------------------------------------------------------------------------------------------------------
|
|
0 input_layernorm [2048] 4.685 0.08947 0.05210 0.00000 nan
|
|
0 self_attn.q_proj [2048, 2048] 72.901 0.02744 0.03559 0.00003 1.57298
|
|
0 self_attn.k_proj [1024, 2048] 52.450 0.02796 0.03622 0.00003 1.59720
|
|
0 self_attn.v_proj [1024, 2048] 49.252 0.02678 0.03401 0.00002 1.53317
|
|
0 self_attn.o_proj [2048, 2048] 67.170 0.02499 0.03280 0.00003 1.46516
|
|
0 self_attn.q_norm [128] 20.866 1.64822 0.84563 0.00000 nan
|
|
0 self_attn.k_norm [128] 81.200 3.03745 6.52823 0.00000 nan
|
|
0 post_attention_layernorm [2048] 11.116 0.24129 0.14591 0.00000 nan
|
|
0 mlp.gate_proj [6144, 2048] 138.356 0.03043 0.03902 0.00002 1.74270
|
|
0 mlp.up_proj [6144, 2048] 112.786 0.02509 0.03180 0.00002 1.43743
|
|
0 mlp.down_proj [2048, 6144] 119.234 0.02647 0.03362 0.00002 2.62472
|
|
|
|
1 input_layernorm [2048] 4.442 0.09140 0.09403 0.00000 nan
|
|
1 self_attn.q_proj [2048, 2048] 71.518 0.02654 0.03492 0.00003 1.51872
|
|
1 self_attn.k_proj [1024, 2048] 49.452 0.02613 0.03415 0.00003 1.49522
|
|
1 self_attn.v_proj [1024, 2048] 50.624 0.02751 0.03496 0.00003 1.57671
|
|
1 self_attn.o_proj [2048, 2048] 67.294 0.02560 0.03286 0.00003 1.47633
|
|
1 self_attn.q_norm [128] 18.303 1.45519 0.79937 0.00000 nan
|
|
1 self_attn.k_norm [128] 40.221 2.25925 2.81085 0.00000 nan
|
|
1 post_attention_layernorm [2048] 24.810 0.52890 0.16376 0.00000 nan
|
|
1 mlp.gate_proj [6144, 2048] 160.751 0.03478 0.04534 0.00002 1.98843
|
|
1 mlp.up_proj [6144, 2048] 102.435 0.02192 0.02889 0.00004 1.25396
|
|
1 mlp.down_proj [2048, 6144] 104.364 0.02248 0.02943 0.00003 2.29928
|
|
|
|
2 input_layernorm [2048] 6.071 0.12772 0.04693 0.00000 nan
|
|
2 self_attn.q_proj [2048, 2048] 71.066 0.02627 0.03470 0.00003 1.50912
|
|
2 self_attn.k_proj [1024, 2048] 49.005 0.02568 0.03384 0.00004 1.47036
|
|
2 self_attn.v_proj [1024, 2048] 50.551 0.02718 0.03491 0.00003 1.55884
|
|
2 self_attn.o_proj [2048, 2048] 65.435 0.02473 0.03195 0.00003 1.43501
|
|
2 self_attn.q_norm [128] 19.083 1.49875 0.78076 0.00000 nan
|
|
2 self_attn.k_norm [128] 44.698 2.31981 3.21067 0.00000 nan
|
|
2 post_attention_layernorm [2048] 23.675 0.50428 0.14753 0.00000 nan
|
|
2 mlp.gate_proj [6144, 2048] 166.168 0.03672 0.04686 0.00002 2.08946
|
|
2 mlp.up_proj [6144, 2048] 105.972 0.02350 0.02988 0.00003 1.34295
|
|
2 mlp.down_proj [2048, 6144] 109.577 0.02421 0.03090 0.00003 2.41458
|
|
|
|
3 input_layernorm [2048] 15.178 0.32260 0.10331 0.00000 nan
|
|
3 self_attn.q_proj [2048, 2048] 72.017 0.02663 0.03516 0.00003 1.52801
|
|
3 self_attn.k_proj [1024, 2048] 50.006 0.02596 0.03453 0.00002 1.49155
|
|
3 self_attn.v_proj [1024, 2048] 53.659 0.02896 0.03705 0.00003 1.66516
|
|
3 self_attn.o_proj [2048, 2048] 69.315 0.02641 0.03384 0.00002 1.52010
|
|
3 self_attn.q_norm [128] 22.712 1.81032 0.87110 0.00000 nan
|
|
3 self_attn.k_norm [128] 29.281 1.76197 1.90323 0.00000 nan
|
|
3 post_attention_layernorm [2048] 26.474 0.57677 0.11189 0.00000 nan
|
|
3 mlp.gate_proj [6144, 2048] 164.227 0.03641 0.04631 0.00002 2.07124
|
|
3 mlp.up_proj [6144, 2048] 108.473 0.02410 0.03059 0.00003 1.37582
|
|
3 mlp.down_proj [2048, 6144] 109.132 0.02419 0.03077 0.00003 2.40517
|
|
|
|
4 input_layernorm [2048] 15.345 0.33078 0.08625 0.00000 nan
|
|
4 self_attn.q_proj [2048, 2048] 70.937 0.02631 0.03464 0.00003 1.50560
|
|
4 self_attn.k_proj [1024, 2048] 48.394 0.02525 0.03342 0.00003 1.44879
|
|
4 self_attn.v_proj [1024, 2048] 52.777 0.02844 0.03644 0.00002 1.63425
|
|
4 self_attn.o_proj [2048, 2048] 67.797 0.02570 0.03310 0.00003 1.48844
|
|
4 self_attn.q_norm [128] 13.751 1.06239 0.59266 0.00000 nan
|
|
4 self_attn.k_norm [128] 54.329 3.15128 3.63764 0.00000 nan
|
|
4 post_attention_layernorm [2048] 28.826 0.62692 0.12249 0.00000 nan
|
|
4 mlp.gate_proj [6144, 2048] 159.325 0.03529 0.04493 0.00002 2.00817
|
|
4 mlp.up_proj [6144, 2048] 109.481 0.02430 0.03087 0.00003 1.38822
|
|
4 mlp.down_proj [2048, 6144] 109.280 0.02420 0.03082 0.00003 2.40825
|
|
|
|
5 input_layernorm [2048] 18.209 0.39405 0.09262 0.00000 nan
|
|
5 self_attn.q_proj [2048, 2048] 73.221 0.02723 0.03575 0.00003 1.56245
|
|
5 self_attn.k_proj [1024, 2048] 49.084 0.02599 0.03389 0.00002 1.48577
|
|
5 self_attn.v_proj [1024, 2048] 51.053 0.02729 0.03525 0.00003 1.58715
|
|
5 self_attn.o_proj [2048, 2048] 68.198 0.02546 0.03330 0.00003 1.49868
|
|
5 self_attn.q_norm [128] 21.127 1.54580 1.05185 0.00000 nan
|
|
5 self_attn.k_norm [128] 49.039 2.45930 3.58324 0.00000 nan
|
|
5 post_attention_layernorm [2048] 24.066 0.52422 0.09310 0.00000 nan
|
|
5 mlp.gate_proj [6144, 2048] 141.749 0.03130 0.03997 0.00002 1.78453
|
|
5 mlp.up_proj [6144, 2048] 117.417 0.02614 0.03311 0.00002 1.49425
|
|
5 mlp.down_proj [2048, 6144] 116.503 0.02585 0.03285 0.00003 2.56775
|
|
|
|
6 input_layernorm [2048] 16.867 0.35778 0.11165 0.00000 nan
|
|
6 self_attn.q_proj [2048, 2048] 71.503 0.02570 0.03491 0.00003 1.47382
|
|
6 self_attn.k_proj [1024, 2048] 47.997 0.02440 0.03314 0.00004 1.39967
|
|
6 self_attn.v_proj [1024, 2048] 52.187 0.02799 0.03604 0.00002 1.61226
|
|
6 self_attn.o_proj [2048, 2048] 65.590 0.02470 0.03202 0.00003 1.43655
|
|
6 self_attn.q_norm [128] 22.756 1.71137 1.06104 0.00000 nan
|
|
6 self_attn.k_norm [128] 36.264 2.40319 2.12933 0.00000 nan
|
|
6 post_attention_layernorm [2048] 27.050 0.58757 0.12428 0.00000 nan
|
|
6 mlp.gate_proj [6144, 2048] 139.719 0.03072 0.03940 0.00002 1.75281
|
|
6 mlp.up_proj [6144, 2048] 117.968 0.02621 0.03326 0.00002 1.49864
|
|
6 mlp.down_proj [2048, 6144] 115.044 0.02540 0.03244 0.00003 2.53218
|
|
|
|
7 input_layernorm [2048] 26.151 0.55813 0.15338 0.00000 nan
|
|
7 self_attn.q_proj [2048, 2048] 69.230 0.02582 0.03380 0.00003 1.47648
|
|
7 self_attn.k_proj [1024, 2048] 46.204 0.02446 0.03190 0.00003 1.39686
|
|
7 self_attn.v_proj [1024, 2048] 50.629 0.02729 0.03496 0.00002 1.57454
|
|
7 self_attn.o_proj [2048, 2048] 65.907 0.02504 0.03218 0.00003 1.44267
|
|
7 self_attn.q_norm [128] 20.607 1.56852 0.92958 0.00000 nan
|
|
7 self_attn.k_norm [128] 31.397 1.88898 2.04107 0.00000 nan
|
|
7 post_attention_layernorm [2048] 30.772 0.66610 0.14451 0.00000 nan
|
|
7 mlp.gate_proj [6144, 2048] 144.020 0.03162 0.04061 0.00002 1.79936
|
|
7 mlp.up_proj [6144, 2048] 116.378 0.02579 0.03282 0.00003 1.47432
|
|
7 mlp.down_proj [2048, 6144] 113.238 0.02495 0.03193 0.00003 2.48453
|
|
|
|
8 input_layernorm [2048] 28.138 0.60021 0.16233 0.00000 nan
|
|
8 self_attn.q_proj [2048, 2048] 70.971 0.02579 0.03465 0.00003 1.48264
|
|
8 self_attn.k_proj [1024, 2048] 47.226 0.02438 0.03261 0.00003 1.40493
|
|
8 self_attn.v_proj [1024, 2048] 51.273 0.02780 0.03540 0.00002 1.59720
|
|
8 self_attn.o_proj [2048, 2048] 64.765 0.02465 0.03162 0.00003 1.41357
|
|
8 self_attn.q_norm [128] 17.455 1.44596 0.54216 0.00000 nan
|
|
8 self_attn.k_norm [128] 21.129 1.64117 0.89530 0.00000 nan
|
|
8 post_attention_layernorm [2048] 30.563 0.66207 0.13507 0.00000 nan
|
|
8 mlp.gate_proj [6144, 2048] 137.524 0.03013 0.03878 0.00002 1.71706
|
|
8 mlp.up_proj [6144, 2048] 117.760 0.02611 0.03321 0.00002 1.49330
|
|
8 mlp.down_proj [2048, 6144] 116.430 0.02562 0.03283 0.00002 2.55323
|
|
|
|
9 input_layernorm [2048] 37.133 0.79334 0.20959 0.00000 nan
|
|
9 self_attn.q_proj [2048, 2048] 71.985 0.02616 0.03515 0.00003 1.50425
|
|
9 self_attn.k_proj [1024, 2048] 46.960 0.02450 0.03243 0.00003 1.41366
|
|
9 self_attn.v_proj [1024, 2048] 50.199 0.02671 0.03466 0.00003 1.55031
|
|
9 self_attn.o_proj [2048, 2048] 66.940 0.02478 0.03268 0.00003 1.45834
|
|
9 self_attn.q_norm [128] 23.023 1.65887 1.18381 0.00000 nan
|
|
9 self_attn.k_norm [128] 30.462 1.98125 1.83048 0.00000 nan
|
|
9 post_attention_layernorm [2048] 31.048 0.67109 0.14262 0.00000 nan
|
|
9 mlp.gate_proj [6144, 2048] 136.793 0.02997 0.03858 0.00002 1.70927
|
|
9 mlp.up_proj [6144, 2048] 122.606 0.02717 0.03457 0.00002 1.55372
|
|
9 mlp.down_proj [2048, 6144] 117.158 0.02569 0.03304 0.00003 2.56601
|
|
|
|
10 input_layernorm [2048] 36.724 0.77820 0.23073 0.00000 nan
|
|
10 self_attn.q_proj [2048, 2048] 70.798 0.02641 0.03457 0.00002 1.51064
|
|
10 self_attn.k_proj [1024, 2048] 48.128 0.02556 0.03323 0.00002 1.45925
|
|
10 self_attn.v_proj [1024, 2048] 49.962 0.02700 0.03450 0.00003 1.55550
|
|
10 self_attn.o_proj [2048, 2048] 65.289 0.02457 0.03188 0.00003 1.42228
|
|
10 self_attn.q_norm [128] 25.073 1.77536 1.33162 0.00000 nan
|
|
10 self_attn.k_norm [128] 31.226 1.89182 2.01759 0.00000 nan
|
|
10 post_attention_layernorm [2048] 31.534 0.68355 0.13537 0.00000 nan
|
|
10 mlp.gate_proj [6144, 2048] 135.160 0.02929 0.03812 0.00002 1.67895
|
|
10 mlp.up_proj [6144, 2048] 123.157 0.02714 0.03473 0.00002 1.55874
|
|
10 mlp.down_proj [2048, 6144] 116.162 0.02527 0.03276 0.00003 2.54469
|
|
|
|
11 input_layernorm [2048] 49.718 1.01369 0.42364 0.00000 nan
|
|
11 self_attn.q_proj [2048, 2048] 75.117 0.02664 0.03668 0.00002 1.53962
|
|
11 self_attn.k_proj [1024, 2048] 48.444 0.02482 0.03345 0.00003 1.43803
|
|
11 self_attn.v_proj [1024, 2048] 47.471 0.02513 0.03278 0.00003 1.46825
|
|
11 self_attn.o_proj [2048, 2048] 61.987 0.02241 0.03027 0.00003 1.34302
|
|
11 self_attn.q_norm [128] 21.207 1.77125 0.61618 0.00000 nan
|
|
11 self_attn.k_norm [128] 25.933 2.05436 1.02094 0.00000 nan
|
|
11 post_attention_layernorm [2048] 35.154 0.75061 0.20229 0.00000 nan
|
|
11 mlp.gate_proj [6144, 2048] 128.540 0.02813 0.03624 0.00002 1.60858
|
|
11 mlp.up_proj [6144, 2048] 125.940 0.02782 0.03551 0.00003 1.59340
|
|
11 mlp.down_proj [2048, 6144] 120.142 0.02632 0.03388 0.00002 2.63503
|
|
|
|
12 input_layernorm [2048] 59.373 1.20554 0.51949 0.00000 nan
|
|
12 self_attn.q_proj [2048, 2048] 71.912 0.02639 0.03511 0.00003 1.51193
|
|
12 self_attn.k_proj [1024, 2048] 47.209 0.02461 0.03260 0.00004 1.41397
|
|
12 self_attn.v_proj [1024, 2048] 48.870 0.02639 0.03374 0.00003 1.52150
|
|
12 self_attn.o_proj [2048, 2048] 62.022 0.02345 0.03028 0.00003 1.34798
|
|
12 self_attn.q_norm [128] 22.008 1.79909 0.74309 0.00000 nan
|
|
12 self_attn.k_norm [128] 28.686 1.77965 1.81323 0.00000 nan
|
|
12 post_attention_layernorm [2048] 35.525 0.75779 0.20713 0.00000 nan
|
|
12 mlp.gate_proj [6144, 2048] 125.090 0.02734 0.03527 0.00002 1.56271
|
|
12 mlp.up_proj [6144, 2048] 125.435 0.02766 0.03537 0.00002 1.58552
|
|
12 mlp.down_proj [2048, 6144] 120.075 0.02627 0.03386 0.00003 2.63047
|
|
|
|
13 input_layernorm [2048] 65.825 1.32442 0.60343 0.00000 nan
|
|
13 self_attn.q_proj [2048, 2048] 72.686 0.02604 0.03549 0.00002 1.49013
|
|
13 self_attn.k_proj [1024, 2048] 44.996 0.02354 0.03107 0.00003 1.35037
|
|
13 self_attn.v_proj [1024, 2048] 48.618 0.02629 0.03357 0.00003 1.51328
|
|
13 self_attn.o_proj [2048, 2048] 60.200 0.02273 0.02939 0.00003 1.30548
|
|
13 self_attn.q_norm [128] 23.733 1.85438 0.99210 0.00000 nan
|
|
13 self_attn.k_norm [128] 22.425 1.77937 0.87723 0.00000 nan
|
|
13 post_attention_layernorm [2048] 37.208 0.79018 0.22723 0.00000 nan
|
|
13 mlp.gate_proj [6144, 2048] 120.812 0.02631 0.03407 0.00003 1.50769
|
|
13 mlp.up_proj [6144, 2048] 122.147 0.02692 0.03444 0.00003 1.54372
|
|
13 mlp.down_proj [2048, 6144] 117.098 0.02556 0.03302 0.00003 2.56153
|
|
|
|
14 input_layernorm [2048] 84.611 1.69142 0.79895 0.00000 nan
|
|
14 self_attn.q_proj [2048, 2048] 70.351 0.02527 0.03435 0.00003 1.45125
|
|
14 self_attn.k_proj [1024, 2048] 44.671 0.02308 0.03085 0.00003 1.32900
|
|
14 self_attn.v_proj [1024, 2048] 47.417 0.02557 0.03274 0.00003 1.47567
|
|
14 self_attn.o_proj [2048, 2048] 60.245 0.02268 0.02941 0.00003 1.30631
|
|
14 self_attn.q_norm [128] 23.459 1.79195 1.04803 0.00000 nan
|
|
14 self_attn.k_norm [128] 24.794 1.83374 1.20587 0.00000 nan
|
|
14 post_attention_layernorm [2048] 38.752 0.82513 0.22901 0.00000 nan
|
|
14 mlp.gate_proj [6144, 2048] 115.589 0.02511 0.03260 0.00003 1.44356
|
|
14 mlp.up_proj [6144, 2048] 118.312 0.02602 0.03336 0.00002 1.49604
|
|
14 mlp.down_proj [2048, 6144] 116.246 0.02533 0.03278 0.00003 2.54382
|
|
|
|
15 input_layernorm [2048] 116.758 2.38523 0.98750 0.00000 nan
|
|
15 self_attn.q_proj [2048, 2048] 72.109 0.02581 0.03521 0.00003 1.48097
|
|
15 self_attn.k_proj [1024, 2048] 42.654 0.02237 0.02945 0.00003 1.28119
|
|
15 self_attn.v_proj [1024, 2048] 47.263 0.02518 0.03264 0.00003 1.46780
|
|
15 self_attn.o_proj [2048, 2048] 61.616 0.02314 0.03008 0.00003 1.33738
|
|
15 self_attn.q_norm [128] 18.262 1.49814 0.60459 0.00000 nan
|
|
15 self_attn.k_norm [128] 38.772 2.29244 2.55740 0.00000 nan
|
|
15 post_attention_layernorm [2048] 38.488 0.82803 0.19409 0.00000 nan
|
|
15 mlp.gate_proj [6144, 2048] 116.356 0.02523 0.03281 0.00003 1.45222
|
|
15 mlp.up_proj [6144, 2048] 120.975 0.02656 0.03411 0.00003 1.52807
|
|
15 mlp.down_proj [2048, 6144] 116.761 0.02546 0.03293 0.00003 2.55310
|
|
|
|
16 input_layernorm [2048] 111.385 2.29854 0.88037 0.00000 nan
|
|
16 self_attn.q_proj [2048, 2048] 77.028 0.02664 0.03761 0.00003 1.53832
|
|
16 self_attn.k_proj [1024, 2048] 44.274 0.02238 0.03057 0.00003 1.29034
|
|
16 self_attn.v_proj [1024, 2048] 47.469 0.02541 0.03278 0.00002 1.47323
|
|
16 self_attn.o_proj [2048, 2048] 61.723 0.02308 0.03014 0.00003 1.33927
|
|
16 self_attn.q_norm [128] 19.869 1.62765 0.66726 0.00000 nan
|
|
16 self_attn.k_norm [128] 26.386 2.03656 1.14519 0.00000 nan
|
|
16 post_attention_layernorm [2048] 38.689 0.84295 0.14264 0.00000 nan
|
|
16 mlp.gate_proj [6144, 2048] 115.659 0.02505 0.03262 0.00003 1.45039
|
|
16 mlp.up_proj [6144, 2048] 124.277 0.02710 0.03504 0.00002 1.57209
|
|
16 mlp.down_proj [2048, 6144] 117.831 0.02567 0.03323 0.00003 2.59632
|
|
|
|
17 input_layernorm [2048] 195.528 4.07512 1.44214 0.00000 nan
|
|
17 self_attn.q_proj [2048, 2048] 73.600 0.02634 0.03594 0.00002 1.51986
|
|
17 self_attn.k_proj [1024, 2048] 44.185 0.02334 0.03051 0.00002 1.33630
|
|
17 self_attn.v_proj [1024, 2048] 49.308 0.02630 0.03405 0.00004 1.52620
|
|
17 self_attn.o_proj [2048, 2048] 62.506 0.02339 0.03052 0.00003 1.36478
|
|
17 self_attn.q_norm [128] 23.826 1.75517 1.16920 0.00000 nan
|
|
17 self_attn.k_norm [128] 28.626 1.99286 1.56686 0.00000 nan
|
|
17 post_attention_layernorm [2048] 47.981 1.03749 0.21853 0.00000 nan
|
|
17 mlp.gate_proj [6144, 2048] 115.404 0.02511 0.03254 0.00003 1.44933
|
|
17 mlp.up_proj [6144, 2048] 129.050 0.02835 0.03639 0.00002 1.63090
|
|
17 mlp.down_proj [2048, 6144] 121.534 0.02669 0.03427 0.00003 2.66801
|
|
|
|
18 input_layernorm [2048] 201.693 4.19416 1.51388 0.00000 nan
|
|
18 self_attn.q_proj [2048, 2048] 75.780 0.02675 0.03700 0.00003 1.53918
|
|
18 self_attn.k_proj [1024, 2048] 43.076 0.02232 0.02974 0.00003 1.28183
|
|
18 self_attn.v_proj [1024, 2048] 46.493 0.02501 0.03210 0.00003 1.45073
|
|
18 self_attn.o_proj [2048, 2048] 58.817 0.02227 0.02872 0.00003 1.28603
|
|
18 self_attn.q_norm [128] 18.318 1.53600 0.51702 0.00000 nan
|
|
18 self_attn.k_norm [128] 23.621 1.86847 0.94057 0.00000 nan
|
|
18 post_attention_layernorm [2048] 54.117 1.17464 0.22414 0.00000 nan
|
|
18 mlp.gate_proj [6144, 2048] 116.670 0.02534 0.03290 0.00003 1.46360
|
|
18 mlp.up_proj [6144, 2048] 129.370 0.02840 0.03648 0.00003 1.63489
|
|
18 mlp.down_proj [2048, 6144] 124.809 0.02738 0.03519 0.00002 2.73824
|
|
|
|
19 input_layernorm [2048] 265.464 5.42234 2.23838 0.00000 nan
|
|
19 self_attn.q_proj [2048, 2048] 78.144 0.02687 0.03816 0.00003 1.54781
|
|
19 self_attn.k_proj [1024, 2048] 43.280 0.02255 0.02989 0.00003 1.29700
|
|
19 self_attn.v_proj [1024, 2048] 46.370 0.02445 0.03202 0.00002 1.43815
|
|
19 self_attn.o_proj [2048, 2048] 61.106 0.02311 0.02984 0.00003 1.33827
|
|
19 self_attn.q_norm [128] 19.842 1.61686 0.68621 0.00000 nan
|
|
19 self_attn.k_norm [128] 24.114 1.87106 1.02789 0.00000 nan
|
|
19 post_attention_layernorm [2048] 59.540 1.29598 0.22682 0.00000 nan
|
|
19 mlp.gate_proj [6144, 2048] 118.280 0.02563 0.03335 0.00003 1.48151
|
|
19 mlp.up_proj [6144, 2048] 131.654 0.02883 0.03712 0.00002 1.66379
|
|
19 mlp.down_proj [2048, 6144] 128.726 0.02814 0.03630 0.00002 2.82859
|
|
|
|
20 input_layernorm [2048] 302.241 6.30764 2.19555 0.00000 nan
|
|
20 self_attn.q_proj [2048, 2048] 79.257 0.02737 0.03870 0.00003 1.59106
|
|
20 self_attn.k_proj [1024, 2048] 45.357 0.02364 0.03132 0.00003 1.36726
|
|
20 self_attn.v_proj [1024, 2048] 48.235 0.02536 0.03331 0.00002 1.49516
|
|
20 self_attn.o_proj [2048, 2048] 63.751 0.02390 0.03113 0.00002 1.39336
|
|
20 self_attn.q_norm [128] 19.990 1.63712 0.66759 0.00000 nan
|
|
20 self_attn.k_norm [128] 23.313 1.79316 1.02053 0.00000 nan
|
|
20 post_attention_layernorm [2048] 67.529 1.46625 0.27958 0.00000 nan
|
|
20 mlp.gate_proj [6144, 2048] 122.886 0.02678 0.03465 0.00003 1.54046
|
|
20 mlp.up_proj [6144, 2048] 134.977 0.02972 0.03806 0.00002 1.70752
|
|
20 mlp.down_proj [2048, 6144] 132.033 0.02897 0.03723 0.00002 2.89669
|
|
|
|
21 input_layernorm [2048] 382.999 7.86958 3.11439 0.00000 nan
|
|
21 self_attn.q_proj [2048, 2048] 79.468 0.02798 0.03880 0.00003 1.61697
|
|
21 self_attn.k_proj [1024, 2048] 46.374 0.02374 0.03202 0.00003 1.36808
|
|
21 self_attn.v_proj [1024, 2048] 52.175 0.02694 0.03603 0.00003 1.58513
|
|
21 self_attn.o_proj [2048, 2048] 67.043 0.02514 0.03273 0.00003 1.46654
|
|
21 self_attn.q_norm [128] 18.176 1.48291 0.62333 0.00000 nan
|
|
21 self_attn.k_norm [128] 22.588 1.64089 1.14500 0.00000 nan
|
|
21 post_attention_layernorm [2048] 74.667 1.62471 0.28739 0.00000 nan
|
|
21 mlp.gate_proj [6144, 2048] 125.949 0.02768 0.03551 0.00002 1.58401
|
|
21 mlp.up_proj [6144, 2048] 137.228 0.03036 0.03870 0.00002 1.73898
|
|
21 mlp.down_proj [2048, 6144] 134.445 0.02958 0.03791 0.00002 2.94081
|
|
|
|
22 input_layernorm [2048] 459.010 9.12582 4.42780 0.00000 nan
|
|
22 self_attn.q_proj [2048, 2048] 77.429 0.02735 0.03781 0.00003 1.58268
|
|
22 self_attn.k_proj [1024, 2048] 45.910 0.02373 0.03170 0.00004 1.36746
|
|
22 self_attn.v_proj [1024, 2048] 51.373 0.02718 0.03547 0.00002 1.59256
|
|
22 self_attn.o_proj [2048, 2048] 66.308 0.02509 0.03238 0.00002 1.45173
|
|
22 self_attn.q_norm [128] 18.181 1.46818 0.65705 0.00000 nan
|
|
22 self_attn.k_norm [128] 19.892 1.52586 0.87863 0.00000 nan
|
|
22 post_attention_layernorm [2048] 82.794 1.80634 0.29039 0.00000 nan
|
|
22 mlp.gate_proj [6144, 2048] 128.172 0.02834 0.03614 0.00002 1.61757
|
|
22 mlp.up_proj [6144, 2048] 138.566 0.03079 0.03907 0.00002 1.75877
|
|
22 mlp.down_proj [2048, 6144] 136.816 0.03010 0.03858 0.00002 2.98038
|
|
|
|
23 input_layernorm [2048] 535.369 10.49632 5.45829 0.00000 nan
|
|
23 self_attn.q_proj [2048, 2048] 72.309 0.02620 0.03531 0.00003 1.51609
|
|
23 self_attn.k_proj [1024, 2048] 45.286 0.02374 0.03127 0.00004 1.37414
|
|
23 self_attn.v_proj [1024, 2048] 50.443 0.02670 0.03483 0.00003 1.56868
|
|
23 self_attn.o_proj [2048, 2048] 66.829 0.02537 0.03263 0.00002 1.46522
|
|
23 self_attn.q_norm [128] 17.107 1.42749 0.50050 0.00000 nan
|
|
23 self_attn.k_norm [128] 27.031 2.02744 1.26929 0.00000 nan
|
|
23 post_attention_layernorm [2048] 92.141 2.01226 0.31043 0.00000 nan
|
|
23 mlp.gate_proj [6144, 2048] 129.759 0.02883 0.03659 0.00002 1.64149
|
|
23 mlp.up_proj [6144, 2048] 139.033 0.03096 0.03921 0.00002 1.76572
|
|
23 mlp.down_proj [2048, 6144] 138.182 0.03038 0.03897 0.00002 3.00505
|
|
|
|
24 input_layernorm [2048] 745.373 13.59591 9.30494 0.00000 nan
|
|
24 self_attn.q_proj [2048, 2048] 75.594 0.02681 0.03691 0.00003 1.55826
|
|
24 self_attn.k_proj [1024, 2048] 45.236 0.02332 0.03124 0.00003 1.35214
|
|
24 self_attn.v_proj [1024, 2048] 50.747 0.02695 0.03504 0.00003 1.58034
|
|
24 self_attn.o_proj [2048, 2048] 68.236 0.02596 0.03332 0.00002 1.48595
|
|
24 self_attn.q_norm [128] 17.003 1.38267 0.59374 0.00000 nan
|
|
24 self_attn.k_norm [128] 28.900 2.09434 1.47164 0.00000 nan
|
|
24 post_attention_layernorm [2048] 95.613 2.08855 0.31907 0.00000 nan
|
|
24 mlp.gate_proj [6144, 2048] 128.384 0.02856 0.03620 0.00002 1.62588
|
|
24 mlp.up_proj [6144, 2048] 139.729 0.03114 0.03940 0.00002 1.77526
|
|
24 mlp.down_proj [2048, 6144] 138.658 0.03053 0.03910 0.00002 3.01847
|
|
|
|
25 input_layernorm [2048] 940.804 15.95273 13.33869 0.00000 nan
|
|
25 self_attn.q_proj [2048, 2048] 73.422 0.02611 0.03585 0.00003 1.51835
|
|
25 self_attn.k_proj [1024, 2048] 43.686 0.02266 0.03017 0.00003 1.31192
|
|
25 self_attn.v_proj [1024, 2048] 56.192 0.02986 0.03880 0.00002 1.74803
|
|
25 self_attn.o_proj [2048, 2048] 71.640 0.02723 0.03498 0.00002 1.56534
|
|
25 self_attn.q_norm [128] 17.970 1.46855 0.60751 0.00000 nan
|
|
25 self_attn.k_norm [128] 25.937 1.89888 1.29004 0.00000 nan
|
|
25 post_attention_layernorm [2048] 101.466 2.20616 0.40003 0.00000 nan
|
|
25 mlp.gate_proj [6144, 2048] 126.516 0.02814 0.03567 0.00002 1.60211
|
|
25 mlp.up_proj [6144, 2048] 140.019 0.03120 0.03948 0.00002 1.77892
|
|
25 mlp.down_proj [2048, 6144] 137.531 0.03044 0.03878 0.00002 3.00895
|
|
|
|
26 input_layernorm [2048] 1102.892 17.94151 16.50383 0.00000 nan
|
|
26 self_attn.q_proj [2048, 2048] 79.566 0.02773 0.03885 0.00003 1.61306
|
|
26 self_attn.k_proj [1024, 2048] 42.406 0.02208 0.02928 0.00004 1.27361
|
|
26 self_attn.v_proj [1024, 2048] 54.631 0.02919 0.03772 0.00002 1.68969
|
|
26 self_attn.o_proj [2048, 2048] 71.338 0.02655 0.03483 0.00002 1.55324
|
|
26 self_attn.q_norm [128] 16.547 1.26933 0.73112 0.00000 nan
|
|
26 self_attn.k_norm [128] 21.617 1.58323 1.07843 0.00000 nan
|
|
26 post_attention_layernorm [2048] 116.430 2.45460 0.77095 0.00000 nan
|
|
26 mlp.gate_proj [6144, 2048] 125.452 0.02789 0.03537 0.00002 1.58906
|
|
26 mlp.up_proj [6144, 2048] 139.144 0.03096 0.03924 0.00002 1.76593
|
|
26 mlp.down_proj [2048, 6144] 135.120 0.02999 0.03810 0.00002 2.96974
|
|
|
|
27 input_layernorm [2048] 810.531 17.28486 4.69322 0.00000 nan
|
|
27 self_attn.q_proj [2048, 2048] 72.699 0.02733 0.03550 0.00002 1.55820
|
|
27 self_attn.k_proj [1024, 2048] 49.078 0.02647 0.03389 0.00002 1.51007
|
|
27 self_attn.v_proj [1024, 2048] 54.406 0.02942 0.03757 0.00003 1.69188
|
|
27 self_attn.o_proj [2048, 2048] 71.960 0.02762 0.03514 0.00002 1.58045
|
|
27 self_attn.q_norm [128] 18.873 1.53224 0.66789 0.00000 nan
|
|
27 self_attn.k_norm [128] 23.474 1.75737 1.14853 0.00000 nan
|
|
27 post_attention_layernorm [2048] 210.356 3.07466 3.48695 0.00000 nan
|
|
27 mlp.gate_proj [6144, 2048] 132.785 0.02938 0.03744 0.00002 1.67322
|
|
27 mlp.up_proj [6144, 2048] 137.233 0.03040 0.03870 0.00002 1.73311
|
|
27 mlp.down_proj [2048, 6144] 125.503 0.02759 0.03539 0.00002 2.76534
|
|
|
|
|
|
[08] Isotropy analysis (distilled, 2D tensors only)
|
|
Sampling up to 2048 rows per layer.
|
|
Score near 0 = isotropic (healthy). Score near 1 = representation collapse.
|
|
|
|
Layer Shape Score
|
|
------------------------------------------------------------------------------------------------------
|
|
model.embed_tokens.weight [151936, 2048] 0.053880
|
|
model.layers.0.self_attn.q_proj.weight [2048, 2048] -0.000024
|
|
model.layers.0.self_attn.k_proj.weight [1024, 2048] -0.000105
|
|
model.layers.0.self_attn.v_proj.weight [1024, 2048] 0.000003
|
|
model.layers.0.self_attn.o_proj.weight [2048, 2048] 0.000003
|
|
model.layers.0.mlp.gate_proj.weight [6144, 2048] 0.004988
|
|
model.layers.0.mlp.up_proj.weight [6144, 2048] -0.000027
|
|
model.layers.0.mlp.down_proj.weight [2048, 6144] -0.000010
|
|
model.layers.1.self_attn.q_proj.weight [2048, 2048] 0.000003
|
|
model.layers.1.self_attn.k_proj.weight [1024, 2048] 0.000002
|
|
model.layers.1.self_attn.v_proj.weight [1024, 2048] -0.000032
|
|
model.layers.1.self_attn.o_proj.weight [2048, 2048] -0.000009
|
|
model.layers.1.mlp.gate_proj.weight [6144, 2048] 0.022451
|
|
model.layers.1.mlp.up_proj.weight [6144, 2048] 0.000007
|
|
model.layers.1.mlp.down_proj.weight [2048, 6144] -0.000011
|
|
model.layers.2.self_attn.q_proj.weight [2048, 2048] 0.000066
|
|
model.layers.2.self_attn.k_proj.weight [1024, 2048] -0.000001
|
|
model.layers.2.self_attn.v_proj.weight [1024, 2048] 0.000026
|
|
model.layers.2.self_attn.o_proj.weight [2048, 2048] -0.000055
|
|
model.layers.2.mlp.gate_proj.weight [6144, 2048] 0.006968
|
|
model.layers.2.mlp.up_proj.weight [6144, 2048] 0.000021
|
|
model.layers.2.mlp.down_proj.weight [2048, 6144] 0.000004
|
|
model.layers.3.self_attn.q_proj.weight [2048, 2048] 0.000014
|
|
model.layers.3.self_attn.k_proj.weight [1024, 2048] -0.000067
|
|
model.layers.3.self_attn.v_proj.weight [1024, 2048] -0.000037
|
|
model.layers.3.self_attn.o_proj.weight [2048, 2048] 0.000007
|
|
model.layers.3.mlp.gate_proj.weight [6144, 2048] 0.007741
|
|
model.layers.3.mlp.up_proj.weight [6144, 2048] 0.000003
|
|
model.layers.3.mlp.down_proj.weight [2048, 6144] 0.000014
|
|
model.layers.4.self_attn.q_proj.weight [2048, 2048] 0.000045
|
|
model.layers.4.self_attn.k_proj.weight [1024, 2048] -0.000034
|
|
model.layers.4.self_attn.v_proj.weight [1024, 2048] 0.000003
|
|
model.layers.4.self_attn.o_proj.weight [2048, 2048] -0.000006
|
|
model.layers.4.mlp.gate_proj.weight [6144, 2048] 0.011344
|
|
model.layers.4.mlp.up_proj.weight [6144, 2048] 0.000008
|
|
model.layers.4.mlp.down_proj.weight [2048, 6144] 0.000012
|
|
model.layers.5.self_attn.q_proj.weight [2048, 2048] 0.000046
|
|
model.layers.5.self_attn.k_proj.weight [1024, 2048] -0.000004
|
|
model.layers.5.self_attn.v_proj.weight [1024, 2048] 0.000016
|
|
model.layers.5.self_attn.o_proj.weight [2048, 2048] 0.000011
|
|
model.layers.5.mlp.gate_proj.weight [6144, 2048] 0.006708
|
|
model.layers.5.mlp.up_proj.weight [6144, 2048] -0.000022
|
|
model.layers.5.mlp.down_proj.weight [2048, 6144] 0.000015
|
|
model.layers.6.self_attn.q_proj.weight [2048, 2048] 0.000093
|
|
model.layers.6.self_attn.k_proj.weight [1024, 2048] -0.000075
|
|
model.layers.6.self_attn.v_proj.weight [1024, 2048] -0.000043
|
|
model.layers.6.self_attn.o_proj.weight [2048, 2048] 0.000007
|
|
model.layers.6.mlp.gate_proj.weight [6144, 2048] 0.007193
|
|
model.layers.6.mlp.up_proj.weight [6144, 2048] -0.000010
|
|
model.layers.6.mlp.down_proj.weight [2048, 6144] 0.000017
|
|
model.layers.7.self_attn.q_proj.weight [2048, 2048] 0.000059
|
|
model.layers.7.self_attn.k_proj.weight [1024, 2048] -0.000034
|
|
model.layers.7.self_attn.v_proj.weight [1024, 2048] -0.000037
|
|
model.layers.7.self_attn.o_proj.weight [2048, 2048] 0.000010
|
|
model.layers.7.mlp.gate_proj.weight [6144, 2048] 0.009189
|
|
model.layers.7.mlp.up_proj.weight [6144, 2048] 0.000011
|
|
model.layers.7.mlp.down_proj.weight [2048, 6144] -0.000001
|
|
model.layers.8.self_attn.q_proj.weight [2048, 2048] 0.000037
|
|
model.layers.8.self_attn.k_proj.weight [1024, 2048] -0.000017
|
|
model.layers.8.self_attn.v_proj.weight [1024, 2048] -0.000040
|
|
model.layers.8.self_attn.o_proj.weight [2048, 2048] 0.000012
|
|
model.layers.8.mlp.gate_proj.weight [6144, 2048] 0.006754
|
|
model.layers.8.mlp.up_proj.weight [6144, 2048] 0.000022
|
|
model.layers.8.mlp.down_proj.weight [2048, 6144] 0.000009
|
|
model.layers.9.self_attn.q_proj.weight [2048, 2048] 0.000025
|
|
model.layers.9.self_attn.k_proj.weight [1024, 2048] 0.000040
|
|
model.layers.9.self_attn.v_proj.weight [1024, 2048] 0.000018
|
|
model.layers.9.self_attn.o_proj.weight [2048, 2048] 0.000007
|
|
model.layers.9.mlp.gate_proj.weight [6144, 2048] 0.005668
|
|
model.layers.9.mlp.up_proj.weight [6144, 2048] 0.000008
|
|
model.layers.9.mlp.down_proj.weight [2048, 6144] 0.000004
|
|
model.layers.10.self_attn.q_proj.weight [2048, 2048] 0.000032
|
|
model.layers.10.self_attn.k_proj.weight [1024, 2048] -0.000022
|
|
model.layers.10.self_attn.v_proj.weight [1024, 2048] 0.000004
|
|
model.layers.10.self_attn.o_proj.weight [2048, 2048] 0.000008
|
|
model.layers.10.mlp.gate_proj.weight [6144, 2048] 0.004282
|
|
model.layers.10.mlp.up_proj.weight [6144, 2048] -0.000008
|
|
model.layers.10.mlp.down_proj.weight [2048, 6144] 0.000002
|
|
model.layers.11.self_attn.q_proj.weight [2048, 2048] 0.000071
|
|
model.layers.11.self_attn.k_proj.weight [1024, 2048] -0.000073
|
|
model.layers.11.self_attn.v_proj.weight [1024, 2048] 0.000048
|
|
model.layers.11.self_attn.o_proj.weight [2048, 2048] 0.000011
|
|
model.layers.11.mlp.gate_proj.weight [6144, 2048] 0.003255
|
|
model.layers.11.mlp.up_proj.weight [6144, 2048] -0.000026
|
|
model.layers.11.mlp.down_proj.weight [2048, 6144] -0.000023
|
|
model.layers.12.self_attn.q_proj.weight [2048, 2048] 0.000031
|
|
model.layers.12.self_attn.k_proj.weight [1024, 2048] -0.000078
|
|
model.layers.12.self_attn.v_proj.weight [1024, 2048] 0.000001
|
|
model.layers.12.self_attn.o_proj.weight [2048, 2048] -0.000010
|
|
model.layers.12.mlp.gate_proj.weight [6144, 2048] 0.002464
|
|
model.layers.12.mlp.up_proj.weight [6144, 2048] 0.000018
|
|
model.layers.12.mlp.down_proj.weight [2048, 6144] -0.000003
|
|
model.layers.13.self_attn.q_proj.weight [2048, 2048] 0.000075
|
|
model.layers.13.self_attn.k_proj.weight [1024, 2048] -0.000093
|
|
model.layers.13.self_attn.v_proj.weight [1024, 2048] 0.000004
|
|
model.layers.13.self_attn.o_proj.weight [2048, 2048] 0.000006
|
|
model.layers.13.mlp.gate_proj.weight [6144, 2048] 0.002376
|
|
model.layers.13.mlp.up_proj.weight [6144, 2048] 0.000014
|
|
model.layers.13.mlp.down_proj.weight [2048, 6144] 0.000006
|
|
model.layers.14.self_attn.q_proj.weight [2048, 2048] 0.000035
|
|
model.layers.14.self_attn.k_proj.weight [1024, 2048] -0.000052
|
|
model.layers.14.self_attn.v_proj.weight [1024, 2048] 0.000018
|
|
model.layers.14.self_attn.o_proj.weight [2048, 2048] 0.000013
|
|
model.layers.14.mlp.gate_proj.weight [6144, 2048] 0.002415
|
|
model.layers.14.mlp.up_proj.weight [6144, 2048] 0.000037
|
|
model.layers.14.mlp.down_proj.weight [2048, 6144] 0.000017
|
|
model.layers.15.self_attn.q_proj.weight [2048, 2048] 0.000125
|
|
model.layers.15.self_attn.k_proj.weight [1024, 2048] 0.000015
|
|
model.layers.15.self_attn.v_proj.weight [1024, 2048] -0.000009
|
|
model.layers.15.self_attn.o_proj.weight [2048, 2048] 0.000011
|
|
model.layers.15.mlp.gate_proj.weight [6144, 2048] 0.002226
|
|
model.layers.15.mlp.up_proj.weight [6144, 2048] -0.000013
|
|
model.layers.15.mlp.down_proj.weight [2048, 6144] -0.000011
|
|
model.layers.16.self_attn.q_proj.weight [2048, 2048] 0.000119
|
|
model.layers.16.self_attn.k_proj.weight [1024, 2048] 0.000000
|
|
model.layers.16.self_attn.v_proj.weight [1024, 2048] -0.000033
|
|
model.layers.16.self_attn.o_proj.weight [2048, 2048] 0.000011
|
|
model.layers.16.mlp.gate_proj.weight [6144, 2048] 0.002481
|
|
model.layers.16.mlp.up_proj.weight [6144, 2048] -0.000019
|
|
model.layers.16.mlp.down_proj.weight [2048, 6144] -0.000020
|
|
model.layers.17.self_attn.q_proj.weight [2048, 2048] 0.000154
|
|
model.layers.17.self_attn.k_proj.weight [1024, 2048] 0.000096
|
|
model.layers.17.self_attn.v_proj.weight [1024, 2048] -0.000025
|
|
model.layers.17.self_attn.o_proj.weight [2048, 2048] 0.000005
|
|
model.layers.17.mlp.gate_proj.weight [6144, 2048] 0.003676
|
|
model.layers.17.mlp.up_proj.weight [6144, 2048] 0.000009
|
|
model.layers.17.mlp.down_proj.weight [2048, 6144] 0.000008
|
|
model.layers.18.self_attn.q_proj.weight [2048, 2048] 0.000152
|
|
model.layers.18.self_attn.k_proj.weight [1024, 2048] -0.000039
|
|
model.layers.18.self_attn.v_proj.weight [1024, 2048] -0.000016
|
|
model.layers.18.self_attn.o_proj.weight [2048, 2048] -0.000004
|
|
model.layers.18.mlp.gate_proj.weight [6144, 2048] 0.004275
|
|
model.layers.18.mlp.up_proj.weight [6144, 2048] -0.000034
|
|
model.layers.18.mlp.down_proj.weight [2048, 6144] 0.000024
|
|
model.layers.19.self_attn.q_proj.weight [2048, 2048] 0.000093
|
|
model.layers.19.self_attn.k_proj.weight [1024, 2048] -0.000042
|
|
model.layers.19.self_attn.v_proj.weight [1024, 2048] -0.000006
|
|
model.layers.19.self_attn.o_proj.weight [2048, 2048] -0.000030
|
|
model.layers.19.mlp.gate_proj.weight [6144, 2048] 0.005028
|
|
model.layers.19.mlp.up_proj.weight [6144, 2048] 0.000002
|
|
model.layers.19.mlp.down_proj.weight [2048, 6144] -0.000000
|
|
model.layers.20.self_attn.q_proj.weight [2048, 2048] 0.000107
|
|
model.layers.20.self_attn.k_proj.weight [1024, 2048] 0.000033
|
|
model.layers.20.self_attn.v_proj.weight [1024, 2048] 0.000032
|
|
model.layers.20.self_attn.o_proj.weight [2048, 2048] 0.000024
|
|
model.layers.20.mlp.gate_proj.weight [6144, 2048] 0.006397
|
|
model.layers.20.mlp.up_proj.weight [6144, 2048] 0.000010
|
|
model.layers.20.mlp.down_proj.weight [2048, 6144] 0.000013
|
|
model.layers.21.self_attn.q_proj.weight [2048, 2048] 0.000059
|
|
model.layers.21.self_attn.k_proj.weight [1024, 2048] -0.000016
|
|
model.layers.21.self_attn.v_proj.weight [1024, 2048] 0.000006
|
|
model.layers.21.self_attn.o_proj.weight [2048, 2048] 0.000017
|
|
model.layers.21.mlp.gate_proj.weight [6144, 2048] 0.007248
|
|
model.layers.21.mlp.up_proj.weight [6144, 2048] -0.000014
|
|
model.layers.21.mlp.down_proj.weight [2048, 6144] -0.000005
|
|
model.layers.22.self_attn.q_proj.weight [2048, 2048] 0.000028
|
|
model.layers.22.self_attn.k_proj.weight [1024, 2048] -0.000029
|
|
model.layers.22.self_attn.v_proj.weight [1024, 2048] -0.000045
|
|
model.layers.22.self_attn.o_proj.weight [2048, 2048] 0.000050
|
|
model.layers.22.mlp.gate_proj.weight [6144, 2048] 0.008393
|
|
model.layers.22.mlp.up_proj.weight [6144, 2048] 0.000002
|
|
model.layers.22.mlp.down_proj.weight [2048, 6144] 0.000002
|
|
model.layers.23.self_attn.q_proj.weight [2048, 2048] 0.000058
|
|
model.layers.23.self_attn.k_proj.weight [1024, 2048] -0.000055
|
|
model.layers.23.self_attn.v_proj.weight [1024, 2048] 0.000044
|
|
model.layers.23.self_attn.o_proj.weight [2048, 2048] -0.000004
|
|
model.layers.23.mlp.gate_proj.weight [6144, 2048] 0.010053
|
|
model.layers.23.mlp.up_proj.weight [6144, 2048] -0.000030
|
|
model.layers.23.mlp.down_proj.weight [2048, 6144] 0.000003
|
|
model.layers.24.self_attn.q_proj.weight [2048, 2048] 0.000135
|
|
model.layers.24.self_attn.k_proj.weight [1024, 2048] -0.000025
|
|
model.layers.24.self_attn.v_proj.weight [1024, 2048] 0.000060
|
|
model.layers.24.self_attn.o_proj.weight [2048, 2048] 0.000004
|
|
model.layers.24.mlp.gate_proj.weight [6144, 2048] 0.008813
|
|
model.layers.24.mlp.up_proj.weight [6144, 2048] 0.000034
|
|
model.layers.24.mlp.down_proj.weight [2048, 6144] -0.000001
|
|
model.layers.25.self_attn.q_proj.weight [2048, 2048] 0.000192
|
|
model.layers.25.self_attn.k_proj.weight [1024, 2048] 0.000218
|
|
model.layers.25.self_attn.v_proj.weight [1024, 2048] -0.000024
|
|
model.layers.25.self_attn.o_proj.weight [2048, 2048] -0.000006
|
|
model.layers.25.mlp.gate_proj.weight [6144, 2048] 0.006651
|
|
model.layers.25.mlp.up_proj.weight [6144, 2048] -0.000004
|
|
model.layers.25.mlp.down_proj.weight [2048, 6144] 0.000001
|
|
model.layers.26.self_attn.q_proj.weight [2048, 2048] 0.000148
|
|
model.layers.26.self_attn.k_proj.weight [1024, 2048] -0.000014
|
|
model.layers.26.self_attn.v_proj.weight [1024, 2048] 0.000003
|
|
model.layers.26.self_attn.o_proj.weight [2048, 2048] 0.000006
|
|
model.layers.26.mlp.gate_proj.weight [6144, 2048] 0.006363
|
|
model.layers.26.mlp.up_proj.weight [6144, 2048] 0.000014
|
|
model.layers.26.mlp.down_proj.weight [2048, 6144] 0.000014
|
|
model.layers.27.self_attn.q_proj.weight [2048, 2048] 0.000033
|
|
model.layers.27.self_attn.k_proj.weight [1024, 2048] -0.000059
|
|
model.layers.27.self_attn.v_proj.weight [1024, 2048] -0.000051
|
|
model.layers.27.self_attn.o_proj.weight [2048, 2048] 0.000006
|
|
model.layers.27.mlp.gate_proj.weight [6144, 2048] 0.009203
|
|
model.layers.27.mlp.up_proj.weight [6144, 2048] -0.000008
|
|
model.layers.27.mlp.down_proj.weight [2048, 6144] 0.000004
|
|
lm_head.weight [151936, 2048] 0.053880
|
|
|
|
Global (across 198 2D layers)
|
|
mean : 0.001484
|
|
min : -0.000105
|
|
max : 0.053880
|
|
|
|
[09] Base vs distilled divergence (all shared layers)
|
|
Shared tensors: 311
|
|
|
|
Layer MaxDelta MeanDelta L2Delta CosSim RelErr SNR_dB Chg
|
|
---------------------------------------------------------------------------------------------------------------------------------------
|
|
lm_head.weight 0.00835 0.000098 8.2285 0.99991 0.00353 37.60 Y
|
|
model.embed_tokens.weight 0.00835 0.000098 8.2285 0.99991 0.00353 37.60 Y
|
|
model.layers.0.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 253.41 N
|
|
model.layers.0.mlp.down_proj.weight 0.00291 0.000030 0.4447 0.99999 0.00112 48.57 Y
|
|
model.layers.0.mlp.gate_proj.weight 0.00278 0.000026 0.4056 1.00000 0.00084 50.66 Y
|
|
model.layers.0.mlp.up_proj.weight 0.00301 0.000031 0.4568 0.99999 0.00125 47.85 Y
|
|
model.layers.0.post_attention_layernorm.weight 0.00154 0.000006 0.0033 1.00000 0.00003 70.61 Y
|
|
model.layers.0.self_attn.k_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 278.19 N
|
|
model.layers.0.self_attn.k_proj.weight 0.00240 0.000026 0.1657 1.00000 0.00094 50.01 Y
|
|
model.layers.0.self_attn.o_proj.weight 0.00251 0.000031 0.2599 0.99999 0.00125 48.25 Y
|
|
model.layers.0.self_attn.q_norm.weight 0.00018 0.000001 0.0002 1.00000 0.00000 101.13 Y
|
|
model.layers.0.self_attn.q_proj.weight 0.00270 0.000027 0.2340 0.99999 0.00099 49.87 Y
|
|
model.layers.0.self_attn.v_proj.weight 0.00266 0.000029 0.1828 0.99999 0.00110 48.61 Y
|
|
model.layers.1.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 252.95 N
|
|
model.layers.1.mlp.down_proj.weight 0.00307 0.000039 0.5086 0.99999 0.00175 46.25 Y
|
|
model.layers.1.mlp.gate_proj.weight 0.00272 0.000026 0.4024 1.00000 0.00076 52.03 Y
|
|
model.layers.1.mlp.up_proj.weight 0.00310 0.000043 0.5255 0.99999 0.00198 45.80 Y
|
|
model.layers.1.post_attention_layernorm.weight 0.00060 0.000001 0.0008 1.00000 0.00000 89.51 Y
|
|
model.layers.1.self_attn.k_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 272.09 N
|
|
model.layers.1.self_attn.k_proj.weight 0.00246 0.000030 0.1827 0.99999 0.00116 48.65 Y
|
|
model.layers.1.self_attn.o_proj.weight 0.00259 0.000031 0.2623 0.99999 0.00121 48.18 Y
|
|
model.layers.1.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.25 N
|
|
model.layers.1.self_attn.q_proj.weight 0.00296 0.000029 0.2489 0.99999 0.00111 49.17 Y
|
|
model.layers.1.self_attn.v_proj.weight 0.00258 0.000029 0.1793 0.99999 0.00104 49.01 Y
|
|
model.layers.10.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 271.30 N
|
|
model.layers.10.mlp.down_proj.weight 0.00322 0.000044 0.5900 0.99999 0.00173 45.89 Y
|
|
model.layers.10.mlp.gate_proj.weight 0.00347 0.000036 0.5295 0.99999 0.00124 48.14 Y
|
|
model.layers.10.mlp.up_proj.weight 0.00333 0.000039 0.5501 0.99999 0.00143 47.00 Y
|
|
model.layers.10.post_attention_layernorm.weight 0.00214 0.000002 0.0028 1.00000 0.00000 80.97 Y
|
|
model.layers.10.self_attn.k_norm.weight 0.00071 0.000013 0.0009 1.00000 0.00001 90.93 Y
|
|
model.layers.10.self_attn.k_proj.weight 0.00366 0.000039 0.2194 0.99999 0.00151 46.82 Y
|
|
model.layers.10.self_attn.o_proj.weight 0.00312 0.000044 0.3382 0.99999 0.00179 45.71 Y
|
|
model.layers.10.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.98 N
|
|
model.layers.10.self_attn.q_proj.weight 0.00278 0.000038 0.3137 0.99999 0.00146 47.07 Y
|
|
model.layers.10.self_attn.v_proj.weight 0.00298 0.000039 0.2219 0.99999 0.00144 47.05 Y
|
|
model.layers.11.input_layernorm.weight 0.00099 0.000000 0.0010 1.00000 0.00000 93.98 Y
|
|
model.layers.11.mlp.down_proj.weight 0.00330 0.000042 0.5863 0.99999 0.00160 46.23 Y
|
|
model.layers.11.mlp.gate_proj.weight 0.00365 0.000038 0.5469 0.99999 0.00136 47.43 Y
|
|
model.layers.11.mlp.up_proj.weight 0.00340 0.000039 0.5542 0.99999 0.00139 47.13 Y
|
|
model.layers.11.post_attention_layernorm.weight 0.00206 0.000002 0.0028 1.00000 0.00000 82.01 Y
|
|
model.layers.11.self_attn.k_norm.weight 0.00061 0.000006 0.0006 1.00000 0.00000 92.39 Y
|
|
model.layers.11.self_attn.k_proj.weight 0.00314 0.000044 0.2314 0.99999 0.00175 46.42 Y
|
|
model.layers.11.self_attn.o_proj.weight 0.00332 0.000050 0.3651 0.99998 0.00224 44.60 Y
|
|
model.layers.11.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.53 N
|
|
model.layers.11.self_attn.q_proj.weight 0.00308 0.000042 0.3231 0.99999 0.00157 47.33 Y
|
|
model.layers.11.self_attn.v_proj.weight 0.00307 0.000047 0.2479 0.99999 0.00187 45.64 Y
|
|
model.layers.12.input_layernorm.weight 0.00122 0.000001 0.0012 1.00000 0.00000 93.71 Y
|
|
model.layers.12.mlp.down_proj.weight 0.00352 0.000044 0.6074 0.99999 0.00168 45.92 Y
|
|
model.layers.12.mlp.gate_proj.weight 0.00385 0.000041 0.5746 0.99999 0.00149 46.76 Y
|
|
model.layers.12.mlp.up_proj.weight 0.00341 0.000040 0.5733 0.99999 0.00145 46.80 Y
|
|
model.layers.12.post_attention_layernorm.weight 0.00207 0.000003 0.0035 1.00000 0.00000 80.16 Y
|
|
model.layers.12.self_attn.k_norm.weight 0.00067 0.000020 0.0012 1.00000 0.00001 87.90 Y
|
|
model.layers.12.self_attn.k_proj.weight 0.00305 0.000043 0.2318 0.99999 0.00173 46.18 Y
|
|
model.layers.12.self_attn.o_proj.weight 0.00328 0.000049 0.3622 0.99998 0.00207 44.67 Y
|
|
model.layers.12.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.85 N
|
|
model.layers.12.self_attn.q_proj.weight 0.00286 0.000040 0.3211 0.99999 0.00153 47.00 Y
|
|
model.layers.12.self_attn.v_proj.weight 0.00271 0.000042 0.2334 0.99999 0.00158 46.42 Y
|
|
model.layers.13.input_layernorm.weight 0.00095 0.000000 0.0009 1.00000 0.00000 96.85 Y
|
|
model.layers.13.mlp.down_proj.weight 0.00358 0.000047 0.6369 0.99999 0.00184 45.29 Y
|
|
model.layers.13.mlp.gate_proj.weight 0.00328 0.000043 0.5999 0.99999 0.00165 46.08 Y
|
|
model.layers.13.mlp.up_proj.weight 0.00400 0.000042 0.5943 0.99999 0.00157 46.26 Y
|
|
model.layers.13.post_attention_layernorm.weight 0.00226 0.000002 0.0030 1.00000 0.00000 81.90 Y
|
|
model.layers.13.self_attn.k_norm.weight 0.00053 0.000014 0.0008 1.00000 0.00001 89.00 Y
|
|
model.layers.13.self_attn.k_proj.weight 0.00278 0.000044 0.2374 0.99999 0.00188 45.55 Y
|
|
model.layers.13.self_attn.o_proj.weight 0.00347 0.000051 0.3762 0.99998 0.00225 44.08 Y
|
|
model.layers.13.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.51 N
|
|
model.layers.13.self_attn.q_proj.weight 0.00315 0.000042 0.3262 0.99999 0.00160 46.96 Y
|
|
model.layers.13.self_attn.v_proj.weight 0.00307 0.000042 0.2380 0.99999 0.00161 46.20 Y
|
|
model.layers.14.input_layernorm.weight 0.00164 0.000001 0.0016 1.00000 0.00000 94.24 Y
|
|
model.layers.14.mlp.down_proj.weight 0.00346 0.000049 0.6642 0.99998 0.00195 44.86 Y
|
|
model.layers.14.mlp.gate_proj.weight 0.00356 0.000047 0.6357 0.99998 0.00189 45.20 Y
|
|
model.layers.14.mlp.up_proj.weight 0.00351 0.000045 0.6247 0.99999 0.00175 45.55 Y
|
|
model.layers.14.post_attention_layernorm.weight 0.00279 0.000003 0.0041 1.00000 0.00000 79.52 Y
|
|
model.layers.14.self_attn.k_norm.weight 0.00095 0.000017 0.0012 1.00000 0.00001 86.20 Y
|
|
model.layers.14.self_attn.k_proj.weight 0.00294 0.000047 0.2449 0.99998 0.00202 45.22 Y
|
|
model.layers.14.self_attn.o_proj.weight 0.00346 0.000053 0.3908 0.99998 0.00234 43.76 Y
|
|
model.layers.14.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.41 N
|
|
model.layers.14.self_attn.q_proj.weight 0.00294 0.000044 0.3352 0.99999 0.00173 46.44 Y
|
|
model.layers.14.self_attn.v_proj.weight 0.00388 0.000045 0.2499 0.99999 0.00177 45.56 Y
|
|
model.layers.15.input_layernorm.weight 0.00200 0.000001 0.0020 1.00000 0.00000 95.34 Y
|
|
model.layers.15.mlp.down_proj.weight 0.00398 0.000051 0.6876 0.99998 0.00201 44.60 Y
|
|
model.layers.15.mlp.gate_proj.weight 0.00379 0.000049 0.6628 0.99998 0.00195 44.89 Y
|
|
model.layers.15.mlp.up_proj.weight 0.00357 0.000047 0.6463 0.99999 0.00176 45.45 Y
|
|
model.layers.15.post_attention_layernorm.weight 0.00395 0.000004 0.0054 1.00000 0.00001 77.13 Y
|
|
model.layers.15.self_attn.k_norm.weight 0.00094 0.000012 0.0011 1.00000 0.00001 90.67 Y
|
|
model.layers.15.self_attn.k_proj.weight 0.00307 0.000052 0.2690 0.99998 0.00233 44.00 Y
|
|
model.layers.15.self_attn.o_proj.weight 0.00349 0.000056 0.4069 0.99998 0.00240 43.60 Y
|
|
model.layers.15.self_attn.q_norm.weight 0.00089 0.000009 0.0009 1.00000 0.00001 86.00 Y
|
|
model.layers.15.self_attn.q_proj.weight 0.00331 0.000050 0.3798 0.99999 0.00193 45.57 Y
|
|
model.layers.15.self_attn.v_proj.weight 0.00330 0.000051 0.2726 0.99998 0.00202 44.78 Y
|
|
model.layers.16.input_layernorm.weight 0.00205 0.000001 0.0020 1.00000 0.00000 94.71 Y
|
|
model.layers.16.mlp.down_proj.weight 0.00372 0.000051 0.6956 0.99998 0.00198 44.58 Y
|
|
model.layers.16.mlp.gate_proj.weight 0.00354 0.000052 0.6944 0.99998 0.00209 44.43 Y
|
|
model.layers.16.mlp.up_proj.weight 0.00415 0.000049 0.6728 0.99999 0.00180 45.33 Y
|
|
model.layers.16.post_attention_layernorm.weight 0.00370 0.000004 0.0052 1.00000 0.00001 77.43 Y
|
|
model.layers.16.self_attn.k_norm.weight 0.00192 0.000021 0.0020 1.00000 0.00001 82.55 Y
|
|
model.layers.16.self_attn.k_proj.weight 0.00351 0.000053 0.2705 0.99998 0.00236 44.28 Y
|
|
model.layers.16.self_attn.o_proj.weight 0.00360 0.000056 0.4125 0.99998 0.00244 43.50 Y
|
|
model.layers.16.self_attn.q_norm.weight 0.00018 0.000001 0.0002 1.00000 0.00000 100.71 Y
|
|
model.layers.16.self_attn.q_proj.weight 0.00307 0.000048 0.3639 0.99999 0.00178 46.51 Y
|
|
model.layers.16.self_attn.v_proj.weight 0.00346 0.000049 0.2702 0.99998 0.00194 44.89 Y
|
|
model.layers.17.input_layernorm.weight 0.00184 0.000001 0.0018 1.00000 0.00000 100.52 Y
|
|
model.layers.17.mlp.down_proj.weight 0.00383 0.000050 0.6924 0.99998 0.00186 44.89 Y
|
|
model.layers.17.mlp.gate_proj.weight 0.00432 0.000052 0.6950 0.99998 0.00207 44.41 Y
|
|
model.layers.17.mlp.up_proj.weight 0.00419 0.000046 0.6594 0.99999 0.00163 45.83 Y
|
|
model.layers.17.post_attention_layernorm.weight 0.00394 0.000007 0.0073 1.00000 0.00001 76.41 Y
|
|
model.layers.17.self_attn.k_norm.weight 0.00046 0.000007 0.0006 1.00000 0.00000 93.71 Y
|
|
model.layers.17.self_attn.k_proj.weight 0.00357 0.000053 0.2760 0.99998 0.00227 44.09 Y
|
|
model.layers.17.self_attn.o_proj.weight 0.00359 0.000059 0.4249 0.99998 0.00250 43.35 Y
|
|
model.layers.17.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 267.54 N
|
|
model.layers.17.self_attn.q_proj.weight 0.00327 0.000051 0.3870 0.99999 0.00195 45.58 Y
|
|
model.layers.17.self_attn.v_proj.weight 0.00369 0.000054 0.2911 0.99998 0.00205 44.58 Y
|
|
model.layers.18.input_layernorm.weight 0.00153 0.000001 0.0015 1.00000 0.00000 102.37 Y
|
|
model.layers.18.mlp.down_proj.weight 0.00391 0.000049 0.6975 0.99998 0.00180 45.06 Y
|
|
model.layers.18.mlp.gate_proj.weight 0.00538 0.000052 0.7063 0.99998 0.00207 44.36 Y
|
|
model.layers.18.mlp.up_proj.weight 0.00451 0.000047 0.6702 0.99999 0.00165 45.71 Y
|
|
model.layers.18.post_attention_layernorm.weight 0.00314 0.000004 0.0046 1.00000 0.00000 81.37 Y
|
|
model.layers.18.self_attn.k_norm.weight 0.00085 0.000010 0.0010 1.00000 0.00001 87.85 Y
|
|
model.layers.18.self_attn.k_proj.weight 0.00306 0.000053 0.2766 0.99998 0.00240 43.85 Y
|
|
model.layers.18.self_attn.o_proj.weight 0.00364 0.000059 0.4308 0.99997 0.00263 42.70 Y
|
|
model.layers.18.self_attn.q_norm.weight 0.00012 0.000001 0.0001 1.00000 0.00000 103.53 Y
|
|
model.layers.18.self_attn.q_proj.weight 0.00320 0.000048 0.3734 0.99999 0.00179 46.15 Y
|
|
model.layers.18.self_attn.v_proj.weight 0.00336 0.000053 0.2861 0.99998 0.00211 44.22 Y
|
|
model.layers.19.input_layernorm.weight 0.00160 0.000001 0.0016 1.00000 0.00000 104.39 Y
|
|
model.layers.19.mlp.down_proj.weight 0.00453 0.000050 0.7104 0.99998 0.00177 45.17 Y
|
|
model.layers.19.mlp.gate_proj.weight 0.00410 0.000053 0.7214 0.99998 0.00208 44.30 Y
|
|
model.layers.19.mlp.up_proj.weight 0.00522 0.000048 0.6880 0.99999 0.00165 45.64 Y
|
|
model.layers.19.post_attention_layernorm.weight 0.00383 0.000004 0.0054 1.00000 0.00000 80.83 Y
|
|
model.layers.19.self_attn.k_norm.weight 0.00066 0.000010 0.0008 1.00000 0.00001 89.56 Y
|
|
model.layers.19.self_attn.k_proj.weight 0.00372 0.000056 0.2909 0.99998 0.00248 43.45 Y
|
|
model.layers.19.self_attn.o_proj.weight 0.00391 0.000060 0.4523 0.99997 0.00262 42.61 Y
|
|
model.layers.19.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.95 N
|
|
model.layers.19.self_attn.q_proj.weight 0.00365 0.000054 0.4036 0.99999 0.00199 45.74 Y
|
|
model.layers.19.self_attn.v_proj.weight 0.00397 0.000061 0.3253 0.99998 0.00251 43.08 Y
|
|
model.layers.2.input_layernorm.weight 0.00003 0.000000 0.0000 1.00000 0.00000 105.97 Y
|
|
model.layers.2.mlp.down_proj.weight 0.00309 0.000034 0.4665 0.99999 0.00140 47.42 Y
|
|
model.layers.2.mlp.gate_proj.weight 0.00284 0.000022 0.3744 1.00000 0.00061 52.95 Y
|
|
model.layers.2.mlp.up_proj.weight 0.00284 0.000035 0.4758 0.99999 0.00148 46.96 Y
|
|
model.layers.2.post_attention_layernorm.weight 0.00164 0.000002 0.0024 1.00000 0.00000 79.71 Y
|
|
model.layers.2.self_attn.k_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 273.01 N
|
|
model.layers.2.self_attn.k_proj.weight 0.00269 0.000032 0.1865 0.99999 0.00123 48.39 Y
|
|
model.layers.2.self_attn.o_proj.weight 0.00289 0.000032 0.2681 0.99999 0.00131 47.75 Y
|
|
model.layers.2.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.61 N
|
|
model.layers.2.self_attn.q_proj.weight 0.00246 0.000030 0.2516 0.99999 0.00114 49.02 Y
|
|
model.layers.2.self_attn.v_proj.weight 0.00323 0.000029 0.1817 0.99999 0.00108 48.89 Y
|
|
model.layers.20.input_layernorm.weight 0.00168 0.000001 0.0017 1.00000 0.00000 105.09 Y
|
|
model.layers.20.mlp.down_proj.weight 0.00505 0.000047 0.6766 0.99999 0.00161 45.81 Y
|
|
model.layers.20.mlp.gate_proj.weight 0.00436 0.000050 0.6897 0.99998 0.00185 45.02 Y
|
|
model.layers.20.mlp.up_proj.weight 0.00443 0.000045 0.6601 0.99999 0.00151 46.22 Y
|
|
model.layers.20.post_attention_layernorm.weight 0.00268 0.000002 0.0033 1.00000 0.00000 86.18 Y
|
|
model.layers.20.self_attn.k_norm.weight 0.00015 0.000002 0.0002 1.00000 0.00000 103.31 Y
|
|
model.layers.20.self_attn.k_proj.weight 0.00351 0.000054 0.2880 0.99998 0.00231 43.95 Y
|
|
model.layers.20.self_attn.o_proj.weight 0.00394 0.000060 0.4510 0.99997 0.00250 43.01 Y
|
|
model.layers.20.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.02 N
|
|
model.layers.20.self_attn.q_proj.weight 0.00348 0.000052 0.3948 0.99999 0.00189 46.05 Y
|
|
model.layers.20.self_attn.v_proj.weight 0.00371 0.000060 0.3226 0.99998 0.00235 43.49 Y
|
|
model.layers.21.input_layernorm.weight 0.00166 0.000001 0.0017 1.00000 0.00000 107.25 Y
|
|
model.layers.21.mlp.down_proj.weight 0.00422 0.000045 0.6490 0.99999 0.00151 46.33 Y
|
|
model.layers.21.mlp.gate_proj.weight 0.00478 0.000045 0.6403 0.99999 0.00163 45.88 Y
|
|
model.layers.21.mlp.up_proj.weight 0.00461 0.000041 0.6157 0.99999 0.00136 46.96 Y
|
|
model.layers.21.post_attention_layernorm.weight 0.00370 0.000004 0.0047 1.00000 0.00000 84.07 Y
|
|
model.layers.21.self_attn.k_norm.weight 0.00037 0.000012 0.0006 1.00000 0.00001 90.93 Y
|
|
model.layers.21.self_attn.k_proj.weight 0.00367 0.000055 0.2872 0.99998 0.00234 44.16 Y
|
|
model.layers.21.self_attn.o_proj.weight 0.00447 0.000059 0.4621 0.99998 0.00236 43.23 Y
|
|
model.layers.21.self_attn.q_norm.weight 0.00053 0.000011 0.0008 1.00000 0.00001 87.66 Y
|
|
model.layers.21.self_attn.q_proj.weight 0.00397 0.000048 0.3821 0.99999 0.00172 46.36 Y
|
|
model.layers.21.self_attn.v_proj.weight 0.00406 0.000061 0.3313 0.99998 0.00226 43.94 Y
|
|
model.layers.22.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 293.24 N
|
|
model.layers.22.mlp.down_proj.weight 0.00450 0.000042 0.6191 0.99999 0.00141 46.89 Y
|
|
model.layers.22.mlp.gate_proj.weight 0.00479 0.000042 0.5993 0.99999 0.00148 46.60 Y
|
|
model.layers.22.mlp.up_proj.weight 0.00500 0.000039 0.5853 0.99999 0.00127 47.49 Y
|
|
model.layers.22.post_attention_layernorm.weight 0.00310 0.000004 0.0049 1.00000 0.00000 84.63 Y
|
|
model.layers.22.self_attn.k_norm.weight 0.00101 0.000026 0.0015 1.00000 0.00002 82.39 Y
|
|
model.layers.22.self_attn.k_proj.weight 0.00382 0.000058 0.3024 0.99998 0.00243 43.63 Y
|
|
model.layers.22.self_attn.o_proj.weight 0.00444 0.000066 0.4993 0.99997 0.00263 42.46 Y
|
|
model.layers.22.self_attn.q_norm.weight 0.00073 0.000012 0.0010 1.00000 0.00001 85.04 Y
|
|
model.layers.22.self_attn.q_proj.weight 0.00376 0.000055 0.4232 0.99999 0.00202 45.25 Y
|
|
model.layers.22.self_attn.v_proj.weight 0.00386 0.000064 0.3426 0.99998 0.00237 43.52 Y
|
|
model.layers.23.input_layernorm.weight 0.00183 0.000001 0.0018 1.00000 0.00000 109.30 Y
|
|
model.layers.23.mlp.down_proj.weight 0.00555 0.000042 0.6098 0.99999 0.00138 47.11 Y
|
|
model.layers.23.mlp.gate_proj.weight 0.00572 0.000040 0.5813 0.99999 0.00140 46.98 Y
|
|
model.layers.23.mlp.up_proj.weight 0.00586 0.000038 0.5674 0.99999 0.00123 47.79 Y
|
|
model.layers.23.post_attention_layernorm.weight 0.00265 0.000003 0.0038 1.00000 0.00000 87.79 Y
|
|
model.layers.23.self_attn.k_norm.weight 0.00066 0.000015 0.0009 1.00000 0.00001 89.57 Y
|
|
model.layers.23.self_attn.k_proj.weight 0.00394 0.000060 0.3171 0.99998 0.00254 43.10 Y
|
|
model.layers.23.self_attn.o_proj.weight 0.00522 0.000063 0.4985 0.99997 0.00247 42.55 Y
|
|
model.layers.23.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 264.66 N
|
|
model.layers.23.self_attn.q_proj.weight 0.00385 0.000056 0.4306 0.99998 0.00214 44.50 Y
|
|
model.layers.23.self_attn.v_proj.weight 0.00435 0.000066 0.3622 0.99997 0.00248 42.88 Y
|
|
model.layers.24.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 297.45 N
|
|
model.layers.24.mlp.down_proj.weight 0.00490 0.000043 0.6213 0.99999 0.00139 46.98 Y
|
|
model.layers.24.mlp.gate_proj.weight 0.00485 0.000041 0.5920 0.99999 0.00144 46.73 Y
|
|
model.layers.24.mlp.up_proj.weight 0.00537 0.000039 0.5803 0.99999 0.00125 47.64 Y
|
|
model.layers.24.post_attention_layernorm.weight 0.00253 0.000003 0.0038 1.00000 0.00000 87.93 Y
|
|
model.layers.24.self_attn.k_norm.weight 0.00043 0.000005 0.0004 1.00000 0.00000 96.20 Y
|
|
model.layers.24.self_attn.k_proj.weight 0.00393 0.000061 0.3163 0.99998 0.00260 43.11 Y
|
|
model.layers.24.self_attn.o_proj.weight 0.00444 0.000064 0.5073 0.99997 0.00248 42.58 Y
|
|
model.layers.24.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 264.61 N
|
|
model.layers.24.self_attn.q_proj.weight 0.00375 0.000054 0.4165 0.99998 0.00200 45.18 Y
|
|
model.layers.24.self_attn.v_proj.weight 0.00462 0.000071 0.3789 0.99997 0.00262 42.54 Y
|
|
model.layers.25.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 299.47 N
|
|
model.layers.25.mlp.down_proj.weight 0.00560 0.000043 0.6433 0.99999 0.00143 46.60 Y
|
|
model.layers.25.mlp.gate_proj.weight 0.00534 0.000044 0.6358 0.99999 0.00157 45.98 Y
|
|
model.layers.25.mlp.up_proj.weight 0.00577 0.000040 0.6062 0.99999 0.00129 47.27 Y
|
|
model.layers.25.post_attention_layernorm.weight 0.00295 0.000004 0.0044 1.00000 0.00000 87.35 Y
|
|
model.layers.25.self_attn.k_norm.weight 0.00090 0.000018 0.0013 1.00000 0.00001 86.33 Y
|
|
model.layers.25.self_attn.k_proj.weight 0.00451 0.000069 0.3509 0.99997 0.00306 41.90 Y
|
|
model.layers.25.self_attn.o_proj.weight 0.00551 0.000070 0.5722 0.99997 0.00257 41.95 Y
|
|
model.layers.25.self_attn.q_norm.weight 0.00082 0.000007 0.0008 1.00000 0.00000 86.70 Y
|
|
model.layers.25.self_attn.q_proj.weight 0.00470 0.000063 0.4760 0.99998 0.00241 43.76 Y
|
|
model.layers.25.self_attn.v_proj.weight 0.00507 0.000072 0.4023 0.99997 0.00242 42.90 Y
|
|
model.layers.26.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 300.85 N
|
|
model.layers.26.mlp.down_proj.weight 0.00528 0.000045 0.6640 0.99999 0.00150 46.17 Y
|
|
model.layers.26.mlp.gate_proj.weight 0.00610 0.000047 0.6667 0.99999 0.00168 45.49 Y
|
|
model.layers.26.mlp.up_proj.weight 0.00630 0.000042 0.6272 0.99999 0.00136 46.92 Y
|
|
model.layers.26.post_attention_layernorm.weight 0.00246 0.000003 0.0040 1.00000 0.00000 89.19 Y
|
|
model.layers.26.self_attn.k_norm.weight 0.00059 0.000009 0.0008 1.00000 0.00001 88.71 Y
|
|
model.layers.26.self_attn.k_proj.weight 0.00468 0.000078 0.3770 0.99996 0.00355 41.02 Y
|
|
model.layers.26.self_attn.o_proj.weight 0.00565 0.000078 0.5811 0.99997 0.00294 41.78 Y
|
|
model.layers.26.self_attn.q_norm.weight 0.00061 0.000012 0.0008 1.00000 0.00001 85.86 Y
|
|
model.layers.26.self_attn.q_proj.weight 0.00470 0.000069 0.4942 0.99998 0.00247 44.14 Y
|
|
model.layers.26.self_attn.v_proj.weight 0.00506 0.000076 0.4092 0.99997 0.00259 42.51 Y
|
|
model.layers.27.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 298.18 N
|
|
model.layers.27.mlp.down_proj.weight 0.00674 0.000048 0.6742 0.99999 0.00173 45.40 Y
|
|
model.layers.27.mlp.gate_proj.weight 0.00656 0.000044 0.6610 0.99999 0.00150 46.06 Y
|
|
model.layers.27.mlp.up_proj.weight 0.00679 0.000041 0.6234 0.99999 0.00137 46.86 Y
|
|
model.layers.27.post_attention_layernorm.weight 0.00257 0.000002 0.0035 1.00000 0.00000 95.48 Y
|
|
model.layers.27.self_attn.k_norm.weight 0.00079 0.000015 0.0010 1.00000 0.00001 87.28 Y
|
|
model.layers.27.self_attn.k_proj.weight 0.00404 0.000056 0.3102 0.99998 0.00212 43.99 Y
|
|
model.layers.27.self_attn.o_proj.weight 0.00525 0.000063 0.4910 0.99998 0.00229 43.32 Y
|
|
model.layers.27.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.52 N
|
|
model.layers.27.self_attn.q_proj.weight 0.00464 0.000056 0.4464 0.99998 0.00207 44.24 Y
|
|
model.layers.27.self_attn.v_proj.weight 0.00400 0.000062 0.3477 0.99998 0.00210 43.89 Y
|
|
model.layers.3.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 263.62 N
|
|
model.layers.3.mlp.down_proj.weight 0.00308 0.000035 0.4759 0.99999 0.00144 47.21 Y
|
|
model.layers.3.mlp.gate_proj.weight 0.00297 0.000023 0.3753 1.00000 0.00062 52.82 Y
|
|
model.layers.3.mlp.up_proj.weight 0.00294 0.000034 0.4731 0.99999 0.00143 47.21 Y
|
|
model.layers.3.post_attention_layernorm.weight 0.00071 0.000001 0.0010 1.00000 0.00000 88.33 Y
|
|
model.layers.3.self_attn.k_norm.weight 0.00082 0.000031 0.0016 1.00000 0.00002 85.17 Y
|
|
model.layers.3.self_attn.k_proj.weight 0.00264 0.000031 0.1822 0.99999 0.00120 48.77 Y
|
|
model.layers.3.self_attn.o_proj.weight 0.00283 0.000030 0.2614 0.99999 0.00114 48.47 Y
|
|
model.layers.3.self_attn.q_norm.weight 0.00144 0.000011 0.0014 1.00000 0.00001 83.93 Y
|
|
model.layers.3.self_attn.q_proj.weight 0.00288 0.000030 0.2514 0.99999 0.00111 49.14 Y
|
|
model.layers.3.self_attn.v_proj.weight 0.00283 0.000028 0.1784 0.99999 0.00096 49.56 Y
|
|
model.layers.4.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 263.72 N
|
|
model.layers.4.mlp.down_proj.weight 0.00302 0.000036 0.4926 0.99999 0.00148 46.92 Y
|
|
model.layers.4.mlp.gate_proj.weight 0.00314 0.000024 0.3943 1.00000 0.00068 52.13 Y
|
|
model.layers.4.mlp.up_proj.weight 0.00315 0.000035 0.4877 0.99999 0.00145 47.03 Y
|
|
model.layers.4.post_attention_layernorm.weight 0.00110 0.000001 0.0013 1.00000 0.00000 86.80 Y
|
|
model.layers.4.self_attn.k_norm.weight 0.00041 0.000004 0.0004 1.00000 0.00000 102.18 Y
|
|
model.layers.4.self_attn.k_proj.weight 0.00231 0.000034 0.1930 0.99999 0.00135 47.99 Y
|
|
model.layers.4.self_attn.o_proj.weight 0.00297 0.000033 0.2751 0.99999 0.00128 47.83 Y
|
|
model.layers.4.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 262.77 N
|
|
model.layers.4.self_attn.q_proj.weight 0.00261 0.000032 0.2630 0.99999 0.00121 48.62 Y
|
|
model.layers.4.self_attn.v_proj.weight 0.00288 0.000030 0.1861 0.99999 0.00106 49.05 Y
|
|
model.layers.5.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 265.21 N
|
|
model.layers.5.mlp.down_proj.weight 0.00332 0.000034 0.4893 0.99999 0.00131 47.54 Y
|
|
model.layers.5.mlp.gate_proj.weight 0.00289 0.000028 0.4340 1.00000 0.00089 50.28 Y
|
|
model.layers.5.mlp.up_proj.weight 0.00301 0.000033 0.4778 0.99999 0.00126 47.81 Y
|
|
model.layers.5.post_attention_layernorm.weight 0.00134 0.000001 0.0015 1.00000 0.00000 83.84 Y
|
|
model.layers.5.self_attn.k_norm.weight 0.00109 0.000009 0.0011 1.00000 0.00000 93.01 Y
|
|
model.layers.5.self_attn.k_proj.weight 0.00241 0.000033 0.1896 0.99999 0.00126 48.26 Y
|
|
model.layers.5.self_attn.o_proj.weight 0.00307 0.000035 0.2836 0.99999 0.00136 47.62 Y
|
|
model.layers.5.self_attn.q_norm.weight 0.00098 0.000008 0.0010 1.00000 0.00000 86.69 Y
|
|
model.layers.5.self_attn.q_proj.weight 0.00267 0.000032 0.2687 0.99999 0.00117 48.71 Y
|
|
model.layers.5.self_attn.v_proj.weight 0.00282 0.000033 0.1941 0.99999 0.00120 48.40 Y
|
|
model.layers.6.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 264.54 N
|
|
model.layers.6.mlp.down_proj.weight 0.00294 0.000036 0.5101 0.99999 0.00142 47.07 Y
|
|
model.layers.6.mlp.gate_proj.weight 0.00287 0.000029 0.4471 0.99999 0.00094 49.90 Y
|
|
model.layers.6.mlp.up_proj.weight 0.00273 0.000034 0.4868 0.99999 0.00129 47.69 Y
|
|
model.layers.6.post_attention_layernorm.weight 0.00194 0.000002 0.0025 1.00000 0.00000 80.77 Y
|
|
model.layers.6.self_attn.k_norm.weight 0.00061 0.000012 0.0008 1.00000 0.00000 93.05 Y
|
|
model.layers.6.self_attn.k_proj.weight 0.00246 0.000039 0.2090 0.99999 0.00160 47.22 Y
|
|
model.layers.6.self_attn.o_proj.weight 0.00284 0.000037 0.2948 0.99999 0.00149 46.95 Y
|
|
model.layers.6.self_attn.q_norm.weight 0.00023 0.000002 0.0002 1.00000 0.00000 99.95 Y
|
|
model.layers.6.self_attn.q_proj.weight 0.00280 0.000035 0.2785 0.99999 0.00135 48.19 Y
|
|
model.layers.6.self_attn.v_proj.weight 0.00285 0.000032 0.1911 0.99999 0.00114 48.73 Y
|
|
model.layers.7.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 268.35 N
|
|
model.layers.7.mlp.down_proj.weight 0.00320 0.000038 0.5276 0.99999 0.00153 46.64 Y
|
|
model.layers.7.mlp.gate_proj.weight 0.00307 0.000029 0.4485 1.00000 0.00091 50.14 Y
|
|
model.layers.7.mlp.up_proj.weight 0.00301 0.000035 0.5010 0.99999 0.00136 47.32 Y
|
|
model.layers.7.post_attention_layernorm.weight 0.00055 0.000000 0.0006 1.00000 0.00000 94.82 Y
|
|
model.layers.7.self_attn.k_norm.weight 0.00106 0.000019 0.0012 1.00000 0.00001 88.06 Y
|
|
model.layers.7.self_attn.k_proj.weight 0.00264 0.000036 0.2023 0.99999 0.00148 47.17 Y
|
|
model.layers.7.self_attn.o_proj.weight 0.00292 0.000037 0.2979 0.99999 0.00146 46.90 Y
|
|
model.layers.7.self_attn.q_norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 266.28 N
|
|
model.layers.7.self_attn.q_proj.weight 0.00261 0.000035 0.2821 0.99999 0.00134 47.80 Y
|
|
model.layers.7.self_attn.v_proj.weight 0.00237 0.000033 0.1980 0.99999 0.00123 48.16 Y
|
|
model.layers.8.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 268.99 N
|
|
model.layers.8.mlp.down_proj.weight 0.00306 0.000039 0.5414 0.99999 0.00153 46.65 Y
|
|
model.layers.8.mlp.gate_proj.weight 0.00285 0.000032 0.4766 0.99999 0.00105 49.21 Y
|
|
model.layers.8.mlp.up_proj.weight 0.00293 0.000037 0.5160 0.99999 0.00140 47.17 Y
|
|
model.layers.8.post_attention_layernorm.weight 0.00121 0.000002 0.0019 1.00000 0.00000 84.34 Y
|
|
model.layers.8.self_attn.k_norm.weight 0.00036 0.000013 0.0006 1.00000 0.00001 90.26 Y
|
|
model.layers.8.self_attn.k_proj.weight 0.00277 0.000036 0.2029 0.99999 0.00148 47.34 Y
|
|
model.layers.8.self_attn.o_proj.weight 0.00294 0.000038 0.3006 0.99999 0.00153 46.67 Y
|
|
model.layers.8.self_attn.q_norm.weight 0.00035 0.000003 0.0004 1.00000 0.00000 93.93 Y
|
|
model.layers.8.self_attn.q_proj.weight 0.00281 0.000034 0.2798 0.99999 0.00133 48.09 Y
|
|
model.layers.8.self_attn.v_proj.weight 0.00269 0.000033 0.1968 0.99999 0.00117 48.32 Y
|
|
model.layers.9.input_layernorm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 271.40 N
|
|
model.layers.9.mlp.down_proj.weight 0.00362 0.000041 0.5626 0.99999 0.00159 46.37 Y
|
|
model.layers.9.mlp.gate_proj.weight 0.00291 0.000033 0.4943 0.99999 0.00110 48.84 Y
|
|
model.layers.9.mlp.up_proj.weight 0.00317 0.000037 0.5275 0.99999 0.00135 47.33 Y
|
|
model.layers.9.post_attention_layernorm.weight 0.00178 0.000002 0.0025 1.00000 0.00000 82.01 Y
|
|
model.layers.9.self_attn.k_norm.weight 0.00100 0.000015 0.0011 1.00000 0.00001 88.79 Y
|
|
model.layers.9.self_attn.k_proj.weight 0.00293 0.000038 0.2147 0.99999 0.00157 46.80 Y
|
|
model.layers.9.self_attn.o_proj.weight 0.00284 0.000042 0.3233 0.99999 0.00169 46.32 Y
|
|
model.layers.9.self_attn.q_norm.weight 0.00060 0.000005 0.0006 1.00000 0.00000 91.63 Y
|
|
model.layers.9.self_attn.q_proj.weight 0.00291 0.000037 0.2997 0.99999 0.00142 47.61 Y
|
|
model.layers.9.self_attn.v_proj.weight 0.00263 0.000038 0.2159 0.99999 0.00144 47.33 Y
|
|
model.norm.weight 0.00000 0.000000 0.0000 1.00000 0.00000 280.25 N
|
|
|
|
Changed : 277 / 311
|
|
Unchanged: 34 / 311
|
|
Unchanged (first 10): ['model.layers.0.input_layernorm.weight', 'model.layers.0.self_attn.k_norm.weight', 'model.layers.1.input_layernorm.weight', 'model.layers.1.self_attn.k_norm.weight', 'model.layers.1.self_attn.q_norm.weight', 'model.layers.10.input_layernorm.weight', 'model.layers.10.self_attn.q_norm.weight', 'model.layers.11.self_attn.q_norm.weight', 'model.layers.12.self_attn.q_norm.weight', 'model.layers.13.self_attn.q_norm.weight']
|
|
|
|
Note: Unchanged tensors are primarily normalization layers (input_layernorm, q_norm, k_norm, model.norm).
|
|
This demonstrates that the SFT/KD process modified the primary semantic projection weights
|
|
(attention and MLP projections) while preserving basic layer scaling characteristics.
|
|
|
|
[10] Cosine similarity distribution histogram
|
|
Range Count Histogram
|
|
[ -inf, 0.90000) 0
|
|
[ 0.90000, 0.99000) 0
|
|
[ 0.99000, 0.99900) 0
|
|
[ 0.99900, 0.99990) 0
|
|
[ 0.99990, 0.99999) 127 ############################
|
|
[ 0.99999, 1.00001) 184 ########################################
|
|
[ 1.00001, 1.00100) 0
|
|
[ 1.00100, inf) 0
|
|
|
|
[11] Attention geometry per transformer block
|
|
Query heads: 16 | KV heads: 8 | head_dim: 128 | GQA: 2:1
|
|
|
|
Blk Q shape K shape V shape O shape Q L2 K L2 V L2 O L2
|
|
----------------------------------------------------------------------------------------------------------------------------------
|
|
0 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.901 52.450 49.252 67.170
|
|
1 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.518 49.452 50.624 67.294
|
|
2 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.066 49.005 50.551 65.435
|
|
3 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.017 50.006 53.659 69.315
|
|
4 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.937 48.394 52.777 67.797
|
|
5 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 73.221 49.084 51.053 68.198
|
|
6 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.503 47.997 52.187 65.590
|
|
7 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 69.230 46.204 50.629 65.907
|
|
8 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.971 47.226 51.273 64.765
|
|
9 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.985 46.960 50.199 66.940
|
|
10 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.798 48.128 49.962 65.289
|
|
11 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 75.117 48.444 47.471 61.987
|
|
12 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 71.912 47.209 48.870 62.022
|
|
13 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.686 44.996 48.618 60.200
|
|
14 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 70.351 44.671 47.417 60.245
|
|
15 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.109 42.654 47.263 61.616
|
|
16 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 77.028 44.274 47.469 61.723
|
|
17 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 73.600 44.185 49.308 62.506
|
|
18 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 75.780 43.076 46.493 58.817
|
|
19 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 78.144 43.280 46.370 61.106
|
|
20 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 79.257 45.357 48.235 63.751
|
|
21 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 79.468 46.374 52.175 67.043
|
|
22 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 77.429 45.910 51.373 66.308
|
|
23 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.309 45.286 50.443 66.829
|
|
24 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 75.594 45.236 50.747 68.236
|
|
25 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 73.422 43.686 56.192 71.640
|
|
26 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 79.566 42.406 54.631 71.338
|
|
27 [2048, 2048] [1024, 2048] [1024, 2048] [2048, 2048] 72.699 49.078 54.406 71.960
|
|
|
|
[12] MLP feed-forward geometry per transformer block
|
|
intermediate_size: 6144 | activation: silu
|
|
|
|
Blk Gate shape Up shape Down shape Gate L2 Up L2 Down L2 GateSp UpSp DnSp
|
|
---------------------------------------------------------------------------------------------------------------------------------------
|
|
0 [6144, 2048] [6144, 2048] [2048, 6144] 138.356 112.786 119.234 0.00002 0.00002 0.00002
|
|
1 [6144, 2048] [6144, 2048] [2048, 6144] 160.751 102.435 104.364 0.00002 0.00004 0.00003
|
|
2 [6144, 2048] [6144, 2048] [2048, 6144] 166.168 105.972 109.577 0.00002 0.00003 0.00003
|
|
3 [6144, 2048] [6144, 2048] [2048, 6144] 164.227 108.473 109.132 0.00002 0.00003 0.00003
|
|
4 [6144, 2048] [6144, 2048] [2048, 6144] 159.325 109.481 109.280 0.00002 0.00003 0.00003
|
|
5 [6144, 2048] [6144, 2048] [2048, 6144] 141.749 117.417 116.503 0.00002 0.00002 0.00003
|
|
6 [6144, 2048] [6144, 2048] [2048, 6144] 139.719 117.968 115.044 0.00002 0.00002 0.00003
|
|
7 [6144, 2048] [6144, 2048] [2048, 6144] 144.020 116.378 113.238 0.00002 0.00003 0.00003
|
|
8 [6144, 2048] [6144, 2048] [2048, 6144] 137.524 117.760 116.430 0.00002 0.00002 0.00002
|
|
9 [6144, 2048] [6144, 2048] [2048, 6144] 136.793 122.606 117.158 0.00002 0.00002 0.00003
|
|
10 [6144, 2048] [6144, 2048] [2048, 6144] 135.160 123.157 116.162 0.00002 0.00002 0.00003
|
|
11 [6144, 2048] [6144, 2048] [2048, 6144] 128.540 125.940 120.142 0.00002 0.00003 0.00002
|
|
12 [6144, 2048] [6144, 2048] [2048, 6144] 125.090 125.435 120.075 0.00002 0.00002 0.00003
|
|
13 [6144, 2048] [6144, 2048] [2048, 6144] 120.812 122.147 117.098 0.00003 0.00003 0.00003
|
|
14 [6144, 2048] [6144, 2048] [2048, 6144] 115.589 118.312 116.246 0.00003 0.00002 0.00003
|
|
15 [6144, 2048] [6144, 2048] [2048, 6144] 116.356 120.975 116.761 0.00003 0.00003 0.00003
|
|
16 [6144, 2048] [6144, 2048] [2048, 6144] 115.659 124.277 117.831 0.00003 0.00002 0.00003
|
|
17 [6144, 2048] [6144, 2048] [2048, 6144] 115.404 129.050 121.534 0.00003 0.00002 0.00003
|
|
18 [6144, 2048] [6144, 2048] [2048, 6144] 116.670 129.370 124.809 0.00003 0.00003 0.00002
|
|
19 [6144, 2048] [6144, 2048] [2048, 6144] 118.280 131.654 128.726 0.00003 0.00002 0.00002
|
|
20 [6144, 2048] [6144, 2048] [2048, 6144] 122.886 134.977 132.033 0.00003 0.00002 0.00002
|
|
21 [6144, 2048] [6144, 2048] [2048, 6144] 125.949 137.228 134.445 0.00002 0.00002 0.00002
|
|
22 [6144, 2048] [6144, 2048] [2048, 6144] 128.172 138.566 136.816 0.00002 0.00002 0.00002
|
|
23 [6144, 2048] [6144, 2048] [2048, 6144] 129.759 139.033 138.182 0.00002 0.00002 0.00002
|
|
24 [6144, 2048] [6144, 2048] [2048, 6144] 128.384 139.729 138.658 0.00002 0.00002 0.00002
|
|
25 [6144, 2048] [6144, 2048] [2048, 6144] 126.516 140.019 137.531 0.00002 0.00002 0.00002
|
|
26 [6144, 2048] [6144, 2048] [2048, 6144] 125.452 139.144 135.120 0.00002 0.00002 0.00002
|
|
27 [6144, 2048] [6144, 2048] [2048, 6144] 132.785 137.233 125.503 0.00002 0.00002 0.00002
|
|
|
|
[13] Weight health diagnostics
|
|
|
|
Sparsity > 10%
|
|
none
|
|
|
|
|Kurtosis Delta| > 5.0
|
|
none
|
|
|
|
Outlier ratio > 1%
|
|
model.layers.0.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.0.self_attn.k_norm.weight outlier_ratio=0.02344
|
|
model.layers.0.input_layernorm.weight outlier_ratio=0.01416
|
|
model.layers.2.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.2.input_layernorm.weight outlier_ratio=0.02100
|
|
model.layers.3.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.3.input_layernorm.weight outlier_ratio=0.02295
|
|
model.layers.4.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.4.input_layernorm.weight outlier_ratio=0.01855
|
|
model.layers.5.self_attn.q_norm.weight outlier_ratio=0.03125
|
|
model.layers.5.self_attn.k_norm.weight outlier_ratio=0.02344
|
|
model.layers.5.input_layernorm.weight outlier_ratio=0.02100
|
|
model.layers.6.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.6.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.6.input_layernorm.weight outlier_ratio=0.02539
|
|
model.layers.7.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.7.input_layernorm.weight outlier_ratio=0.01074
|
|
model.layers.8.input_layernorm.weight outlier_ratio=0.02002
|
|
model.layers.9.self_attn.q_norm.weight outlier_ratio=0.02344
|
|
model.layers.9.self_attn.k_norm.weight outlier_ratio=0.03125
|
|
model.layers.10.self_attn.q_norm.weight outlier_ratio=0.03125
|
|
model.layers.10.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.11.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.11.input_layernorm.weight outlier_ratio=0.01172
|
|
model.layers.12.input_layernorm.weight outlier_ratio=0.01025
|
|
model.layers.13.self_attn.q_norm.weight outlier_ratio=0.02344
|
|
model.layers.13.input_layernorm.weight outlier_ratio=0.01123
|
|
model.layers.14.self_attn.q_norm.weight outlier_ratio=0.03125
|
|
model.layers.14.input_layernorm.weight outlier_ratio=0.01074
|
|
model.layers.15.self_attn.k_norm.weight outlier_ratio=0.02344
|
|
model.layers.16.input_layernorm.weight outlier_ratio=0.01025
|
|
model.layers.16.post_attention_layernorm.weight outlier_ratio=0.01221
|
|
model.layers.17.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.17.post_attention_layernorm.weight outlier_ratio=0.01172
|
|
model.layers.18.input_layernorm.weight outlier_ratio=0.01611
|
|
model.layers.18.post_attention_layernorm.weight outlier_ratio=0.01270
|
|
model.layers.19.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.19.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.19.input_layernorm.weight outlier_ratio=0.01416
|
|
model.layers.19.post_attention_layernorm.weight outlier_ratio=0.01416
|
|
model.layers.20.self_attn.q_norm.weight outlier_ratio=0.01562
|
|
model.layers.20.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.20.input_layernorm.weight outlier_ratio=0.02295
|
|
model.layers.20.post_attention_layernorm.weight outlier_ratio=0.01221
|
|
model.layers.21.input_layernorm.weight outlier_ratio=0.02295
|
|
model.layers.22.self_attn.k_norm.weight outlier_ratio=0.02344
|
|
model.layers.22.input_layernorm.weight outlier_ratio=0.02588
|
|
model.layers.22.post_attention_layernorm.weight outlier_ratio=0.01025
|
|
model.layers.23.input_layernorm.weight outlier_ratio=0.02197
|
|
model.layers.23.post_attention_layernorm.weight outlier_ratio=0.01123
|
|
model.layers.24.self_attn.k_norm.weight outlier_ratio=0.01562
|
|
model.layers.24.input_layernorm.weight outlier_ratio=0.02197
|
|
model.layers.24.post_attention_layernorm.weight outlier_ratio=0.01367
|
|
model.layers.25.input_layernorm.weight outlier_ratio=0.03174
|
|
model.layers.25.post_attention_layernorm.weight outlier_ratio=0.01514
|
|
model.layers.26.input_layernorm.weight outlier_ratio=0.04883
|
|
model.layers.26.post_attention_layernorm.weight outlier_ratio=0.01172
|
|
model.layers.27.self_attn.q_norm.weight outlier_ratio=0.02344
|
|
model.layers.27.input_layernorm.weight outlier_ratio=0.02148
|
|
model.norm.weight outlier_ratio=0.01416
|
|
|
|
Dead rows (L2 < 1e-6)
|
|
none
|
|
|
|
Low cosine sim vs base (<0.95)
|
|
none
|
|
|
|
Low SNR vs base (< 20 dB)
|
|
none
|
|
|
|
Note on kurtosis delta: Kurtosis values are reported as the difference (delta) compared to the base model.
|
|
A high kurtosis delta on tiny vectors (like norm/q-k-norm vectors of size 128) is statistically expected
|
|
due to small sample sizes and does not indicate a model health or representation collapse issue.
|
|
|
|
[14] Executive summary
|
|
shared tensors : 311
|
|
tensors changed vs base : 277 / 311
|
|
cosine similarity : mean = 0.999991 | median = 0.999992 | p10 = 0.999980 | p90 = 1.000000
|
|
relative error : mean = 0.001093 | median = 0.001293 | p10 = 0.000000 | p90 = 0.002338
|
|
SNR dB : mean = 81.86 | median = 47.79 | p10 = 43.89 | p90 = 262.77
|
|
high-sparsity layers (>10%) : 0
|
|
heavy-tail layers (|kurt_d|>5.0) : 0
|
|
dead-row layers : 0
|
|
low-cos layers (<0.95) : 0
|
|
low-SNR layers (<20 dB) : 0
|
|
distillation alpha : 0.3
|
|
|
|
checkpoint size on disk : 3.218 GiB
|
|
base weights in memory : 3.784 GiB
|
|
distilled weights in memory : 3.784 GiB
|
|
|
|
========================================================================================
|
|
END OF REPORT
|
|
========================================================================================
|