2e3e3c757c
Model: ligeng-dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume Source: Original Platform
597 lines
413 KiB
Plaintext
597 lines
413 KiB
Plaintext
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
./slurm_setup.sh: line 48: GLOBAL_TRAIN_BATCH_SIZE / NNODES / GPUS_PER_NODE / GRADIENT_ACCUMULATION_STEPS: division by 0 (error token is "GRADIENT_ACCUMULATION_STEPS")
|
|
2026-04-15 09:07:19,079 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,079 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,089 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,089 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,090 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,090 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,094 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,094 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,107 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,107 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,111 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,111 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,116 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,116 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,135 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,135 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,150 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,150 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,154 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,154 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,155 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,155 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,164 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,165 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,174 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,174 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,175 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,175 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,175 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,175 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,188 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,188 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,197 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,197 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,205 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,205 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,206 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,206 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,208 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,208 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,210 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,210 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,210 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,210 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,210 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,210 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,225 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,225 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,227 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,227 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,236 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,240 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,236 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,240 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,238 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,238 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,244 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,244 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,245 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,245 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,249 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,249 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,249 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,249 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,255 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,255 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,254 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,254 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,255 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,256 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,268 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,268 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,272 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,272 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,272 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,273 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,271 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,271 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,280 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,280 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,319 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,319 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,323 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,323 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,323 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,323 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,327 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,327 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,330 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,330 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,330 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,330 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,336 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,336 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,345 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,345 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,352 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,352 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,352 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,352 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,356 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,356 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,366 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,366 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,366 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,367 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,367 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,367 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,376 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,376 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,376 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,376 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,409 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,409 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,410 - INFO - PyTorch version 2.6.0 available.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,410 - INFO - PyTorch version 2.6.0 available.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,410 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,411 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,412 - INFO - PyTorch version 2.6.0 available.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,415 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,415 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,418 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,433 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,439 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,441 - INFO - PyTorch version 2.6.0 available.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,451 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,451 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,453 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,453 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,453 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,453 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,454 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,454 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,459 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,472 - INFO - Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
|
|
2026-04-15 09:07:19,472 - INFO - Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
|
|
2026-04-15 09:07:19,478 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,478 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,483 - INFO - PyTorch version 2.6.0 available.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,490 - INFO - PyTorch version 2.6.0 available.
|
|
/home/ligengz/anaconda3/lib/python3.11/site-packages/pandas/core/arrays/masked.py:60: UserWarning: Pandas requires version '1.3.6' or newer of 'bottleneck' (version '1.3.5' currently installed).
|
|
from pandas.core import (
|
|
2026-04-15 09:07:19,511 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,511 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,512 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,520 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,538 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,538 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,540 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,560 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,561 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,561 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,561 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,561 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,564 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,566 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,571 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,573 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,570 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,570 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,578 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,579 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,579 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,580 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,584 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,601 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,602 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,605 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,605 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,609 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,649 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,652 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,653 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,654 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,654 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,655 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,661 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,670 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,684 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,684 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,691 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,702 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,702 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,706 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,712 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,715 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,731 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,731 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,738 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,804 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,804 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,809 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,809 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:19,816 - INFO - PyTorch version 2.6.0 available.
|
|
2026-04-15 09:07:36,521 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,567 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,577 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,609 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,621 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,622 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,648 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,651 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,663 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,673 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,689 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,692 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,743 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,569 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,782 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,799 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,800 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,825 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,957 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,989 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:36,989 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:37,064 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:37,077 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:37,097 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3060', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:41,966 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,001 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,005 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,005 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,007 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,017 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,053 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,063 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,078 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2096', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,095 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_s2026-04-15 09:07:42,096 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
cheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,110 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,116 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,118 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,125 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,127 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,127 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,129 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,130 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_s2026-04-15 09:07:42,130 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
cheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,134 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_sec2026-04-15 09:07:42,136 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_sond': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
cheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,147 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,134 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2095', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:42,135 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_s2026-04-15 09:07:42,140 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloadcheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-35_batch-block1-2023', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_cer_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_secolumn_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
ond': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,130 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,149 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,168 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,248 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,248 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 6, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,289 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 4, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,308 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,316 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,353 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 7, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:48,364 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-2081', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:52,108 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3261', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:52,162 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3564', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:52,173 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 5, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
2026-04-15 09:07:52,218 - INFO - Training config: {'model_name': 'Qwen/Qwen3-8B', 'template_name': 'qwen', 'block_size': 40960, 'wandb_project': 'ThreadWeaver', 'train_file_path': './data-train_final_v2_nb2_mt8192_replaced_fix', 'dagger': False, 'attn_implementation': 'flex_attention', 'output_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 1, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 1e-05, 'weight_decay': 0.0001, 'adam_beta1': 0.9, 'adam_beta2': 0.95, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 8.0, 'max_steps': -1, 'lr_scheduler_type': <SchedulerType.COSINE: 'cosine'>, 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.05, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/runs/Apr15_09-07-36_batch-block1-3462', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': False, 'logging_steps': 1.0, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 20, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': 'runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, 'deepspeed': 'configs/deepspeed_zero3.json', 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH: 'adamw_torch'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': True, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': [], 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, '_n_gpu': 1, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'use_liger_kernel': True, 'eval_use_gather_object': False, 'average_tokens_across_devices': True, 'model_init_kwargs': None, 'chat_template_path': None, 'dataset_text_field': 'qwen_text', 'dataset_kwargs': None, 'dataset_num_proc': None, 'eos_token': None, 'pad_token': None, 'max_length': 1024, 'packing': False, 'packing_strategy': 'ffd', 'padding_free': False, 'pad_to_multiple_of': None, 'eval_packing': None, 'completion_only_loss': None, 'assistant_only_loss': False, 'activation_offloading': False, 'max_seq_length': None}
|
|
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.93s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.93s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.93s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.90s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.93s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.93s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.92s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:13, 3.32s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:07<00:11, 3.68s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading | 1/5 [00:02<00:11, 2.91s/it]
|
|
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.93s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.54s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.55s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.62s/it]
|
|
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.71s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
:03, 3.40s/it]
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.40s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.75s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.74s/it]
|
|
|
|
Loading checkpoint shards: 80%|████████ | 4/5 [00:14<00:03, 3.47s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.44s/it]
|
|
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.94s/it]
|
|
|
|
Tokenizing train dataset: 0%| | 0/964 [00:00<?, ? examples/s]
|
|
Tokenizing train dataset: 0%| | 4/964 [00:00<00:33, 28.52 examples/s]
|
|
Tokenizing train dataset: 1%| | 8/964 [00:00<00:29, 32.35 examples/s]
|
|
Tokenizing train dataset: 1%| | 12/964 [00:00<00:29, 32.14 examples/s]
|
|
Tokenizing train dataset: 2%|▏ | 16/964 [00:00<00:31, 30.19 examples/s]
|
|
Tokenizing train dataset: 2%|▏ | 21/964 [00:00<00:26, 35.25 examples/s]
|
|
Tokenizing train dataset: 3%|▎ | 25/964 [00:00<00:26, 35.20 examples/s]
|
|
Tokenizing train dataset: 3%|▎ | 29/964 [00:00<00:28, 32.87 examples/s]
|
|
Tokenizing train dataset: 4%|▎ | 34/964 [00:01<00:31, 29.56 examples/s]
|
|
Tokenizing train dataset: 4%|▍ | 38/964 [00:01<00:32, 28.54 examples/s]
|
|
Tokenizing train dataset: 4%|▍ | 42/964 [00:01<00:35, 26.00 examples/s]
|
|
Tokenizing train dataset: 5%|▍ | 48/964 [00:01<00:30, 30.46 examples/s]
|
|
Tokenizing train dataset: 5%|▌ | 52/964 [00:01<00:29, 31.02 examples/s]
|
|
Tokenizing train dataset: 6%|▌ | 56/964 [00:01<00:34, 26.48 examples/s]
|
|
Tokenizing train dataset: 6%|▌ | 60/964 [00:02<00:33, 26.72 examples/s]
|
|
Tokenizing train dataset: 7%|▋ | 64/964 [00:02<00:31, 28.80 examples/s]
|
|
Tokenizing train dataset: 7%|▋ | 69/964 [00:02<00:31, 28.04 examples/s]
|
|
Tokenizing train dataset: 8%|▊ | 74/964 [00:02<00:28, 31.10 examples/s]
|
|
Tokenizing train dataset: 8%|▊ | 79/964 [00:02<00:26, 34.04 examples/s]
|
|
Tokenizing train dataset: 9%|▉ | 85/964 [00:02<00:26, 33.13 examples/s]
|
|
Tokenizing train dataset: 9%|▉ | 89/964 [00:02<00:27, 31.62 examples/s]
|
|
Tokenizing train dataset: 10%|▉ | 93/964 [00:03<00:27, 31.86 examples/s]
|
|
Tokenizing train dataset: 10%|█ | 98/964 [00:03<00:27, 31.95 examples/s]
|
|
Tokenizing train dataset: 11%|█ | 102/964 [00:03<00:31, 27.66 examples/s]
|
|
Tokenizing train dataset: 11%|█ | 106/964 [00:03<00:30, 28.34 examples/s]
|
|
Tokenizing train dataset: 12%|█▏ | 111/964 [00:03<00:28, 29.98 examples/s]
|
|
Tokenizing train dataset: 12%|█▏ | 116/964 [00:03<00:30, 28.21 examples/s]
|
|
Tokenizing train dataset: 13%|█▎ | 121/964 [00:04<00:28, 29.16 examples/s]
|
|
Tokenizing train dataset: 13%|█▎ | 126/964 [00:04<00:27, 30.97 examples/s]
|
|
Tokenizing train dataset: 14%|█▎ | 132/964 [00:04<00:27, 30.27 examples/s]
|
|
Tokenizing train dataset: 14%|█▍ | 136/964 [00:04<00:28, 28.75 examples/s]
|
|
Tokenizing train dataset: 14%|█▍ | 139/964 [00:04<00:30, 27.27 examples/s]
|
|
Tokenizing train dataset: 15%|█▍ | 143/964 [00:04<00:30, 27.14 examples/s]
|
|
Tokenizing train dataset: 15%|█▌ | 147/964 [00:04<00:28, 28.74 examples/s]
|
|
Tokenizing train dataset: 16%|█▌ | 152/964 [00:05<00:25, 32.28 examples/s]
|
|
Tokenizing train dataset: 16%|█▋ | 158/964 [00:05<00:24, 32.65 examples/s]
|
|
Tokenizing train dataset: 17%|█▋ | 163/964 [00:05<00:26, 29.85 examples/s]
|
|
Tokenizing train dataset: 17%|█▋ | 168/964 [00:05<00:26, 30.43 examples/s]
|
|
Tokenizing train dataset: 18%|█▊ | 172/964 [00:05<00:27, 28.41 examples/s]
|
|
Tokenizing train dataset: 18%|█▊ | 176/964 [00:05<00:26, 29.48 examples/s]
|
|
Tokenizing train dataset: 19%|█▊ | 180/964 [00:05<00:25, 31.03 examples/s]
|
|
Tokenizing train dataset: 19%|█▉ | 184/964 [00:06<00:32, 23.99 examples/s]
|
|
Tokenizing train dataset: 19%|█▉ | 187/964 [00:06<00:32, 23.81 examples/s]
|
|
Tokenizing train dataset: 20%|█▉ | 190/964 [00:06<00:33, 22.87 examples/s]
|
|
Tokenizing train dataset: 20%|██ | 193/964 [00:06<00:36, 21.27 examples/s]
|
|
Tokenizing train dataset: 20%|██ | 197/964 [00:06<00:33, 22.88 examples/s]
|
|
Tokenizing train dataset: 21%|██ | 201/964 [00:06<00:30, 25.35 examples/s]
|
|
Tokenizing train dataset: 21%|██▏ | 205/964 [00:07<00:29, 25.52 examples/s]
|
|
Tokenizing train dataset: 22%|██▏ | 209/964 [00:07<00:29, 25.45 examples/s]
|
|
Tokenizing train dataset: 22%|██▏ | 213/964 [00:07<00:28, 26.03 examples/s]
|
|
Tokenizing train dataset: 22%|██▏ | 216/964 [00:07<00:29, 25.27 examples/s]
|
|
Tokenizing train dataset: 23%|██▎ | 221/964 [00:07<00:25, 29.49 examples/s]
|
|
Tokenizing train dataset: 23%|██▎ | 225/964 [00:07<00:26, 28.12 examples/s]
|
|
Tokenizing train dataset: 24%|██▍ | 229/964 [00:07<00:25, 29.04 examples/s]
|
|
Tokenizing train dataset: 24%|██▍ | 232/964 [00:08<00:25, 28.92 examples/s]
|
|
Tokenizing train dataset: 24%|██▍ | 236/964 [00:08<00:26, 27.13 examples/s]
|
|
Tokenizing train dataset: 25%|██▍ | 239/964 [00:08<00:28, 25.34 examples/s]
|
|
Tokenizing train dataset: 25%|██▌ | 243/964 [00:08<00:28, 25.31 examples/s]
|
|
Tokenizing train dataset: 26%|██▌ | 248/964 [00:08<00:24, 29.48 examples/s]
|
|
Tokenizing train dataset: 26%|██▌ | 252/964 [00:08<00:22, 31.10 examples/s]
|
|
Tokenizing train dataset: 27%|██▋ | 256/964 [00:08<00:21, 32.53 examples/s]
|
|
Tokenizing train dataset: 27%|██▋ | 260/964 [00:09<00:23, 30.35 examples/s]
|
|
Tokenizing train dataset: 27%|██▋ | 264/964 [00:09<00:23, 30.17 examples/s]
|
|
Tokenizing train dataset: 28%|██▊ | 270/964 [00:09<00:21, 32.23 examples/s]
|
|
Tokenizing train dataset: 28%|██▊ | 274/964 [00:09<00:25, 27.43 examples/s]
|
|
Tokenizing train dataset: 29%|██▊ | 277/964 [00:09<00:26, 26.16 examples/s]
|
|
Tokenizing train dataset: 29%|██▉ | 282/964 [00:09<00:22, 29.70 examples/s]
|
|
Tokenizing train dataset: 30%|██▉ | 287/964 [00:09<00:22, 30.23 examples/s]
|
|
Tokenizing train dataset: 30%|███ | 291/964 [00:10<00:22, 29.44 examples/s]
|
|
Tokenizing train dataset: 31%|███ | 295/964 [00:10<00:22, 29.69 examples/s]
|
|
Tokenizing train dataset: 31%|███ | 300/964 [00:10<00:22, 28.87 examples/s]
|
|
Tokenizing train dataset: 31%|███▏ | 303/964 [00:10<00:23, 28.44 examples/s]
|
|
Tokenizing train dataset: 32%|███▏ | 307/964 [00:10<00:23, 28.22 examples/s]
|
|
Tokenizing train dataset: 32%|███▏ | 311/964 [00:10<00:21, 30.69 examples/s]
|
|
Tokenizing train dataset: 33%|███▎ | 316/964 [00:10<00:22, 29.39 examples/s]
|
|
Tokenizing train dataset: 33%|███▎ | 322/964 [00:11<00:19, 32.79 examples/s]
|
|
Tokenizing train dataset: 34%|███▍ | 326/964 [00:11<00:20, 31.04 examples/s]
|
|
Tokenizing train dataset: 34%|███▍ | 331/964 [00:11<00:18, 34.04 examples/s]
|
|
Tokenizing train dataset: 35%|███▍ | 335/964 [00:11<00:19, 32.24 examples/s]
|
|
Tokenizing train dataset: 35%|███▌ | 339/964 [00:11<00:20, 30.12 examples/s]
|
|
Tokenizing train dataset: 36%|███▌ | 343/964 [00:11<00:22, 28.14 examples/s]
|
|
Tokenizing train dataset: 36%|███▌ | 347/964 [00:11<00:20, 29.62 examples/s]
|
|
Tokenizing train dataset: 37%|███▋ | 353/964 [00:12<00:17, 34.20 examples/s]
|
|
Tokenizing train dataset: 37%|███▋ | 357/964 [00:12<00:17, 34.23 examples/s]
|
|
Tokenizing train dataset: 38%|███▊ | 362/964 [00:12<00:17, 34.89 examples/s]
|
|
Tokenizing train dataset: 38%|███▊ | 367/964 [00:12<00:18, 32.50 examples/s]
|
|
Tokenizing train dataset: 39%|███▊ | 372/964 [00:12<00:18, 31.22 examples/s]
|
|
Tokenizing train dataset: 39%|███▉ | 377/964 [00:12<00:19, 30.10 examples/s]
|
|
Tokenizing train dataset: 40%|███▉ | 381/964 [00:13<00:20, 28.90 examples/s]
|
|
Tokenizing train dataset: 40%|████ | 386/964 [00:13<00:18, 30.55 examples/s]
|
|
Tokenizing train dataset: 40%|████ | 390/964 [00:13<00:19, 29.14 examples/s]
|
|
Tokenizing train dataset: 41%|████ | 394/964 [00:13<00:20, 27.96 examples/s]
|
|
Tokenizing train dataset: 41%|████▏ | 399/964 [00:13<00:20, 27.54 examples/s]
|
|
Tokenizing train dataset: 42%|████▏ | 402/964 [00:13<00:22, 25.16 examples/s]
|
|
Tokenizing train dataset: 42%|████▏ | 407/964 [00:13<00:19, 28.04 examples/s]
|
|
Tokenizing train dataset: 43%|████▎ | 411/964 [00:14<00:21, 26.17 examples/s]
|
|
Tokenizing train dataset: 43%|████▎ | 415/964 [00:14<00:20, 26.46 examples/s]
|
|
Tokenizing train dataset: 43%|████▎ | 419/964 [00:14<00:20, 26.95 examples/s]
|
|
Tokenizing train dataset: 44%|████▍ | 423/964 [00:14<00:20, 26.96 examples/s]
|
|
Tokenizing train dataset: 44%|████▍ | 426/964 [00:14<00:19, 26.97 examples/s]
|
|
Tokenizing train dataset: 45%|████▍ | 431/964 [00:14<00:18, 29.08 examples/s]
|
|
Tokenizing train dataset: 45%|████▌ | 435/964 [00:14<00:19, 27.71 examples/s]
|
|
Tokenizing train dataset: 46%|████▌ | 439/964 [00:15<00:18, 28.40 examples/s]
|
|
Tokenizing train dataset: 46%|████▌ | 445/964 [00:15<00:16, 31.73 examples/s]
|
|
Tokenizing train dataset: 47%|████▋ | 451/964 [00:15<00:16, 31.25 examples/s]
|
|
Tokenizing train dataset: 47%|████▋ | 455/964 [00:15<00:16, 30.81 examples/s]
|
|
Tokenizing train dataset: 48%|████▊ | 459/964 [00:15<00:17, 28.93 examples/s]
|
|
Tokenizing train dataset: 48%|████▊ | 464/964 [00:15<00:17, 29.16 examples/s]
|
|
Tokenizing train dataset: 49%|████▊ | 469/964 [00:16<00:17, 28.87 examples/s]
|
|
Tokenizing train dataset: 49%|████▉ | 473/964 [00:16<00:16, 30.01 examples/s]
|
|
Tokenizing train dataset: 50%|████▉ | 479/964 [00:16<00:14, 32.36 examples/s]
|
|
Tokenizing train dataset: 50%|█████ | 483/964 [00:16<00:15, 32.06 examples/s]
|
|
Tokenizing train dataset: 51%|█████ | 489/964 [00:16<00:16, 29.47 examples/s]
|
|
Tokenizing train dataset: 51%|█████ | 493/964 [00:16<00:15, 29.89 examples/s]
|
|
Tokenizing train dataset: 52%|█████▏ | 498/964 [00:17<00:16, 28.96 examples/s]
|
|
Tokenizing train dataset: 52%|█████▏ | 503/964 [00:17<00:15, 29.62 examples/s]
|
|
Tokenizing train dataset: 52%|█████▏ | 506/964 [00:17<00:16, 27.28 examples/s]
|
|
Tokenizing train dataset: 53%|█████▎ | 510/964 [00:17<00:15, 28.55 examples/s]
|
|
Tokenizing train dataset: 53%|█████▎ | 513/964 [00:17<00:16, 27.71 examples/s]
|
|
Tokenizing train dataset: 54%|█████▎ | 517/964 [00:17<00:16, 26.66 examples/s]
|
|
Tokenizing train dataset: 54%|█████▍ | 522/964 [00:17<00:15, 27.76 examples/s]
|
|
Tokenizing train dataset: 55%|█████▍ | 527/964 [00:18<00:14, 31.11 examples/s]
|
|
Tokenizing train dataset: 55%|█████▌ | 532/964 [00:18<00:12, 33.52 examples/s]
|
|
Tokenizing train dataset: 56%|█████▌ | 536/964 [00:18<00:13, 31.29 examples/s]
|
|
Tokenizing train dataset: 56%|█████▌ | 541/964 [00:18<00:13, 31.90 examples/s]
|
|
Tokenizing train dataset: 57%|█████▋ | 545/964 [00:18<00:13, 29.99 examples/s]
|
|
Tokenizing train dataset: 57%|█████▋ | 550/964 [00:18<00:12, 32.30 examples/s]
|
|
Tokenizing train dataset: 57%|█████▋ | 554/964 [00:18<00:13, 29.96 examples/s]
|
|
Tokenizing train dataset: 58%|█████▊ | 558/964 [00:19<00:13, 29.44 examples/s]
|
|
Tokenizing train dataset: 58%|█████▊ | 562/964 [00:19<00:15, 26.08 examples/s]
|
|
Tokenizing train dataset: 59%|█████▊ | 565/964 [00:19<00:15, 25.90 examples/s]
|
|
Tokenizing train dataset: 59%|█████▉ | 569/964 [00:19<00:14, 26.82 examples/s]
|
|
Tokenizing train dataset: 60%|█████▉ | 574/964 [00:19<00:13, 28.77 examples/s]
|
|
Tokenizing train dataset: 60%|██████ | 579/964 [00:19<00:12, 31.11 examples/s]
|
|
Tokenizing train dataset: 61%|██████ | 584/964 [00:20<00:12, 29.89 examples/s]
|
|
Tokenizing train dataset: 61%|██████ | 589/964 [00:20<00:11, 33.90 examples/s]
|
|
Tokenizing train dataset: 62%|██████▏ | 593/964 [00:20<00:10, 34.90 examples/s]
|
|
Tokenizing train dataset: 62%|██████▏ | 598/964 [00:20<00:11, 32.49 examples/s]
|
|
Tokenizing train dataset: 63%|██████▎ | 604/964 [00:20<00:11, 31.87 examples/s]
|
|
Tokenizing train dataset: 63%|██████▎ | 609/964 [00:20<00:12, 29.55 examples/s]
|
|
Tokenizing train dataset: 64%|██████▎ | 613/964 [00:20<00:11, 29.70 examples/s]
|
|
Tokenizing train dataset: 64%|██████▍ | 618/964 [00:21<00:11, 29.43 examples/s]
|
|
Tokenizing train dataset: 65%|██████▍ | 622/964 [00:21<00:11, 29.27 examples/s]
|
|
Tokenizing train dataset: 65%|██████▌ | 627/964 [00:21<00:11, 30.50 examples/s]
|
|
Tokenizing train dataset: 66%|██████▌ | 632/964 [00:21<00:12, 26.44 examples/s]
|
|
Tokenizing train dataset: 66%|██████▌ | 635/964 [00:21<00:13, 25.26 examples/s]
|
|
Tokenizing train dataset: 66%|██████▋ | 639/964 [00:21<00:11, 27.18 examples/s]
|
|
Tokenizing train dataset: 67%|██████▋ | 643/964 [00:22<00:11, 27.24 examples/s]
|
|
Tokenizing train dataset: 67%|██████▋ | 649/964 [00:22<00:09, 31.71 examples/s]
|
|
Tokenizing train dataset: 68%|██████▊ | 653/964 [00:22<00:09, 32.12 examples/s]
|
|
Tokenizing train dataset: 68%|██████▊ | 657/964 [00:22<00:10, 29.91 examples/s]
|
|
Tokenizing train dataset: 69%|██████▊ | 661/964 [00:22<00:09, 30.99 examples/s]
|
|
Tokenizing train dataset: 69%|██████▉ | 665/964 [00:22<00:10, 29.45 examples/s]
|
|
Tokenizing train dataset: 69%|██████▉ | 668/964 [00:22<00:10, 28.53 examples/s]
|
|
Tokenizing train dataset: 70%|██████▉ | 673/964 [00:22<00:09, 29.36 examples/s]
|
|
Tokenizing train dataset: 70%|███████ | 676/964 [00:23<00:10, 28.49 examples/s]
|
|
Tokenizing train dataset: 71%|███████ | 680/964 [00:23<00:09, 28.55 examples/s]
|
|
Tokenizing train dataset: 71%|███████ | 685/964 [00:23<00:09, 30.87 examples/s]
|
|
Tokenizing train dataset: 72%|███████▏ | 690/964 [00:23<00:10, 26.37 examples/s]
|
|
Tokenizing train dataset: 72%|███████▏ | 695/964 [00:23<00:09, 28.27 examples/s]
|
|
Tokenizing train dataset: 73%|███████▎ | 699/964 [00:23<00:08, 30.55 examples/s]
|
|
Tokenizing train dataset: 73%|███████▎ | 704/964 [00:24<00:08, 29.76 examples/s]
|
|
Tokenizing train dataset: 74%|███████▎ | 710/964 [00:24<00:07, 33.61 examples/s]
|
|
Tokenizing train dataset: 74%|███████▍ | 714/964 [00:24<00:07, 33.65 examples/s]
|
|
Tokenizing train dataset: 75%|███████▍ | 719/964 [00:24<00:07, 34.88 examples/s]
|
|
Tokenizing train dataset: 75%|███████▌ | 723/964 [00:24<00:07, 30.69 examples/s]
|
|
Tokenizing train dataset: 76%|███████▌ | 728/964 [00:24<00:07, 31.27 examples/s]
|
|
Tokenizing train dataset: 76%|███████▌ | 732/964 [00:24<00:07, 29.91 examples/s]
|
|
Tokenizing train dataset: 76%|███████▋ | 736/964 [00:25<00:07, 30.04 examples/s]
|
|
Tokenizing train dataset: 77%|███████▋ | 740/964 [00:25<00:08, 26.56 examples/s]
|
|
Tokenizing train dataset: 77%|███████▋ | 744/964 [00:25<00:08, 24.90 examples/s]
|
|
Tokenizing train dataset: 78%|███████▊ | 748/964 [00:25<00:08, 25.39 examples/s]
|
|
Tokenizing train dataset: 78%|███████▊ | 752/964 [00:25<00:08, 26.05 examples/s]
|
|
Tokenizing train dataset: 79%|███████▊ | 757/964 [00:25<00:07, 28.15 examples/s]
|
|
Tokenizing train dataset: 79%|███████▉ | 763/964 [00:26<00:06, 31.11 examples/s]
|
|
Tokenizing train dataset: 80%|███████▉ | 769/964 [00:26<00:05, 35.14 examples/s]
|
|
Tokenizing train dataset: 80%|████████ | 774/964 [00:26<00:05, 32.89 examples/s]
|
|
Tokenizing train dataset: 81%|████████ | 780/964 [00:26<00:06, 29.78 examples/s]
|
|
Tokenizing train dataset: 81%|████████▏ | 785/964 [00:26<00:05, 30.98 examples/s]
|
|
Tokenizing train dataset: 82%|████████▏ | 790/964 [00:26<00:05, 32.48 examples/s]
|
|
Tokenizing train dataset: 82%|████████▏ | 794/964 [00:26<00:05, 33.25 examples/s]
|
|
Tokenizing train dataset: 83%|████████▎ | 800/964 [00:27<00:05, 30.06 examples/s]
|
|
Tokenizing train dataset: 83%|████████▎ | 804/964 [00:27<00:05, 30.78 examples/s]
|
|
Tokenizing train dataset: 84%|████████▍ | 808/964 [00:27<00:05, 30.52 examples/s]
|
|
Tokenizing train dataset: 84%|████████▍ | 812/964 [00:27<00:05, 29.00 examples/s]
|
|
Tokenizing train dataset: 85%|████████▍ | 816/964 [00:27<00:05, 27.76 examples/s]
|
|
Tokenizing train dataset: 85%|████████▍ | 819/964 [00:27<00:05, 27.36 examples/s]
|
|
Tokenizing train dataset: 85%|████████▌ | 824/964 [00:28<00:05, 27.91 examples/s]
|
|
Tokenizing train dataset: 86%|████████▌ | 827/964 [00:28<00:04, 27.73 examples/s]
|
|
Tokenizing train dataset: 86%|████████▌ | 831/964 [00:28<00:05, 22.40 examples/s]
|
|
Tokenizing train dataset: 87%|████████▋ | 835/964 [00:28<00:05, 24.56 examples/s]
|
|
Tokenizing train dataset: 87%|████████▋ | 839/964 [00:28<00:04, 25.86 examples/s]
|
|
Tokenizing train dataset: 87%|████████▋ | 843/964 [00:28<00:04, 26.98 examples/s]
|
|
Tokenizing train dataset: 88%|████████▊ | 848/964 [00:28<00:04, 28.58 examples/s]
|
|
Tokenizing train dataset: 88%|████████▊ | 852/964 [00:29<00:04, 27.52 examples/s]
|
|
Tokenizing train dataset: 89%|████████▉ | 856/964 [00:29<00:03, 27.58 examples/s]
|
|
Tokenizing train dataset: 89%|████████▉ | 860/964 [00:29<00:03, 29.58 examples/s]
|
|
Tokenizing train dataset: 90%|████████▉ | 864/964 [00:29<00:03, 30.37 examples/s]
|
|
Tokenizing train dataset: 90%|█████████ | 868/964 [00:29<00:03, 29.38 examples/s]
|
|
Tokenizing train dataset: 90%|█████████ | 872/964 [00:29<00:03, 30.30 examples/s]
|
|
Tokenizing train dataset: 91%|█████████ | 877/964 [00:29<00:02, 31.05 examples/s]
|
|
Tokenizing train dataset: 91%|█████████▏| 881/964 [00:30<00:02, 30.77 examples/s]
|
|
Tokenizing train dataset: 92%|█████████▏| 885/964 [00:30<00:02, 31.56 examples/s]
|
|
Tokenizing train dataset: 92%|█████████▏| 889/964 [00:30<00:02, 31.74 examples/s]
|
|
Tokenizing train dataset: 93%|█████████▎| 894/964 [00:30<00:02, 34.79 examples/s]
|
|
Tokenizing train dataset: 93%|█████████▎| 898/964 [00:30<00:01, 33.39 examples/s]
|
|
Tokenizing train dataset: 94%|█████████▎| 902/964 [00:30<00:02, 29.37 examples/s]
|
|
Tokenizing train dataset: 94%|█████████▍| 906/964 [00:30<00:02, 28.00 examples/s]
|
|
Tokenizing train dataset: 95%|█████████▍| 911/964 [00:31<00:02, 26.12 examples/s]
|
|
Tokenizing train dataset: 95%|█████████▍| 915/964 [00:31<00:01, 27.68 examples/s]
|
|
Tokenizing train dataset: 95%|█████████▌| 918/964 [00:31<00:01, 26.09 examples/s]
|
|
Tokenizing train dataset: 96%|█████████▌| 923/964 [00:31<00:01, 28.19 examples/s]
|
|
Tokenizing train dataset: 96%|█████████▌| 926/964 [00:31<00:01, 27.96 examples/s]
|
|
Tokenizing train dataset: 97%|█████████▋| 931/964 [00:31<00:01, 31.41 examples/s]
|
|
Tokenizing train dataset: 97%|█████████▋| 936/964 [00:31<00:00, 31.77 examples/s]
|
|
Tokenizing train dataset: 98%|█████████▊| 941/964 [00:32<00:00, 29.88 examples/s]
|
|
Tokenizing train dataset: 98%|█████████▊| 946/964 [00:32<00:00, 31.14 examples/s]
|
|
Tokenizing train dataset: 99%|█████████▊| 951/964 [00:32<00:00, 26.81 examples/s]
|
|
Tokenizing train dataset: 99%|█████████▉| 956/964 [00:32<00:00, 28.84 examples/s]
|
|
Tokenizing train dataset: 100%|█████████▉| 961/964 [00:32<00:00, 27.84 examples/s]
|
|
Tokenizing train dataset: 100%|██████████| 964/964 [00:34<00:00, 6.09 examples/s]
|
|
Tokenizing train dataset: 100%|██████████| 964/964 [00:34<00:00, 27.70 examples/s]
|
|
|
|
Truncating train dataset: 0%| | 0/964 [00:00<?, ? examples/s]
|
|
Truncating train dataset: 100%|██████████| 964/964 [00:00<00:00, 4632.84 examples/s]
|
|
Truncating train dataset: 100%|██████████| 964/964 [00:00<00:00, 4458.85 examples/s]
|
|
2026-04-15 09:08:48,498 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,568 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,568 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,569 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,570 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,570 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,571 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,572 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,572 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,572 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,575 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,574 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,573 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,575 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,576 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,578 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,576 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,577 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,576 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,578 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,579 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,578 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,578 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,578 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,579 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,581 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,580 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,582 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,579 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,581 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,580 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,582 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,583 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,583 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,583 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,584 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,583 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,588 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,586 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,587 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,587 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,587 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,590 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,590 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,589 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,589 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,591 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,591 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,591 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,592 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,592 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,601 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,608 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,610 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,626 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,623 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
2026-04-15 09:08:48,632 - INFO - Applying Liger kernels to model instance with model type: qwen3 with kwargs: {}
|
|
wandb: WARNING The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.
|
|
wandb: Currently logged in as: ligeng-zhu to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
|
wandb: Tracking run with wandb version 0.21.0
|
|
wandb: Run data is saved locally in runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume/wandb/run-20260415_090852-tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume
|
|
wandb: Run `wandb offline` to turn off syncing.
|
|
wandb: Syncing run runs/dev/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume
|
|
wandb: ⭐️ View project at https://wandb.ai/ligeng-zhu/ThreadWeaver
|
|
wandb: 🚀 View run at https://wandb.ai/ligeng-zhu/ThreadWeaver/runs/tw-data-train_final_v2_nb2_mt8192_replaced_fix-8node-resume
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
|
|
0%| | 0/64 [00:00<?, ?it/s]`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
|
|
|
|
2%|▏ | 1/64 [03:07<3:16:54, 187.53s/it]
|
|
|
|
|
|
2%|▏ | 1/64 [03:07<3:16:54, 187.53s/it]
|
|
3%|▎ | 2/64 [05:44<2:54:55, 169.29s/it]
|
|
|
|
|
|
3%|▎ | 2/64 [05:44<2:54:55, 169.29s/it]
|
|
5%|▍ | 3/64 [08:21<2:46:42, 163.98s/it]
|
|
|
|
|
|
5%|▍ | 3/64 [08:21<2:46:42, 163.98s/it]
|
|
6%|▋ | 4/64 [11:01<2:42:29, 162.49s/it]
|
|
|
|
|
|
6%|▋ | 4/64 [11:01<2:42:29, 162.49s/it]
|
|
8%|▊ | 5/64 [13:54<2:43:19, 166.10s/it]
|
|
|
|
|
|
8%|▊ | 5/64 [13:54<2:43:19, 166.10s/it]
|
|
9%|▉ | 6/64 [16:32<2:37:54, 163.35s/it]
|
|
|
|
|
|
9%|▉ | 6/64 [16:32<2:37:54, 163.35s/it]
|
|
11%|█ | 7/64 [19:17<2:35:46, 163.97s/it]
|
|
|
|
|
|
11%|█ | 7/64 [19:17<2:35:46, 163.97s/it]
|
|
12%|█▎ | 8/64 [21:50<2:29:46, 160.48s/it]
|
|
|
|
|
|
12%|█▎ | 8/64 [21:50<2:29:46, 160.48s/it]
|
|
14%|█▍ | 9/64 [24:50<2:32:42, 166.59s/it]
|
|
|
|
|
|
14%|█▍ | 9/64 [24:50<2:32:42, 166.59s/it]
|
|
16%|█▌ | 10/64 [27:23<2:26:08, 162.38s/it]
|
|
|
|
|
|
16%|█▌ | 10/64 [27:23<2:26:08, 162.38s/it]
|
|
17%|█▋ | 11/64 [29:59<2:21:32, 160.24s/it]
|
|
|
|
|
|
17%|█▋ | 11/64 [29:59<2:21:32, 160.24s/it]
|
|
19%|█▉ | 12/64 [32:38<2:18:45, 160.11s/it]
|
|
|
|
|
|
19%|█▉ | 12/64 [32:38<2:18:45, 160.11s/it]
|
|
20%|██ | 13/64 [35:34<2:20:06, 164.83s/it]
|
|
|
|
|
|
20%|██ | 13/64 [35:34<2:20:06, 164.83s/it]
|
|
22%|██▏ | 14/64 [38:10<2:15:14, 162.29s/it]
|
|
|
|
|
|
22%|██▏ | 14/64 [38:10<2:15:14, 162.29s/it]
|
|
23%|██▎ | 15/64 [40:45<2:10:40, 160.01s/it]
|
|
|
|
|
|
23%|██▎ | 15/64 [40:45<2:10:40, 160.01s/it]
|
|
25%|██▌ | 16/64 [43:18<2:06:13, 157.78s/it]
|
|
|
|
|
|
25%|██▌ | 16/64 [43:18<2:06:13, 157.78s/it]
|
|
27%|██▋ | 17/64 [46:16<2:08:22, 163.87s/it]
|
|
|
|
|
|
27%|██▋ | 17/64 [46:16<2:08:22, 163.87s/it]
|
|
28%|██▊ | 18/64 [48:49<2:03:08, 160.62s/it]
|
|
|
|
|
|
28%|██▊ | 18/64 [48:49<2:03:08, 160.62s/it]
|
|
30%|██▉ | 19/64 [51:25<1:59:20, 159.12s/it]
|
|
|
|
|
|
30%|██▉ | 19/64 [51:25<1:59:20, 159.12s/it]
|
|
31%|███▏ | 20/64 [54:05<1:56:56, 159.46s/it]
|
|
|
|
|
|
31%|███▏ | 20/64 [54:05<1:56:56, 159.46s/it]wandb: WARNING The get_url method is deprecated and will be removed in a future release. Please use `run.url` instead.
|
|
|
|
33%|███▎ | 21/64 [57:12<2:00:10, 167.68s/it]
|
|
|
|
|
|
33%|███▎ | 21/64 [57:12<2:00:10, 167.68s/it]
|
|
34%|███▍ | 22/64 [59:54<1:56:17, 166.13s/it]
|
|
|
|
|
|
34%|███▍ | 22/64 [59:54<1:56:17, 166.13s/it]
|
|
36%|███▌ | 23/64 [1:02:37<1:52:55, 165.25s/it]
|
|
|
|
|
|
36%|███▌ | 23/64 [1:02:37<1:52:55, 165.25s/it]
|
|
38%|███▊ | 24/64 [1:05:11<1:47:45, 161.63s/it]
|
|
|
|
|
|
38%|███▊ | 24/64 [1:05:11<1:47:45, 161.63s/it]
|
|
39%|███▉ | 25/64 [1:08:09<1:48:20, 166.69s/it]
|
|
|
|
|
|
39%|███▉ | 25/64 [1:08:09<1:48:20, 166.69s/it]
|
|
41%|████ | 26/64 [1:10:42<1:42:58, 162.59s/it]
|
|
|
|
|
|
41%|████ | 26/64 [1:10:42<1:42:58, 162.59s/it]
|
|
42%|████▏ | 27/64 [1:13:17<1:38:56, 160.46s/it]
|
|
|
|
|
|
42%|████▏ | 27/64 [1:13:18<1:38:56, 160.46s/it]
|
|
44%|████▍ | 28/64 [1:15:59<1:36:31, 160.89s/it]
|
|
|
|
|
|
44%|████▍ | 28/64 [1:15:59<1:36:31, 160.89s/it]
|
|
45%|████▌ | 29/64 [1:18:54<1:36:16, 165.06s/it]
|
|
|
|
|
|
45%|████▌ | 29/64 [1:18:54<1:36:16, 165.06s/it]
|
|
47%|████▋ | 30/64 [1:21:40<1:33:42, 165.37s/it]
|
|
|
|
|
|
47%|████▋ | 30/64 [1:21:40<1:33:42, 165.37s/it]
|
|
48%|████▊ | 31/64 [1:24:15<1:29:11, 162.16s/it]
|
|
|
|
|
|
48%|████▊ | 31/64 [1:24:15<1:29:11, 162.16s/it]
|
|
50%|█████ | 32/64 [1:26:48<1:24:57, 159.29s/it]
|
|
|
|
|
|
50%|█████ | 32/64 [1:26:48<1:24:57, 159.29s/it]
|
|
52%|█████▏ | 33/64 [1:29:51<1:26:04, 166.61s/it]
|
|
|
|
|
|
52%|█████▏ | 33/64 [1:29:51<1:26:04, 166.61s/it]
|
|
53%|█████▎ | 34/64 [1:32:25<1:21:23, 162.77s/it]
|
|
|
|
|
|
53%|█████▎ | 34/64 [1:32:25<1:21:23, 162.77s/it]
|
|
55%|█████▍ | 35/64 [1:35:00<1:17:34, 160.50s/it]
|
|
|
|
|
|
55%|█████▍ | 35/64 [1:35:00<1:17:34, 160.50s/it]
|
|
56%|█████▋ | 36/64 [1:37:40<1:14:51, 160.42s/it]
|
|
|
|
|
|
56%|█████▋ | 36/64 [1:37:40<1:14:51, 160.42s/it]
|
|
58%|█████▊ | 37/64 [1:40:28<1:13:09, 162.57s/it]
|
|
|
|
|
|
58%|█████▊ | 37/64 [1:40:28<1:13:09, 162.57s/it]
|
|
59%|█████▉ | 38/64 [1:43:15<1:10:57, 163.77s/it]
|
|
|
|
|
|
59%|█████▉ | 38/64 [1:43:15<1:10:57, 163.77s/it]
|
|
61%|██████ | 39/64 [1:45:56<1:07:55, 163.01s/it]
|
|
|
|
|
|
61%|██████ | 39/64 [1:45:56<1:07:55, 163.01s/it]
|
|
62%|██████▎ | 40/64 [1:48:28<1:03:55, 159.79s/it]
|
|
|
|
|
|
62%|██████▎ | 40/64 [1:48:28<1:03:55, 159.79s/it]
|
|
64%|██████▍ | 41/64 [1:51:45<1:05:33, 171.03s/it]
|
|
|
|
|
|
64%|██████▍ | 41/64 [1:51:45<1:05:33, 171.03s/it]
|
|
66%|██████▌ | 42/64 [1:54:19<1:00:49, 165.89s/it]
|
|
|
|
|
|
66%|██████▌ | 42/64 [1:54:19<1:00:49, 165.89s/it]
|
|
67%|██████▋ | 43/64 [1:56:54<56:51, 162.43s/it]
|
|
|
|
|
|
67%|██████▋ | 43/64 [1:56:54<56:51, 162.43s/it]
|
|
69%|██████▉ | 44/64 [1:59:34<53:55, 161.78s/it]
|
|
|
|
|
|
69%|██████▉ | 44/64 [1:59:34<53:55, 161.78s/it]
|
|
70%|███████ | 45/64 [2:02:13<51:00, 161.06s/it]
|
|
|
|
|
|
70%|███████ | 45/64 [2:02:13<51:00, 161.06s/it]
|
|
72%|███████▏ | 46/64 [2:04:50<47:57, 159.88s/it]
|
|
|
|
|
|
72%|███████▏ | 46/64 [2:04:50<47:57, 159.88s/it]
|
|
73%|███████▎ | 47/64 [2:07:38<45:59, 162.32s/it]
|
|
|
|
|
|
73%|███████▎ | 47/64 [2:07:39<45:59, 162.32s/it]
|
|
75%|███████▌ | 48/64 [2:10:11<42:31, 159.50s/it]
|
|
|
|
|
|
75%|███████▌ | 48/64 [2:10:11<42:31, 159.50s/it]
|
|
77%|███████▋ | 49/64 [2:13:10<41:17, 165.14s/it]
|
|
|
|
|
|
77%|███████▋ | 49/64 [2:13:10<41:17, 165.14s/it]
|
|
78%|███████▊ | 50/64 [2:15:43<37:43, 161.68s/it]
|
|
|
|
|
|
78%|███████▊ | 50/64 [2:15:43<37:43, 161.68s/it]
|
|
80%|███████▉ | 51/64 [2:18:18<34:33, 159.48s/it]
|
|
|
|
|
|
80%|███████▉ | 51/64 [2:18:18<34:33, 159.48s/it]
|
|
81%|████████▏ | 52/64 [2:20:57<31:53, 159.50s/it]
|
|
|
|
|
|
81%|████████▏ | 52/64 [2:20:57<31:53, 159.50s/it]
|
|
83%|████████▎ | 53/64 [2:23:48<29:50, 162.77s/it]
|
|
|
|
|
|
83%|████████▎ | 53/64 [2:23:48<29:50, 162.77s/it]
|
|
84%|████████▍ | 54/64 [2:26:27<26:58, 161.85s/it]
|
|
|
|
|
|
84%|████████▍ | 54/64 [2:26:27<26:58, 161.85s/it]
|
|
86%|████████▌ | 55/64 [2:29:02<23:57, 159.68s/it]
|
|
|
|
|
|
86%|████████▌ | 55/64 [2:29:02<23:57, 159.68s/it]
|
|
88%|████████▊ | 56/64 [2:31:34<21:00, 157.50s/it]
|
|
|
|
|
|
88%|████████▊ | 56/64 [2:31:34<21:00, 157.50s/it]
|
|
89%|████████▉ | 57/64 [2:34:37<19:15, 165.01s/it]
|
|
|
|
|
|
89%|████████▉ | 57/64 [2:34:37<19:15, 165.01s/it]
|
|
91%|█████████ | 58/64 [2:37:10<16:09, 161.55s/it]
|
|
|
|
|
|
91%|█████████ | 58/64 [2:37:10<16:09, 161.55s/it]
|
|
92%|█████████▏| 59/64 [2:39:44<13:16, 159.30s/it]
|
|
|
|
|
|
92%|█████████▏| 59/64 [2:39:44<13:16, 159.30s/it]
|
|
94%|█████████▍| 60/64 [2:42:24<10:37, 159.49s/it]
|
|
|
|
|
|
94%|█████████▍| 60/64 [2:42:24<10:37, 159.49s/it]
|
|
95%|█████████▌| 61/64 [2:45:37<08:27, 169.32s/it]
|
|
|
|
|
|
95%|█████████▌| 61/64 [2:45:37<08:27, 169.32s/it]
|
|
97%|█████████▋| 62/64 [2:48:21<05:35, 167.97s/it]
|
|
|
|
|
|
97%|█████████▋| 62/64 [2:48:21<05:35, 167.97s/it]
|
|
98%|█████████▊| 63/64 [2:51:04<02:46, 166.28s/it]
|
|
|
|
|
|
98%|█████████▊| 63/64 [2:51:04<02:46, 166.28s/it]
|
|
100%|██████████| 64/64 [2:53:36<00:00, 162.12s/it]
|
|
|
|
|
|
100%|██████████| 64/64 [2:53:36<00:00, 162.12s/it]
|
|
|
|
|
|
100%|██████████| 64/64 [2:53:55<00:00, 162.12s/it]
|
|
100%|██████████| 64/64 [2:53:55<00:00, 163.05s/it]
|