==> Python + torch sanity
Python 3.12.13
torch 2.7.1+cu128 cuda True NVIDIA L40S

==> apt deps

==> upgrade setuptools/wheel/pip
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

==> install training deps (TRL + peft + accelerate + wandb + flash-attn + aws sdk)
Collecting trl>=0.12
  Downloading trl-1.8.0-py3-none-any.whl.metadata (12 kB)
Collecting peft>=0.13
  Downloading peft-0.19.1-py3-none-any.whl.metadata (15 kB)
Collecting accelerate>=1.0
  Downloading accelerate-1.14.0-py3-none-any.whl.metadata (19 kB)
Collecting transformers<5,>=4.45
  Downloading transformers-4.57.6-py3-none-any.whl.metadata (43 kB)
Collecting datasets>=2.20
  Downloading datasets-5.0.0-py3-none-any.whl.metadata (23 kB)
Collecting tokenizers>=0.20
  Downloading tokenizers-0.23.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.8 kB)
Collecting wandb>=0.18
  Downloading wandb-0.28.0-py3-none-manylinux_2_28_x86_64.whl.metadata (11 kB)
Collecting tqdm
  Downloading tqdm-4.68.4-py3-none-any.whl.metadata (57 kB)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.12/dist-packages (6.0.3)
Collecting python-dotenv
  Downloading python_dotenv-1.2.2-py3-none-any.whl.metadata (27 kB)
Collecting rich
  Downloading rich-15.0.0-py3-none-any.whl.metadata (18 kB)
Collecting typer
  Downloading typer-0.26.8-py3-none-any.whl.metadata (15 kB)
Collecting zstandard
  Downloading zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (3.3 kB)
Collecting sentencepiece
  Downloading sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (33 kB)
Collecting protobuf<6,>=5.26
  Downloading protobuf-5.29.6-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Requirement already satisfied: requests in /usr/local/lib/python3.12/dist-packages (2.34.2)
Collecting huggingface_hub>=0.25 (from huggingface_hub[cli]>=0.25)
  Downloading huggingface_hub-1.23.0-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from transformers<5,>=4.45) (3.29.7)
  Downloading huggingface_hub-0.36.2-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.12/dist-packages (from transformers<5,>=4.45) (2.4.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from transformers<5,>=4.45) (26.2)
Collecting regex!=2019.12.17 (from transformers<5,>=4.45)
  Downloading regex-2026.7.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
Collecting tokenizers>=0.20
  Downloading tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Collecting safetensors>=0.4.3 (from transformers<5,>=4.45)
  Downloading safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub>=0.25->huggingface_hub[cli]>=0.25) (2026.4.0)
Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface_hub>=0.25->huggingface_hub[cli]>=0.25)
  Downloading hf_xet-1.5.1-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (4.9 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub>=0.25->huggingface_hub[cli]>=0.25) (4.16.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from trl>=0.12) (3.1.6)
Requirement already satisfied: psutil in /usr/local/lib/python3.12/dist-packages (from peft>=0.13) (7.2.2)
Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.12/dist-packages (from peft>=0.13) (2.7.1+cu128)
Collecting pyarrow>=21.0.0 (from datasets>=2.20)
  Downloading pyarrow-25.0.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Collecting dill<0.4.2,>=0.3.0 (from datasets>=2.20)
  Downloading dill-0.4.1-py3-none-any.whl.metadata (10 kB)
Collecting pandas (from datasets>=2.20)
  Downloading pandas-3.0.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (79 kB)
Requirement already satisfied: httpx<1.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets>=2.20) (0.28.1)
Collecting xxhash (from datasets>=2.20)
  Downloading xxhash-3.8.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (15 kB)
Collecting multiprocess<0.70.20 (from datasets>=2.20)
  Downloading multiprocess-0.70.19-py312-none-any.whl.metadata (7.5 kB)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1 (from fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading aiohttp-3.14.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (8.3 kB)
Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets>=2.20) (4.14.1)
Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets>=2.20) (2026.6.17)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets>=2.20) (1.0.9)
Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets>=2.20) (3.18)
Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1.0.0->datasets>=2.20) (0.16.0)
Collecting click>=8.2.0 (from wandb>=0.18)
  Downloading click-8.4.2-py3-none-any.whl.metadata (2.6 kB)
Collecting gitpython!=3.1.29,>=1.0.0 (from wandb>=0.18)
  Downloading gitpython-3.1.51-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: platformdirs in /usr/local/lib/python3.12/dist-packages (from wandb>=0.18) (4.10.0)
Collecting pydantic<3,>=2.6 (from wandb>=0.18)
  Downloading pydantic-2.13.4-py3-none-any.whl.metadata (109 kB)
Collecting sentry-sdk>=2.0.0 (from wandb>=0.18)
  Downloading sentry_sdk-2.65.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests) (3.4.9)
Requirement already satisfied: urllib3<3,>=1.26 in /usr/local/lib/python3.12/dist-packages (from requests) (2.7.0)
Collecting annotated-types>=0.6.0 (from pydantic<3,>=2.6->wandb>=0.18)
  Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.46.4 (from pydantic<3,>=2.6->wandb>=0.18)
  Downloading pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting typing-inspection>=0.4.2 (from pydantic<3,>=2.6->wandb>=0.18)
  Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)
Collecting markdown-it-py>=2.2.0 (from rich)
  Downloading markdown_it_py-4.2.0-py3-none-any.whl.metadata (7.4 kB)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich) (2.20.0)
Collecting shellingham>=1.3.0 (from typer)
  Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting annotated-doc>=0.0.2 (from typer)
  Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)
Collecting aiohappyeyeballs>=2.5.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading aiohappyeyeballs-2.7.1-py3-none-any.whl.metadata (5.9 kB)
Collecting aiosignal>=1.4.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20) (26.1.0)
Collecting frozenlist>=1.1.1 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (20 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (5.3 kB)
Collecting propcache>=0.2.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading propcache-0.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (16 kB)
Collecting yarl<2.0,>=1.17.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.4.0,>=2023.1.0->datasets>=2.20)
  Downloading yarl-1.24.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (94 kB)
Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.29,>=1.0.0->wandb>=0.18)
  Downloading gitdb-4.0.12-py3-none-any.whl.metadata (1.2 kB)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.29,>=1.0.0->wandb>=0.18)
  Downloading smmap-5.0.3-py3-none-any.whl.metadata (4.6 kB)
INFO: pip is looking at multiple versions of huggingface-hub[cli] to determine which version is compatible with other requirements. This could take a while.
Collecting huggingface_hub[cli]>=0.25
  Downloading huggingface_hub-1.22.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.21.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.20.1-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.20.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.19.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.18.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.17.0-py3-none-any.whl.metadata (14 kB)
INFO: pip is still looking at multiple versions of huggingface-hub[cli] to determine which version is compatible with other requirements. This could take a while.
  Downloading huggingface_hub-1.16.4-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.16.1-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.16.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.15.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.14.0-py3-none-any.whl.metadata (14 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading huggingface_hub-1.13.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.12.2-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.12.1-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.12.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.11.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.10.2-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.10.1-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.10.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.9.2-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.9.1-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.9.0-py3-none-any.whl.metadata (14 kB)
  Downloading huggingface_hub-1.8.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.7.2-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.7.1-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.7.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.6.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.5.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.4.1-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.4.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.7-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.5-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.4-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.3-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.2-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.1-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.3.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.2.4-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.2.3-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.2.2-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.2.1-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.2.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.7-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.6-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.5-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.4-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.3-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.2-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.1-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.1.0-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.0.1-py3-none-any.whl.metadata (13 kB)
  Downloading huggingface_hub-1.0.0-py3-none-any.whl.metadata (13 kB)
Collecting InquirerPy==0.3.4 (from huggingface_hub[cli]>=0.25)
  Downloading InquirerPy-0.3.4-py3-none-any.whl.metadata (8.1 kB)
Collecting pfzy<0.4.0,>=0.3.1 (from InquirerPy==0.3.4->huggingface_hub[cli]>=0.25)
  Downloading pfzy-0.3.4-py3-none-any.whl.metadata (4.9 kB)
Requirement already satisfied: prompt-toolkit<4.0.0,>=3.0.1 in /usr/local/lib/python3.12/dist-packages (from InquirerPy==0.3.4->huggingface_hub[cli]>=0.25) (3.0.52)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from prompt-toolkit<4.0.0,>=3.0.1->InquirerPy==0.3.4->huggingface_hub[cli]>=0.25) (0.8.2)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich)
  Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (83.0.0)
Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (1.14.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (3.6.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.61 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.8.61)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.57 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.8.57)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.57 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.8.57)
Requirement already satisfied: nvidia-cudnn-cu12==9.7.1.26 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (9.7.1.26)
Requirement already satisfied: nvidia-cublas-cu12==12.8.3.14 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.8.3.14)
Requirement already satisfied: nvidia-cufft-cu12==11.3.3.41 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (11.3.3.41)
Requirement already satisfied: nvidia-curand-cu12==10.3.9.55 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (10.3.9.55)
Requirement already satisfied: nvidia-cusolver-cu12==11.7.2.55 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (11.7.2.55)
Requirement already satisfied: nvidia-cusparse-cu12==12.5.7.53 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.5.7.53)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (0.6.3)
Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (2.26.2)
Requirement already satisfied: nvidia-nvtx-cu12==12.8.55 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.8.55)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.61 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (12.8.61)
Requirement already satisfied: nvidia-cufile-cu12==1.13.0.11 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (1.13.0.11)
Requirement already satisfied: triton==3.3.1 in /usr/local/lib/python3.12/dist-packages (from torch>=1.13.0->peft>=0.13) (3.3.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch>=1.13.0->peft>=0.13) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->trl>=0.12) (3.0.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets>=2.20) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas->datasets>=2.20) (1.16.0)
Downloading transformers-4.57.6-py3-none-any.whl (12.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 171.0 MB/s  0:00:00
Downloading tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 256.2 MB/s  0:00:00
Downloading protobuf-5.29.6-cp38-abi3-manylinux2014_x86_64.whl (320 kB)
Downloading huggingface_hub-0.36.2-py3-none-any.whl (566 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 566.4/566.4 kB 271.8 MB/s  0:00:00
Downloading hf_xet-1.5.1-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 290.0 MB/s  0:00:00
Downloading trl-1.8.0-py3-none-any.whl (863 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 863.2/863.2 kB 528.5 MB/s  0:00:00
Downloading peft-0.19.1-py3-none-any.whl (680 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 680.7/680.7 kB 537.1 MB/s  0:00:00
Downloading accelerate-1.14.0-py3-none-any.whl (389 kB)
Downloading datasets-5.0.0-py3-none-any.whl (555 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 555.1/555.1 kB 452.6 MB/s  0:00:00
Downloading dill-0.4.1-py3-none-any.whl (120 kB)
Downloading multiprocess-0.70.19-py312-none-any.whl (150 kB)
Downloading wandb-0.28.0-py3-none-manylinux_2_28_x86_64.whl (26.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.8/26.8 MB 181.7 MB/s  0:00:00
Downloading pydantic-2.13.4-py3-none-any.whl (472 kB)
Downloading pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 272.6 MB/s  0:00:00
Downloading tqdm-4.68.4-py3-none-any.whl (676 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 676.6/676.6 kB 390.9 MB/s  0:00:00
Downloading python_dotenv-1.2.2-py3-none-any.whl (22 kB)
Downloading rich-15.0.0-py3-none-any.whl (310 kB)
Downloading typer-0.26.8-py3-none-any.whl (122 kB)
Downloading zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (5.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 239.2 MB/s  0:00:00
Downloading sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 256.7 MB/s  0:00:00
Downloading aiohttp-3.14.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 268.6 MB/s  0:00:00
Downloading multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (256 kB)
Downloading yarl-1.24.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (105 kB)
Downloading aiohappyeyeballs-2.7.1-py3-none-any.whl (15 kB)
Downloading aiosignal-1.4.0-py3-none-any.whl (7.5 kB)
Downloading annotated_doc-0.0.4-py3-none-any.whl (5.3 kB)
Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Downloading click-8.4.2-py3-none-any.whl (119 kB)
Downloading frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (242 kB)
Downloading gitpython-3.1.51-py3-none-any.whl (215 kB)
Downloading gitdb-4.0.12-py3-none-any.whl (62 kB)
Downloading smmap-5.0.3-py3-none-any.whl (24 kB)
Downloading InquirerPy-0.3.4-py3-none-any.whl (67 kB)
Downloading pfzy-0.3.4-py3-none-any.whl (8.5 kB)
Downloading markdown_it_py-4.2.0-py3-none-any.whl (91 kB)
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Downloading propcache-0.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (61 kB)
Downloading pyarrow-25.0.0-cp312-cp312-manylinux_2_28_x86_64.whl (50.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.1/50.1 MB 157.5 MB/s  0:00:00
Downloading regex-2026.7.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (801 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 801.3/801.3 kB 477.2 MB/s  0:00:00
Downloading safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (516 kB)
Downloading sentry_sdk-2.65.0-py3-none-any.whl (503 kB)
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading typing_inspection-0.4.2-py3-none-any.whl (14 kB)
Downloading pandas-3.0.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (10.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 133.6 MB/s  0:00:00
Downloading xxhash-3.8.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (220 kB)
Installing collected packages: zstandard, xxhash, typing-inspection, tqdm, smmap, shellingham, sentry-sdk, sentencepiece, safetensors, regex, python-dotenv, pydantic-core, pyarrow, protobuf, propcache, pfzy, multidict, mdurl, hf-xet, frozenlist, dill, click, annotated-types, annotated-doc, aiohappyeyeballs, yarl, pydantic, pandas, multiprocess, markdown-it-py, InquirerPy, huggingface_hub, gitdb, aiosignal, tokenizers, rich, gitpython, aiohttp, wandb, typer, transformers, accelerate, peft, datasets, trl

Successfully installed InquirerPy-0.3.4 accelerate-1.14.0 aiohappyeyeballs-2.7.1 aiohttp-3.14.1 aiosignal-1.4.0 annotated-doc-0.0.4 annotated-types-0.7.0 click-8.4.2 datasets-5.0.0 dill-0.4.1 frozenlist-1.8.0 gitdb-4.0.12 gitpython-3.1.51 hf-xet-1.5.1 huggingface_hub-0.36.2 markdown-it-py-4.2.0 mdurl-0.1.2 multidict-6.7.1 multiprocess-0.70.19 pandas-3.0.3 peft-0.19.1 pfzy-0.3.4 propcache-0.5.2 protobuf-5.29.6 pyarrow-25.0.0 pydantic-2.13.4 pydantic-core-2.46.4 python-dotenv-1.2.2 regex-2026.7.10 rich-15.0.0 safetensors-0.8.0 sentencepiece-0.2.2 sentry-sdk-2.65.0 shellingham-1.5.4 smmap-5.0.3 tokenizers-0.22.2 tqdm-4.68.4 transformers-4.57.6 trl-1.8.0 typer-0.26.8 typing-inspection-0.4.2 wandb-0.28.0 xxhash-3.8.1 yarl-1.24.2 zstandard-0.25.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

==> install flash-attention 2 (2-3x speedup on Qwen3)

Successfully installed einops-0.8.2 flash-attn-2.8.3.post1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

==> install aws-cli
aws-cli/2.35.23 Python/3.14.6 Linux/6.8.0-52-generic exe/x86_64.ubuntu.22

==> install tanglish (editable, --no-deps to skip repo pyproject deps)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

==> wandb login
Current Settings
{
  "_extra_http_headers": null,

==> S3-sync Tanglish training corpus (~7 MB compressed)
Completed 256.0 KiB/6.2 MiB (693.2 KiB/s) with 2 file(s) remaining
Completed 512.0 KiB/6.2 MiB (1.2 MiB/s) with 2 file(s) remaining  
Completed 768.0 KiB/6.2 MiB (1.8 MiB/s) with 2 file(s) remaining  
Completed 1.0 MiB/6.2 MiB (2.3 MiB/s) with 2 file(s) remaining    
Completed 1.1 MiB/6.2 MiB (2.5 MiB/s) with 2 file(s) remaining    
download: s3://bo2olk8uqw/pairs/tanglish_v1/tanglish_v1.val.jsonl.zst to data/pairs/tanglish_v1.val.jsonl.zst
Completed 1.1 MiB/6.2 MiB (2.5 MiB/s) with 1 file(s) remaining
Completed 1.3 MiB/6.2 MiB (3.0 MiB/s) with 1 file(s) remaining
Completed 1.6 MiB/6.2 MiB (3.5 MiB/s) with 1 file(s) remaining
Completed 1.8 MiB/6.2 MiB (3.9 MiB/s) with 1 file(s) remaining
Completed 2.1 MiB/6.2 MiB (4.4 MiB/s) with 1 file(s) remaining
Completed 2.3 MiB/6.2 MiB (4.8 MiB/s) with 1 file(s) remaining
Completed 2.6 MiB/6.2 MiB (5.2 MiB/s) with 1 file(s) remaining
Completed 2.8 MiB/6.2 MiB (5.7 MiB/s) with 1 file(s) remaining
Completed 3.1 MiB/6.2 MiB (6.1 MiB/s) with 1 file(s) remaining
Completed 3.3 MiB/6.2 MiB (6.5 MiB/s) with 1 file(s) remaining
Completed 3.6 MiB/6.2 MiB (6.9 MiB/s) with 1 file(s) remaining
Completed 3.8 MiB/6.2 MiB (7.4 MiB/s) with 1 file(s) remaining
Completed 4.1 MiB/6.2 MiB (7.7 MiB/s) with 1 file(s) remaining
Completed 4.3 MiB/6.2 MiB (8.2 MiB/s) with 1 file(s) remaining
Completed 4.6 MiB/6.2 MiB (8.6 MiB/s) with 1 file(s) remaining
Completed 4.8 MiB/6.2 MiB (9.0 MiB/s) with 1 file(s) remaining
Completed 5.1 MiB/6.2 MiB (9.5 MiB/s) with 1 file(s) remaining
Completed 5.3 MiB/6.2 MiB (9.9 MiB/s) with 1 file(s) remaining
Completed 5.6 MiB/6.2 MiB (10.3 MiB/s) with 1 file(s) remaining
Completed 5.8 MiB/6.2 MiB (10.7 MiB/s) with 1 file(s) remaining
Completed 6.1 MiB/6.2 MiB (11.1 MiB/s) with 1 file(s) remaining
Completed 6.2 MiB/6.2 MiB (11.2 MiB/s) with 1 file(s) remaining
download: s3://bo2olk8uqw/pairs/tanglish_v1/tanglish_v1.train.jsonl.zst to data/pairs/tanglish_v1.train.jsonl.zst

==> pre-download Qwen3-8B-Instruct base weights (~16 GB, cached to /workspace/hf)
Repository Not Found for url: https://huggingface.co/api/models/Qwen/Qwen3-8B-Instruct/revision/main.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authentication