17. 스크래치 사전훈련

EulerStack (모델 조립)          EulerForge (훈련)
──────────────────          ──────────────────
YAML 스펙
  ↓ compile + export
HF model_dir ──────────────→ eulerforge pretrain
  (config.json +                 ↓
   model.safetensors +       사전훈련 완료
   modeling_*.py)                ↓
                             final/ (HF save_pretrained)
                                 ↓
                             eulerforge train (LoRA 파인튜닝)

/path/to/eulerstack/outputs/full_hybrid_moe/
├── config.json                    # HF config (model_type: eulerstack)
├── configuration_eulerstack.py    # 커스텀 config 클래스
├── modeling_eulerstack.py         # 커스텀 모델 클래스
├── model.safetensors              # 모델 가중치
└── generation_config.json

{"text": "LABOR'S MARTYRS\n\nHaymarket\n1887\n\nSacco and Vanzetti..."}
{"text": "THROUGH THE WALL\n\nBY CLEVELAND MOFFETT..."}

"tokenizer": {"type": "hf", "pretrained": "gpt2", "add_bos": true, "add_eos": true}

# ── 디바이스 ──
device: "cuda:0"             # cuda:0, cuda:1, cpu

# ── 모델 ──
model_dir: "outputs/full_hybrid_moe"
trust_remote_code: true

# ── 토크나이저 ──
tokenizer: "gpt2"

# ── 데이터 ──
data:
  path: "data/dolma_10k.jsonl"
  text_column: "text"
  max_length: 1024
  packing: true              # packed chunking

# ── 훈련 ──
training:
  max_steps: 500
  batch_size: 2
  grad_accum_steps: 4        # 유효 배치 = 2 × 4 = 8
  lr: 3.0e-4
  weight_decay: 0.1
  warmup_steps: 50
  max_grad_norm: 1.0
  log_steps: 10
  save_steps: 250
  dtype: "float32"           # Hybrid 모델(Hyena/RetNet FFT)은 float32 필수
  amp: false                 # FFT 연산이 bf16 미지원 → AMP 비활성화
  seed: 42

eulerforge pretrain --preset configs/presets/pretrain/eulerstack_hybrid_moe.yml --validate-only

[Pretrain] Config validation passed.
[Pretrain] model_dir: /path/to/eulerstack/outputs/full_hybrid_moe
[Pretrain] data.path: data/dolma_10k.jsonl
[Pretrain] training.max_steps: 500

eulerforge pretrain --preset configs/presets/pretrain/eulerstack_hybrid_moe.yml

[Pretrain] Device: cuda
[Pretrain] Loading model from /path/to/eulerstack/outputs/full_hybrid_moe
[Pretrain] Model loaded: 1,932,345,344 params, 1,932,345,344 trainable, dtype=torch.bfloat16
[Pretrain] Tokenizer: gpt2 (vocab=50257)
[Pretrain] Loaded 10000 texts from data/dolma_10k.jsonl
[Pretrain] Packed: 4,521,600 tokens → 2,207 chunks (max_length=2048)
[Pretrain] Starting training: max_steps=500, batch_size=2, grad_accum=4, lr=0.0003, warmup=50
[Pretrain] step=10/500 | loss=10.4523 | lr=6.00e-05 | 0.42 steps/s
[Pretrain] step=20/500 | loss=9.8712 | lr=1.20e-04 | 0.43 steps/s
...
[Pretrain] step=250/500 | loss=6.1234 | lr=2.85e-04 | 0.44 steps/s
[Pretrain] Checkpoint saved: outputs/pretrain_20260327_143000/checkpoint-250
...
[Pretrain] step=500/500 | loss=5.4321 | lr=0.00e+00 | 0.44 steps/s
[Pretrain] Training complete. 500 steps in 1187.3s. Final model saved to outputs/pretrain_20260327_143000/final

# 학습률 변경
eulerforge pretrain --preset ... --set training.lr=1e-4

# 더 많은 스텝
eulerforge pretrain --preset ... --set training.max_steps=2000

# 출력 디렉토리 지정
eulerforge pretrain --preset ... --output-dir outputs/my_pretrain

# 여러 오버라이드 조합
eulerforge pretrain --preset ... \
  --set training.max_steps=1000 \
  --set training.lr=1e-4 \
  --set data.max_length=1024

outputs/pretrain_20260327_143000/
├── pretrain_config.json         # 사용된 설정 스냅샷
├── metrics.jsonl                # 스텝별 loss, lr, 경과 시간
├── checkpoint-250/              # 중간 체크포인트
│   ├── config.json
│   ├── model.safetensors
│   ├── tokenizer.json
│   └── tokenizer_config.json
└── final/                       # 최종 모델
    ├── config.json
    ├── model.safetensors
    ├── tokenizer.json
    └── tokenizer_config.json

{"step": 10, "loss": 10.4523, "lr": 6e-05, "elapsed_sec": 23.8}
{"step": 20, "loss": 9.8712, "lr": 0.00012, "elapsed_sec": 47.1}

텍스트 1: "Hello world. This is a test."
텍스트 2: "Another document with more content."
텍스트 3: "Third piece of text..."
         ↓ 토큰화 + EOS 삽입 + 연결
[tok1, tok2, ..., EOS, tok1, tok2, ..., EOS, tok1, tok2, ..., EOS]
         ↓ max_length=2048 단위로 분할
청크 1: [tok1, tok2, ..., tok2048]    ← labels = input_ids (causal LM)
청크 2: [tok2049, tok2050, ..., tok4096]
청크 3: ...
(나머지 토큰이 max_length 미만이면 버림)

# configs/presets/finetune_after_pretrain.yml
device: cuda:0
backbone: qwen3              # 또는 적절한 백본
model_name: "outputs/pretrain_20260327_143000/final"

injection:
  strategy: dense_lora
  lora_r: 32
  lora_alpha: 64
  target_keywords: ["gate_proj", "up_proj", "down_proj"]
  attn_lora:
    enabled: true
    keywords: ["q_proj", "v_proj"]

training:
  type: sft
  phases:
    - step: 0
      trainable: ["lora", "attn_lora"]
  lr: 1.0e-5
  max_train_steps: 5000

eulerforge train --preset configs/presets/finetune_after_pretrain.yml \
  --set data.path=data/sft_10k_raw.jsonl \
  --set data.format=raw \
  --set data.task=sft

[Pretrain] Tokenizer vocab (50257) > model vocab (32000). Resizing model embeddings to 50257.