Home > EulerForge > Tutorials > 23. Lab: Full MoE Pipeline (SFT → DPO → RM → PPO)

23. Lab: Full MoE Pipeline (SFT → DPO → RM → PPO)

Difficulty: Highest | Estimated Time: 24-48 hours on a single RTX 5090 | Path: SFT -> DPO -> RM -> PPO (full)

This lab is a full pipeline exercise that imbues a single model with the core capabilities of state-of-the-art 2025-2026 LLMs. It implements the model characteristics commonly pursued by DeepSeek-V3/R1's MoE + reasoning, GPT o-series' extended thinking, and Claude's structured tool use through 4-stage training.

1. Lab Objective

Final Model Profile: Korean/English bilingual + MoE expert specialization + structured reasoning + code generation

Capability Target Level Related SOTA Models
Structured reasoning (thinking) Output thought process via <think> tags GPT o1/o3, DeepSeek-R1
Code generation Function implementation + explanation Claude, GPT-4o
Math problem solving Step-by-step formula derivation DeepSeek-V3, Qwen-2.5-Math
Korean conversation Natural Korean responses Existing KoAlpaca data
MoE specialization Per-expert domain specialization DeepSeek-V3, Mixtral
Preference alignment Prefer accurate and safe responses Full RLHF path (DPO -> RM -> PPO)

2. Full Pipeline

Phase 1: Data Collection + Conversion (1-2 hours)
    │
Phase 2: SFT — Multi-Domain Foundation (6-8 hours)
    │   ├── Korean conversation (KoAlpaca)
    │   ├── Math (GSM8K + MATH)
    │   ├── Coding (CodeAlpaca + CodeFeedback)
    │   └── Thinking patterns (rule conversion + LLM generation)
    │
Phase 3: DPO — Multi-Domain Preference Alignment (4-6 hours)
    │   ├── Math correct/incorrect preference
    │   ├── Thinking/Non-thinking preference
    │   └── Korean response quality preference
    │
Phase 4: RM — Reward Model Training (3-4 hours)
    │
Phase 5: PPO — RLHF Policy Optimization (4-6 hours)
    │
Phase 6: Bench — Original vs Final Model Comparison

3. Data Collection Guide

Common Data Collection Guide: Refer to 19_data_collection.md for collection scripts and size adjustment methods for all datasets (SFT/DPO, Korean/English, math/coding). Below is a mixing guide specific to this lab.

3.1 SFT Data -- 4-Domain Mix

Domain Dataset Scale Source
Korean conversation KoAlpaca 10K data/sft_10k_raw.jsonl (already available)
Math GSM8K 8.5K HuggingFace gsm8k
Math (advanced) MATH 7.5K HuggingFace hendrycks/competition_math
Coding CodeAlpaca 10K HuggingFace sahil2801/CodeAlpaca-20k
Thinking patterns Converted from above data 5-10K Rule/LLM generation

Total SFT data: ~40K rows (shuffled after mixing)

# -- 3.1.1 Korean: already available
cp data/sft_10k_raw.jsonl data/pipeline/ko_sft.jsonl

# -- 3.1.2 Math (GSM8K)
python -c "
from datasets import load_dataset
import json

ds = load_dataset('gsm8k', 'main', split='train')
with open('data/pipeline/math_gsm8k_sft.jsonl', 'w') as f:
    for row in ds:
        prompt = f'Solve the following math problem step by step.\n\nProblem: {row[\"question\"]}'
        response = row['answer']
        f.write(json.dumps({'prompt': prompt, 'response': response}, ensure_ascii=False) + '\n')
print(f'GSM8K: {len(ds)} rows')
"

# -- 3.1.3 Math advanced (MATH)
python -c "
from datasets import load_dataset
import json

ds = load_dataset('hendrycks/competition_math', split='train')
with open('data/pipeline/math_competition_sft.jsonl', 'w') as f:
    count = 0
    for row in ds:
        prompt = f'Solve this math problem. Show your work.\n\n{row[\"problem\"]}'
        response = row['solution']
        if prompt and response:
            f.write(json.dumps({'prompt': prompt, 'response': response}, ensure_ascii=False) + '\n')
            count += 1
print(f'MATH: {count} rows')
"

# -- 3.1.4 Coding
python -c "
from datasets import load_dataset
import json

ds = load_dataset('sahil2801/CodeAlpaca-20k', split='train')
with open('data/pipeline/code_sft.jsonl', 'w') as f:
    count = 0
    for row in ds:
        prompt = row.get('instruction', '')
        inp = row.get('input', '')
        if inp:
            prompt = f'{prompt}\n\nInput: {inp}'
        response = row.get('output', '')
        if prompt and response:
            f.write(json.dumps({'prompt': prompt, 'response': response}, ensure_ascii=False) + '\n')
            count += 1
            if count >= 10000:
                break
print(f'Code: {count} rows')
"

# -- 3.1.5 Thinking pattern conversion
python -c "
import json

# Convert GSM8K step-by-step into <think> format
with open('data/pipeline/math_gsm8k_sft.jsonl') as f_in, \
     open('data/pipeline/thinking_sft.jsonl', 'w') as f_out:
    count = 0
    for line in f_in:
        row = json.loads(line)
        answer = row['response']
        parts = answer.rsplit('####', 1)
        if len(parts) == 2:
            reasoning = parts[0].strip()
            final = parts[1].strip()
            response = f'<think>\nLet me work through this step by step.\n{reasoning}\n</think>\nThe answer is {final}.'
        else:
            response = f'<think>\n{answer}\n</think>\n{answer}'
        f_out.write(json.dumps({
            'prompt': row['prompt'].replace('step by step', 'step by step. Show your reasoning in <think> tags'),
            'response': response,
        }, ensure_ascii=False) + '\n')
        count += 1
print(f'Thinking: {count} rows')
"

# -- 3.1.6 Merge + Shuffle
cat data/pipeline/ko_sft.jsonl \
    data/pipeline/math_gsm8k_sft.jsonl \
    data/pipeline/math_competition_sft.jsonl \
    data/pipeline/code_sft.jsonl \
    data/pipeline/thinking_sft.jsonl \
    | shuf --random-source=<(yes 42) > data/pipeline/all_sft_raw.jsonl

wc -l data/pipeline/all_sft_raw.jsonl
# Expected: ~40K rows

3.2 DPO Data -- 3 Preference Axes

mkdir -p data/pipeline

# -- Math correct/incorrect preference
python -c "
from datasets import load_dataset
import json, random
random.seed(42)

ds = load_dataset('gsm8k', 'main', split='train')
with open('data/pipeline/math_dpo.jsonl', 'w') as f:
    for row in ds:
        prompt = f'Solve: {row[\"question\"]}'
        chosen = row['answer']
        parts = chosen.rsplit('####', 1)
        if len(parts) == 2:
            wrong = random.randint(1, 9999)
            rejected = parts[0] + f'#### {wrong}'
        else:
            rejected = 'I am not sure about this problem.'
        f.write(json.dumps({
            'prompt': prompt, 'chosen': chosen, 'rejected': rejected
        }, ensure_ascii=False) + '\n')
print(f'Math DPO: {len(ds)} rows')
"

# -- Korean preference (already available)
cp data/dpo_10k_raw.jsonl data/pipeline/ko_dpo.jsonl

# -- Thinking preference (with thinking > without thinking)
python -c "
import json

with open('data/pipeline/thinking_sft.jsonl') as f_think, \
     open('data/pipeline/math_gsm8k_sft.jsonl') as f_plain, \
     open('data/pipeline/thinking_dpo.jsonl', 'w') as f_out:
    think_data = {json.loads(l)['prompt']: json.loads(l)['response'] for l in f_think}
    count = 0
    for line in f_plain:
        row = json.loads(line)
        # Find if a thinking version exists
        for tp, tr in think_data.items():
            if row['prompt'][:50] in tp:
                f_out.write(json.dumps({
                    'prompt': tp,
                    'chosen': tr,
                    'rejected': row['response'],
                }, ensure_ascii=False) + '\n')
                count += 1
                break
        if count >= 5000:
            break
print(f'Thinking DPO: {count} rows')
"

# -- Merge
cat data/pipeline/math_dpo.jsonl \
    data/pipeline/ko_dpo.jsonl \
    data/pipeline/thinking_dpo.jsonl \
    | shuf --random-source=<(yes 42) > data/pipeline/all_dpo_raw.jsonl

wc -l data/pipeline/all_dpo_raw.jsonl

3.3 RM/PPO Data

RM reuses the DPO data, and PPO only requires prompts.

# RM: Use DPO data directly
cp data/pipeline/all_dpo_raw.jsonl data/pipeline/rm_raw.jsonl

# PPO: Extract prompts only
python -c "
import json
seen = set()
with open('data/pipeline/all_sft_raw.jsonl') as f_in, \
     open('data/pipeline/ppo_prompts_raw.jsonl', 'w') as f_out:
    count = 0
    for line in f_in:
        row = json.loads(line)
        p = row['prompt'][:200]
        if p not in seen:
            seen.add(p)
            f_out.write(json.dumps({'prompt': row['prompt']}, ensure_ascii=False) + '\n')
            count += 1
        if count >= 5000:
            break
print(f'PPO prompts: {count}')
"

4. Phase 2: MoE SFT -- Multi-Domain Foundation

Strategy: moe_expert_lora (4 experts) -- expecting per-expert domain specialization

eulerforge train \
    --preset configs/presets/qwen3.5_0.8b_moe_expert_lora_sft.yml \
    --set data.format=raw \
    --set data.task=sft \
    --set data.path=data/pipeline/all_sft_raw.jsonl \
    --set data.max_length=1024 \
    --set training.max_train_steps=15000 \
    --set training.batch_size=2 \
    --set training.grad_accum_steps=8 \
    --set training.log_steps=100 \
    --set training.save_steps=3000 \
    --output-dir outputs/pipeline/01_sft

Estimated time: ~6-8 hours (RTX 5090, ~40K data, 15K micro-steps)

3-Phase Schedule: - Phase 0 (step 0-2000): Router warmup -- learn expert routing patterns - Phase 1 (step 2000+): LoRA training -- per-domain expert specialization

Checkpoints: - In metrics.jsonl, loss should decrease steeply until step 5000 - Converge below 2.0 after step 10000 - Check expert utilization balance via MoE routing stats (advanced metrics)

5. Phase 3: Preference Alignment -- DPO or ORPO

Option A: DPO (uses reference model)

eulerforge train \
    --preset configs/presets/qwen3.5_0.8b_moe_expert_lora_dpo.yml \
    --set model_name=outputs/pipeline/01_sft/final \
    --set data.format=raw \
    --set data.task=prompted_preference \
    --set data.path=data/pipeline/all_dpo_raw.jsonl \
    --set data.max_length=1024 \
    --set training.max_train_steps=6000 \
    --set training.batch_size=2 \
    --set training.grad_accum_steps=8 \
    --output-dir outputs/pipeline/02_dpo

Advantages over DPO: 50% memory savings (1x forward), Handoff compatible, combines SFT+preference Disadvantages vs DPO: Without reference comparison, preference boundaries may be slightly less distinct

eulerforge train \
    --preset configs/presets/qwen3.5_0.8b_dense_lora_orpo.yml \
    --set model_name=outputs/pipeline/01_sft/final \
    --set injection.strategy=moe_expert_lora \
    --set injection.num_experts=4 --set injection.top_k=2 \
    --set data.format=raw \
    --set data.task=preference \
    --set data.path=data/pipeline/all_dpo_raw.jsonl \
    --set data.max_length=1024 \
    --set training.max_train_steps=6000 \
    --output-dir outputs/pipeline/02_orpo

Estimated time: ~4-6 hours (DPO), ~3-4 hours (ORPO)

DPO Checkpoints: - Phase 0 (router): reward_margin=0, loss=0.6931 (normal -- policy=reference when LoRA is frozen) - Phase 1 (LoRA): reward_margin gradually positive, accuracy > 0.5

ORPO Checkpoints: - Phase 0: sft_loss starts decreasing (ORPO includes SFT loss, so it is effective even during router phase) - Phase 1: orpo_loss decreases + log_odds_ratio increases positively

6. Phase 4: RM -- Reward Model Training

eulerforge train \
    --preset configs/presets/qwen3.5_0.8b_dense_lora_rm.yml \
    --set model_name=outputs/pipeline/01_sft/final \
    --set data.format=raw \
    --set data.task=preference \
    --set data.path=data/pipeline/rm_raw.jsonl \
    --set data.max_length=1024 \
    --set training.max_train_steps=6000 \
    --output-dir outputs/pipeline/03_rm

Estimated time: ~3-4 hours

Note: dense_lora strategy is recommended for RM -- MoE routing instability can negatively affect reward prediction.

Checkpoints: - RM accuracy > 0.65 (rate at which chosen scores higher than rejected) - Verify that reward_head.pt is saved in final/

7. Phase 5: PPO -- RLHF Policy Optimization

eulerforge train \
    --preset configs/presets/qwen3.5_0.8b_dense_lora_ppo.yml \
    --set model_name=outputs/pipeline/02_dpo/final \
    --set training.reward_model.checkpoint_path=outputs/pipeline/03_rm/final \
    --set data.format=raw \
    --set data.task=sft \
    --set data.path=data/pipeline/ppo_prompts_raw.jsonl \
    --set data.max_length=512 \
    --set training.max_train_steps=3000 \
    --output-dir outputs/pipeline/04_ppo

Estimated time: ~4-6 hours

Checkpoints: - reward_mean gradually increases - Verify that KL divergence is not too large (prevent excessive reward hacking)

8. Phase 6: Benchmark -- 4-Stage Comparison

8.1 Math Ability Comparison

# configs/bench/pipeline_math.yml
bench:
  task: sft
  data_path: data/pipeline/math_gsm8k_sft.jsonl
  sample:
    k: 20
    seed: 42
  generation:
    max_new_tokens: 512
    temperature: 0.1
  models:
    target:
      model_dir: "outputs/pipeline/04_ppo/final"
      device: "cuda:0"
    baseline:
      enabled: true
      provider: ollama
      model: "qwen3.5:0.8b"
    judge:
      enabled: true
      provider: ollama
      model: "gpt-oss:20b"
      mode: pointwise
  output:
    out_dir: outputs/bench_pipeline_math
    save_jsonl: true
    print_examples: true

eulerforge bench --preset configs/bench/pipeline_math.yml

8.2 Korean Conversation Comparison

eulerforge bench --preset configs/bench/pipeline_math.yml \
    --set bench.data_path=data/sft_1k_bench_raw.jsonl \
    --set bench.generation.temperature=0.7 \
    --set bench.generation.max_new_tokens=400 \
    --set bench.output.out_dir=outputs/bench_pipeline_ko

8.3 Stage-by-Stage Comparison (SFT -> DPO -> PPO Evolution Tracking)

# SFT model
eulerforge bench --preset configs/bench/pipeline_math.yml \
    --set bench.models.target.model_dir=outputs/pipeline/01_sft/final \
    --set bench.output.out_dir=outputs/bench_pipeline_sft

# DPO model
eulerforge bench --preset configs/bench/pipeline_math.yml \
    --set bench.models.target.model_dir=outputs/pipeline/02_dpo/final \
    --set bench.output.out_dir=outputs/bench_pipeline_dpo

# PPO model (final)
eulerforge bench --preset configs/bench/pipeline_math.yml \
    --set bench.models.target.model_dir=outputs/pipeline/04_ppo/final \
    --set bench.output.out_dir=outputs/bench_pipeline_ppo

9. Expected Results Analysis

9.1 Capability Evolution by Stage

Stage Math Accuracy Thinking Rate Korean Quality Preference Alignment
Original (Qwen3.5-0.8B) Low 0% Basic None
SFT Medium ~30% Good None
DPO Medium-High ~60% Good Initial
PPO High ~70%+ Good Reinforced

9.2 Verifying MoE Expert Specialization

Training with --metrics-level advanced allows you to check expert routing statistics:

[MoE Routing] Layer 5: expert_0=32%, expert_1=28%, expert_2=22%, expert_3=18%

Ideally: - Expert 0: Math/reasoning (high rate for numeric-heavy inputs) - Expert 1: Coding (high rate for code tokens) - Expert 2: Korean (high rate for Korean tokens) - Expert 3: General/English

Whether this level of specialization naturally emerges depends on the data mixing ratio and training duration. Observing and analyzing this is itself the core learning point of this lab.

10. Advanced Variant Experiments

10.1 LoRA Handoff

Applying LoRA Handoff during the SFT stage transfers knowledge from LoRA to the base FFN in the later phase:

training:
  lora_handoff:
    expert_lora:
      start_step: 8000
      duration_steps: 5000
      end_scale: 0.0
      curve: cosine
      end_action: freeze
    base_ffn_ramp:
      start_step: 8000
      end_step: 13000
      start_multiplier: 1.0
      end_multiplier: 3.0

10.2 Strategy Comparison: dense_lora vs moe_expert_lora

Compare by changing only the strategy on the same data: - dense_lora: Simple LoRA, all parameters as a single expert - moe_expert_lora: 4 experts, per-domain specialization possible

10.3 Data Ratio Experiments

Explore the optimal ratio by varying domain mixing proportions: - Math heavy: 50% math + 50% rest - Coding heavy: 50% coding + 50% rest - Balanced: 25% each

11. Checkpoint Chain Summary

data/pipeline/
├── all_sft_raw.jsonl           # ~40K (4-domain mix)
├── all_dpo_raw.jsonl           # ~23K (3 preference axes)
├── rm_raw.jsonl                # = all_dpo_raw.jsonl
└── ppo_prompts_raw.jsonl       # ~5K (prompts only)

outputs/pipeline/
├── 01_sft/final/               # MoE SFT completed model
├── 02_dpo/final/               # MoE DPO aligned model
├── 03_rm/final/                # Reward Model + reward_head.pt
│   └── reward_head.pt
├── 04_ppo/final/               # Final RLHF model ★
└── bench_pipeline_*/           # Benchmark results for each stage

12. What You Will Learn from This Lab

Learning Point Content
Data engineering Multi-domain data collection/conversion/mixing/shuffling
Significance of training order How each stage (SFT -> DPO -> RM -> PPO) changes the model
Understanding MoE behavior Expert routing, specialization, load balance
Phase scheduling The rationale behind router warmup -> LoRA -> full unfreeze order
Limitations of preference alignment Whether DPO alone is sufficient; the value PPO adds
Benchmark interpretation The meaning and limitations of pointwise scores
Hands-on with latest trends Thinking patterns, MoE specialization, progressive alignment
← Prev 22. Lab: Korean Finance Copilot