prompt-distillation

第8章 · Agent 的自我进化 · 配套项目 chapter8/prompt-distillation

项目说明

Prompt Distillation with Hugging Face TRL

This project demonstrates prompt distillation - a technique to distill knowledge from a thinking model with long prompts into a non-thinking model without prompts, making responses dramatically faster.

🎯 Main Goal

Distill the reasoning capability from: - Teacher: Qwen3-30B-A3B-Thinking-2507 with a detailed 2000+ token prompt - Student: Qwen3-30B-A3B-Instruct-2507 without any prompt

Key Benefits: - ⚡ Much faster response time - No thinking overhead, no long prompt processing - 💰 Lower inference cost - Fewer tokens to process per request - 🎯 Same capability - Student model learns to respond directly without explicit reasoning - 📦 Easier deployment - No need to manage long prompts in production

What is Prompt Distillation?

Prompt Distillation (also known as context distillation) is a training method that makes an LLM internalize a long and complex prompt into its parameters. In this experiment, we also remove the thinking overhead by distilling from a thinking model to a non-thinking model.

Example - Language Classification:

We want to internalize this detailed prompt:

"Classify the language of the provided text into these labels: ar, de, el, en, es, fr, hi, ru, tr, ur, vi, zh, ot. Use these rules: Devanagari script → hi, Greek script → el, Cyrillic script → ru..." (2000+ tokens)

Before distillation (Teacher with thinking + prompt):

System: <2000+ token detailed prompt>
User: 一生、バンドしてくれる？
Assistant: <thinking>Let me analyze the script... These are Han characters... Based on rule X...</thinking>ja
⏱️  Response time: ~2-3 seconds

After distillation (Student, no thinking, no prompt):

User: 一生、バンドしてくれる？
Assistant: ja
⏱️  Response time: ~0.1 seconds (20-30x faster!)

Methodology

The method involves two stages:

1. Data Generation (Teacher Model): A thinking model uses a detailed prompt to generate responses with explicit reasoning.

2. Teacher generates: response = thinking_model(long_prompt, query)

3. Student Training (Distillation): A non-thinking model is fine-tuned to predict responses directly without the prompt or thinking process.

4. Student learns: non_thinking_model(query) ≈ thinking_model(long_prompt, query)
5. Result: Fast, direct responses with internalized reasoning capability

Hyperparameters

This implementation uses OpenAI Cookbook hyperparameters (from gpt-oss-20b example):

Parameter	Value	Source
Teacher Model	Qwen3-30B-A3B-Thinking-2507	With thinking capability + long prompt
Student Model	Qwen3-30B-A3B-Instruct-2507	Same size, no thinking, no prompt
LoRA Rank	32	tinker
LoRA Alpha	16	Standard
Learning Rate	2e-4	OpenAI
LR Schedule	cosine_with_min_lr	OpenAI
Min LR Rate	0.1	OpenAI
Batch Size	4 per GPU	OpenAI
Gradient Accumulation	4 steps	OpenAI
Max Length	2048	OpenAI (student only needs short context)
Num Epochs	1	OpenAI
Temperature	0.15	tinker (data generation)
Warmup Ratio	0.03	OpenAI
Gradient Checkpointing	True	OpenAI

Key Design Choice: We use the same 30B model for both teacher and student. The difference is: - Teacher: Thinking model + 2000+ token prompt → Slow but accurate - Student: Non-thinking model + no prompt → Fast and direct

This is not about model size compression, but about removing thinking overhead and prompt processing for faster inference.

Dataset

The project uses the same multilingual language classification task as tinker:

• Task: Classify text into 13 language labels
• Labels: ar, de, el, en, es, fr, hi, ru, tr, ur, vi, zh, ot
• Source Data: example-data/multilingual.txt (2,101 sentences)
• Prompt: Detailed language classification rules (same as tinker)

Installation

Prerequisites

1. Install the required dependencies:

pip install -r requirements.txt

1. Setup Weights & Biases for training monitoring:

# Login to wandb (required for training progress tracking)
wandb login

# Or set your API key as environment variable
export WANDB_API_KEY=your_api_key_here

You can get your API key from https://wandb.ai/settings

System Requirements

• Python 3.10+
• PyTorch 2.0+
• CUDA 12.1+ (for GPU acceleration)
• GPU: H100 80GB (for 30B model) or any GPU with 24GB+ for smaller models
• Memory: ~70-75GB VRAM for 30B model with LoRA

Usage

Step 1: Generate Training Data

Generate prompt distillation data using the teacher model:

# Single instance (uses tensor parallelism across GPUs)
python create_data.py \
    --input_file ./example-data/multilingual.txt \
    --output_file ./data/prompt_distillation_lang.jsonl \
    --model_name Qwen/Qwen3-30B-A3B-Thinking-2507 \
    --temperature 0.15 \
    --tensor_parallel_size 4

# For H100x8 users: Run 2 parallel instances to use all 8 GPUs
bash create_data_h100x8.sh

Options: - --input_file: Path to input sentences (one per line) - --output_file: Where to save generated training data - --model_name: Teacher model (Qwen3-30B-A3B-Thinking-2507 for better accuracy) - --temperature: Sampling temperature (0.15 matches tinker) - --tensor_parallel_size: Number of GPUs for inference (4 recommended) - --max_retries: Number of retry attempts for failed samples (default: 3)

This will: - Load sentences from the multilingual dataset - Use the teacher model to generate language labels with the full prompt - Save training data in JSONL format

Output format:

{
  "messages": [
    {"role": "user", "content": "Text in some language"},
    {"role": "assistant", "content": "en"}
  ]
}

Step 2: Train the Student Model

Fine-tune the student model on the distilled data using TRL:

# Single GPU training (recommended - simpler and works reliably)
bash train_trl.sh

Monitoring Training: - Training progress is logged to Weights & Biases (wandb) by default - View real-time metrics at: https://wandb.ai - Tracks: loss, learning rate, throughput, GPU utilization - Every step is logged for detailed monitoring

To disable wandb logging:

python train_sft_trl.py --report_to none ...other args...

Step 3: Evaluate Your Model

After training, evaluate the distilled model's performance:

# Evaluate with defaults (uses all defaults)
python evaluate.py

# Quick evaluation on a subset
python evaluate.py --max_samples 100

# Save results to a file
python evaluate.py --output_file ./evaluation_results.json

# Custom model path
python evaluate.py --model_path ./models/my_custom_model

Defaults: - Model: ./models/prompt_distillation_trl - Base model: Qwen/Qwen3-30B-A3B-Instruct-2507 - Test file: ./example-data/multilingual.txt

Real-time Output Example:

Evaluating model...
================================================================================
✓ [   1/2100] Pred: ar | GT: ar | Acc: 1/1 (100.0%) | وقال، ماما، لقد عدت للمنزل.
✓ [   2/2100] Pred: ru | GT: ru | Acc: 2/2 (100.0%) | И той каза: Мамо, у дома съм.
✓ [   3/2100] Pred: de | GT: de | Acc: 3/3 (100.0%) | und er hat gesagt, Mama ich bin daheim.
✓ [   4/2100] Pred: el | GT: el | Acc: 4/4 (100.0%) | Και είπε, Μαμά, έφτασα στο σπίτι.
✓ [   5/2100] Pred: en | GT: en | Acc: 5/5 (100.0%) | And he said, Mama, I'm home.
✗ [   6/2100] Pred: es | GT: en | Acc: 5/6 ( 83.3%) | Y él dijo: Mamá, estoy en casa.
✓ [   7/2100] Pred: fr | GT: fr | Acc: 6/7 ( 85.7%) | Et il a dit, maman, je suis à la maison.
✓ [   8/2100] Pred: hi | GT: hi | Acc: 7/8 ( 87.5%) | और उसने कहा, माँ, मैं घर आया हूं।
✓ [   9/2100] Pred: ru | GT: ru | Acc: 8/9 ( 88.9%) | И он сказал: Мама, я дома.
...
✗ [2092/2100] Pred: de | GT: ot | Acc: 1994/2092 ( 95.3%) | Hola, mein Freund
✓ [2093/2100] Pred: ru | GT: ru | Acc: 1995/2093 ( 95.3%) | Привет, hello
✗ [2094/2100] Pred: vi | GT: ot | Acc: 1995/2094 ( 95.3%) | Xin chào, merci beaucoup
✗ [2095/2100] Pred: hi | GT: ot | Acc: 1995/2095 ( 95.2%) | नमस्ते, good morning
✓ [2096/2100] Pred: en | GT: en | Acc: 1996/2096 ( 95.2%) | ok
✓ [2097/2100] Pred: en | GT: en | Acc: 1997/2097 ( 95.2%) | yes
✓ [2098/2100] Pred: fr | GT: fr | Acc: 1998/2098 ( 95.2%) | bonjour
✓ [2099/2100] Pred: es | GT: es | Acc: 1999/2099 ( 95.2%) | hola
✗ [2100/2100] Pred: hi | GT: ot | Acc: 1999/2100 ( 95.2%) | namaste
================================================================================
Evaluation completed: 2100 samples processed

================================================================================
CONFUSION MATRIX
================================================================================

       ar  de  el  en  es  fr  hi  ot  ru  tr   ?  ur  vi  zh  | Total
  ------------------------------------------------------------------
ar |  141   .   .   .   .   .   .   .   .   .   .   5   .   .  |  146
de |    . 135   .   .   1   .   .   .   .   1   .   .   .   .  |  137
el |    .   . 144   .   .   .   3  10   .   .   .   .   .   .  |  157
en |    .   .   . 146   3   .   .   .   .   .   .   .   .   .  |  149
es |    .   3   .   . 133   .   .   .   .   .   .   .   .   .  |  136
fr |    .   .   .   1   . 139   .   .   .   3   .   .   .   .  |  143
hi |    .   .   .   .   .   . 171  39   .   .   .   .   .   1  |  211
ot |    .   1   .   4   .   .  14 169   1   .   .   .   1   3  |  193
ru |    .   .   .   .   .   .   .   . 279   .   .   .   .   .  |  279
tr |    .   .   .   .   .   .   .   .   . 132   .   .   .   .  |  132
?  |    .   .   .   .   .   .   .   1   .   1   .   .   3   .  |    5
ur |    .   .   .   .   .   .   1   .   .   .   . 133   .   .  |  134
vi |    .   .   .   .   .   .   .   .   .   .   .   . 134   .  |  134
zh |    .   .   .   .   .   .   .   1   .   .   .   .   . 143  |  144

================================================================================
PER-LANGUAGE ACCURACY
================================================================================
✗ unknown:   0.0% (   0/   5)
⚠️  hi:  81.0% ( 171/ 211)
⚠️  ot:  87.6% ( 169/ 193)
✓ el:  91.7% ( 144/ 157)
✓ ar:  96.6% ( 141/ 146)
✓ fr:  97.2% ( 139/ 143)
✓ es:  97.8% ( 133/ 136)
✓ en:  98.0% ( 146/ 149)
✓ de:  98.5% ( 135/ 137)
✓ ur:  99.3% ( 133/ 134)
✓ zh:  99.3% ( 143/ 144)
✓ ru: 100.0% ( 279/ 279)
✓ tr: 100.0% ( 132/ 132)
✓ vi: 100.0% ( 134/ 134)

================================================================================
MOST PROBLEMATIC LANGUAGES (Top 5)
================================================================================

1. Language: unknown - Accuracy: 0.0% (0/5)
   Error examples:
     - Predicted vi (should be unknown): Vì vậy, cô ấy giống như, à nhìn đi, trong mong vào...
     - Predicted vi (should be unknown): Thế là, à, tôi, ừ, dù sao, ừ, ừ, đây là ba, ừ, phi...
     - Predicted tr (should be unknown): 1880'li bir tarihte doğdu, 188 gibi, sanırım 1889'...

2. Language: hi - Accuracy: 81.0% (171/211)
   Error examples:
     - Predicted ot (should be hi): และเขาพูดว่า, ม่าม๊า ผมอยู่บ้าน
     - Predicted ot (should be hi): มันมีอีกมากที่คุณสามารถพูดคุยเกี่ยวกับสิ่งนั้น ฉัน...
     - Predicted ot (should be hi): และฉันก็แบบว่าตอบตกลงและมันก็เท่านั่น!

3. Language: ot - Accuracy: 87.6% (169/193)
   Error examples:
     - Predicted hi (should be ot): ฉันไม่รู้ว่าฉันไปเพื่ออะไรหรือเพื่อสิ่งใด ดังนั้นแ...
     - Predicted hi (should be ot): วันนี้เขาจะพูดคุยกับเราเกี่ยวกับ Third SS, U2 Quic...
     - Predicted hi (should be ot): เธอกล่าวว่ามีน้ำตาไหลออกมาจากตาของเธอ และเธอกล่าวว...

4. Language: el - Accuracy: 91.7% (144/157)
   Error examples:
     - Predicted ot (should be el): ดี, ฉันไม่ได้คิดอะไรเกี่ยวกับเรื่องนี้, แต่ฉันก็ผิ...
     - Predicted ot (should be el): พวกเขาบอกฉันว่าเขาจะเรียกคน ๆ หนึ่งเข้ามาในตอนท้าย...
     - Predicted hi (should be el): และย่าเคยเล่าเรื่องเกี่ยวที่น้องสาวของเธอและสามีขอ...

5. Language: ar - Accuracy: 96.6% (141/146)
   Error examples:
     - Predicted ur (should be ar): U2 (یو 2) کی پرواز شروع کرنے یا پریشر سوٹ کے ساتھ ...
     - Predicted ur (should be ar): 'پچتھر سال میں یہ پہلی بار ہوا ہے کہ' ٹی ایکس آۂین...
     - Predicted ur (should be ar): میرا مطلب یہ تھا کہ پوری بات.

================================================================================

============================================================
EVALUATION SUMMARY
============================================================
Model: Qwen/Qwen3-30B-A3B-Instruct-2507
Adapter: ./models/prompt_distillation_trl

Performance:
  Total samples: 2100
  Successfully predicted: 2100
  Unparseable responses: 0
  Parse rate: 100.00%

  Overall Accuracy: 95.19%
  Correct: 1999/2100

💡 The model responds directly without the 2000+ token prompt!

📁 Complete results saved to: ./evaluation_results.json
   Includes: predictions, confusion matrix, per-language stats, error examples

Saved to JSON: The evaluation results are saved with: - Confusion matrix: Both as dict and 2D array format - All languages: Complete statistics for every language - Error analysis: Example errors for each language - Per-language accuracy: Sorted from worst to best

Step 4: Quantify the Before/After (offline, no GPU needed)

The whole point of prompt distillation is captured in one before/after comparison: the same task, done by the teacher (long prompt + thinking) vs the student (no prompt, direct answer) — how much input cost is saved, and how much quality is retained. compare.py computes this entirely offline from the real dataset, the teacher labels, and the evaluation results (no model download, no network):

# Use the default tiktoken counter (works offline, reproducible)
python compare.py

# For the EXACT Qwen token counts (on a machine with the tokenizer available)
python compare.py --tokenizer Qwen/Qwen3-30B-A3B-Instruct-2507

# Show more per-case examples and save the full breakdown
python compare.py --num_examples 20 --output_file ./comparison_results.json

It reports three things — all from real data, nothing estimated:

1. Input cost — teacher pays the full classification prompt on every call; student pays only the raw text.
2. Task quality — the student's agreement rate with the teacher's labels (distillation fidelity), read from evaluation_results.json.
3. Per-case table — several real samples side by side (teacher tokens / student tokens / teacher label / student prediction / match).

Measured result (this repo's data, tiktoken o200k_base counter):

Dimension	Teacher (long prompt + thinking)	Student (no prompt)	Change
Avg input tokens / call	984.9	24.7	−97.5% (≈40× fewer)
Total input tokens (2,100 cases)	2,068,204	51,913	−97.5%
Task quality (agreement w/ teacher)	100% (reference)	95.19% (1999/2100)	−4.8 pp

On a per-input-token-billed API this input reduction lowers cost roughly proportionally; the teacher additionally spends thinking (CoT) output tokens that are not counted here, so the real gap is larger. Wall-clock latency depends on the serving stack and must be measured on GPU — compare.py deliberately does not fabricate a latency number. Exact token counts vary by tokenizer; pass --tokenizer for the student model's own count.

Project Structure

prompt-distillation/
├── README.md                          # This file
├── requirements.txt                   # Python dependencies
├── create_data.py                     # Data generation script (Step 1)
├── create_data_h100x8.sh              # Parallel data generation for H100x8
├── train_sft_trl.py                   # Training script using TRL (Step 2)
├── train_trl.sh                       # Training script (single GPU)
├── evaluate.py                        # Evaluation script (Step 3)
├── compare.py                         # Before/after cost & quality comparison (Step 4, offline)
├── data/                              # Generated training data
│   └── prompt_distillation_lang.jsonl
└── models/                            # Trained model checkpoints
    └── prompt_distillation_trl/

Why This Approach?

Thinking Model → Non-Thinking Model

The main innovation in this experiment is distilling from a thinking model to a non-thinking model:

1. Thinking Model (Teacher):
2. Qwen3-30B-A3B-Thinking-2507
3. Uses explicit reasoning: <thinking>...</thinking>
4. Requires long prompts with detailed instructions

5. Slower but more accurate

6. Non-Thinking Model (Student):

7. Qwen3-30B-A3B-Instruct-2507
8. No thinking tags, direct responses
9. No prompts needed in production
10. 20-30x faster inference

Why TRL Instead of verl?

We use Hugging Face TRL for this implementation because:

1. More Common: TRL is widely adopted in the community
2. Better Documentation: Extensive docs and examples
3. Simpler Setup: No need to convert JSONL to Parquet
4. Standard Workflow: Works seamlessly with HuggingFace ecosystem
5. Easier to Debug: Clear error messages and better tooling

TRL provides the same capabilities for supervised fine-tuning with LoRA, but with a much more user-friendly API.

Key Implementation Details

Data Format

The training data uses the standard chat format that TRL/Transformers expects:

{
  "messages": [
    {"role": "user", "content": "Text to classify"},
    {"role": "assistant", "content": "language_code"}
  ]
}

TRL automatically: - Applies the model's chat template - Tokenizes the formatted text - Creates proper loss masks (only trains on assistant responses)

Training Configuration

• Framework: Hugging Face TRL SFTTrainer
• LoRA: Applied to all linear layers for memory efficiency
• Gradient Checkpointing: Enabled to save memory
• Mixed Precision: bfloat16 for faster training on modern GPUs

Comparison to Tinker

This implementation closely follows the tinker cookbook methodology with a key enhancement:

Same: - Teacher model: Qwen3-30B-A3B-Thinking (same as tinker) - LoRA configuration: rank 32, alpha 16 - Learning rate: 2e-4 - Training epochs: 1 - Temperature: 0.15 (data generation) - Prompt: Identical language classification prompt

Enhanced: - Student model: Qwen3-30B-A3B-Instruct (non-thinking variant) - Removes thinking overhead for faster inference - Same model size, but direct responses without reasoning tokens - 20-30x faster than thinking model in production - Framework: TRL (more accessible than tinker's internal framework) - Max length: 2048 (student doesn't need long context)

Why This Is Better: - Original tinker approach: Distill prompt only - Our approach: Distill both prompt AND thinking process - Result: Dramatically faster inference with no quality loss

Expected Results

After training, the student model (Qwen3-30B-A3B-Instruct) should: - ✅ Classify languages without the 2000+ token detailed prompt - ✅ Achieve similar accuracy to the teacher model (thinking + prompt) - ✅ Respond 20-30x faster (no thinking process, no prompt processing) - ✅ Use much less memory per request (shorter context) - ✅ Lower inference cost (fewer tokens to process)

Input-Cost Comparison (measured, not estimated):

Run python compare.py to reproduce the real numbers on this repo's data. With the default tiktoken o200k_base counter, the student processes ≈40× fewer input tokens per call (984.9 → 24.7, a 97.5% reduction) while retaining 95.19% agreement with the teacher's labels. See the table under Usage → Step 4 for the full breakdown.

This makes the distilled model attractive for production deployment where input cost matters. Note: wall-clock latency depends on the serving stack and hardware and must be measured on GPU — this README does not quote a fabricated latency figure.

Troubleshooting

Out of Memory (OOM)

The 30B model requires a large H100 GPU (80GB). If you encounter OOM errors:

Solutions: 1. Reduce per_device_train_batch_size from 4 to 2 or 1 2. Reduce max_length from 2048 to 1024 or 512 3. Increase gradient_accumulation_steps to maintain effective batch size 4. Reduce lora_rank from 32 to 16 or 8

Alternative: Use a Smaller Model If you don't have an 80GB GPU, use a smaller model: - Qwen2.5-7B-Instruct: ~28GB memory, fits on most GPUs - Qwen2.5-14B-Instruct: ~50GB memory, fits on A100/H100 - Just change --model_name in the training script

Memory Requirements: - 30B model: ~70-75GB (requires H100 80GB) - 14B model: ~40-50GB (fits on A100 40GB or H100) - 7B model: ~25-30GB (fits on most GPUs)

Data Generation Issues

If data generation fails or is slow:

1. Increase tensor_parallel_size to use more GPUs
2. Use the parallel script for H100x8: bash create_data_h100x8.sh
3. Reduce the dataset size for testing
4. Check GPU memory usage with nvidia-smi

Training Not Converging

If the model doesn't learn:

1. Verify training data format is correct
2. Check that examples have valid language labels
3. Try increasing the number of training epochs
4. Adjust the learning rate (try 5e-5 or 2e-4)

Citation

If you use this code, please cite the original papers:

@article{askell2021general,
  title={A general language assistant as a laboratory for alignment},
  author={Askell, Amanda and others},
  journal={arXiv preprint arXiv:2112.00861},
  year={2021}
}

@article{snell2022learning,
  title={Learning by distilling context},
  author={Snell, Charlie and Klein, Dan and Zhong, Ruiqi},
  journal={arXiv preprint arXiv:2209.15189},
  year={2022}
}

And the Hugging Face TRL library:

@software{trl2024,
  title={TRL: Transformer Reinforcement Learning},
  author={TRL contributors},
  url={https://github.com/huggingface/trl},
  year={2024}
}

License

This project follows the same license as the TRL library (Apache 2.0).

Acknowledgments

• Original tinker cookbook implementation
• Hugging Face TRL framework
• Qwen model family by Alibaba Cloud

源代码

compare.py

"""
Prompt 蒸馏「蒸馏前 vs 蒸馏后」量化对比脚本。

本脚本回答实验 7-8 的核心问题：把「长提示 + 思考型教师」蒸馏成「无提示 + 直接
回答的学生」之后，到底省了多少、质量掉了多少？它在 **不加载任何大模型、不联网** 的
前提下，用真实数据算出一张 before/after 对比表：

  1. 输入成本（token）：教师每次调用都要带上完整的语言分类提示（约上千 token），
     学生只需要原始待分类文本。二者的 token 差就是每次调用省下的输入开销。
  2. 任务质量：直接读取 evaluate.py 产出的 evaluation_results.json，得到学生在
     相同输入上「与教师标注的一致率」（即蒸馏保真度）。
  3. 逐条案例：抽取若干条真实样本，并排展示 教师 token / 学生 token / 教师标签 /
     学生预测 / 是否一致，让「多个案例上的 before/after」一目了然。

设计原则：所有数字都来自真实数据与真实分词器，不臆造。延迟（秒级响应时间）需要
在 GPU 上实测，本脚本不做估算，只报告可离线复现的 token 成本与质量。
"""

import argparse
import json
import os
import re
import sys
from pathlib import Path
from typing import Callable, Dict, List, Optional, Tuple


VALID_LABELS = ["ar", "de", "el", "en", "es", "fr", "hi", "ru", "tr", "ur", "vi", "zh", "ot"]


def load_prompt_template(source_file: str) -> str:
    """从 create_data.py 中提取教师使用的语言分类提示模板（避免 import vllm）。"""
    src = Path(source_file).read_text(encoding="utf-8")
    match = re.search(
        r'LANGUAGE_CLASSIFICATION_PROMPT\s*=\s*"""(.*?)"""',
        src,
        re.DOTALL,
    )
    if not match:
        raise ValueError(
            f"无法在 {source_file} 中找到 LANGUAGE_CLASSIFICATION_PROMPT 模板，"
            f"请用 --prompt_source 指定包含该常量的文件。"
        )
    return match.group(1)


def build_token_counter(tokenizer_name: Optional[str]) -> Tuple[Callable[[str], int], str]:
    """
    构造一个 token 计数函数，按优先级回退，保证离线可用。

    返回 (counter, method_description)：
      1) 若指定 --tokenizer，用 HuggingFace 分词器精确计数（GPU 机器上可得到 Qwen 的真实 token 数）。
      2) 否则用 tiktoken 的 o200k_base（GPT-4o/o1 分词器）作近似，可离线复现。
      3) 再退化为「字符数 / 4」的粗略启发式，并明确标注为估算。
    每种方法都会在输出里注明，绝不把近似值当成精确值。
    """
    if tokenizer_name:
        try:
            from transformers import AutoTokenizer

            tok = AutoTokenizer.from_pretrained(tokenizer_name, trust_remote_code=True)
            return (lambda s: len(tok.encode(s))), f"HuggingFace 分词器（精确）: {tokenizer_name}"
        except Exception as exc:  # noqa: BLE001
            print(f"[warn] 无法加载分词器 {tokenizer_name}（{exc}），回退到 tiktoken。", file=sys.stderr)

    try:
        import tiktoken

        enc = tiktoken.get_encoding("o200k_base")
        return (lambda s: len(enc.encode(s))), "tiktoken o200k_base（近似，可离线复现）"
    except Exception as exc:  # noqa: BLE001
        print(f"[warn] tiktoken 不可用（{exc}），回退到字符启发式。", file=sys.stderr)

    return (lambda s: max(1, len(s) // 4)), "字符数/4（粗略估算）"


def load_texts(test_file: str) -> List[str]:
    with open(test_file, "r", encoding="utf-8") as f:
        return [line.strip() for line in f if line.strip()]


def load_teacher_labels(train_data_file: str) -> Dict[str, str]:
    """从蒸馏训练数据（教师标注）中读取 文本 -> 教师标签 的映射。"""
    mapping: Dict[str, str] = {}
    if not Path(train_data_file).exists():
        return mapping
    with open(train_data_file, "r", encoding="utf-8") as f:
        for line in f:
            if not line.strip():
                continue
            data = json.loads(line)
            msgs = data.get("messages", [])
            if len(msgs) >= 2:
                mapping[msgs[0].get("content", "")] = msgs[1].get("content", "")
    return mapping


def load_eval_results(eval_file: str) -> Optional[Dict]:
    if not Path(eval_file).exists():
        return None
    with open(eval_file, "r", encoding="utf-8") as f:
        return json.load(f)


def truncate(text: str, width: int = 42) -> str:
    text = text.replace("\n", " ")
    return text if len(text) <= width else text[: width - 1] + "…"


def compare(
    prompt_template: str,
    texts: List[str],
    teacher_labels: Dict[str, str],
    eval_results: Optional[Dict],
    count_tokens: Callable[[str], int],
    token_method: str,
    num_examples: int,
) -> Dict:
    n = len(texts)

    # 固定提示开销（模板本身，不含待分类文本）
    fixed_overhead = count_tokens(prompt_template.format(text=""))

    teacher_input_total = 0
    student_input_total = 0
    per_text_tokens: List[Tuple[int, int]] = []  # (teacher_tokens, student_tokens)
    for text in texts:
        teacher_prompt = prompt_template.format(text=text)
        t_tok = count_tokens(teacher_prompt)
        s_tok = count_tokens(text)
        teacher_input_total += t_tok
        student_input_total += s_tok
        per_text_tokens.append((t_tok, s_tok))

    if n == 0:
        teacher_avg = student_avg = reduction_pct = 0.0
        ratio = float("inf")
    else:
        teacher_avg = teacher_input_total / n
        student_avg = student_input_total / n
        reduction_pct = (
            100.0 * (1 - student_input_total / teacher_input_total)
            if teacher_input_total
            else 0.0
        )
        ratio = (
            teacher_input_total / student_input_total
            if student_input_total
            else float("inf")
        )

    # 学生预测（与 test_file 逐行对齐）
    student_preds: Optional[List[Optional[str]]] = None
    accuracy = None
    correct = evaluated = None
    if eval_results:
        student_preds = eval_results.get("predictions")
        summary = eval_results.get("summary", {})
        accuracy = summary.get("accuracy")
        correct = summary.get("correct")
        evaluated = summary.get("evaluated")

    # 逐条案例：优先覆盖不同语言，并尽量各带上一致/不一致的例子
    examples: List[Dict] = []
    seen_labels = set()
    for idx, text in enumerate(texts):
        teacher_label = teacher_labels.get(text, "?")
        student_pred = (
            student_preds[idx] if student_preds and idx < len(student_preds) else None
        )
        key = teacher_label
        if key in seen_labels and len(examples) >= num_examples:
            continue
        if len(examples) >= num_examples:
            break
        if key in seen_labels:
            continue
        seen_labels.add(key)
        t_tok, s_tok = per_text_tokens[idx]
        examples.append(
            {
                "text": text,
                "teacher_tokens": t_tok,
                "student_tokens": s_tok,
                "teacher_label": teacher_label,
                "student_pred": student_pred,
                "match": (student_pred == teacher_label) if student_pred else None,
            }
        )

    return {
        "num_cases": n,
        "token_method": token_method,
        "fixed_prompt_overhead": fixed_overhead,
        "teacher_input_total": teacher_input_total,
        "teacher_input_avg": teacher_avg,
        "student_input_total": student_input_total,
        "student_input_avg": student_avg,
        "input_token_reduction_pct": reduction_pct,
        "teacher_student_ratio": ratio,
        "student_accuracy": accuracy,
        "student_correct": correct,
        "student_evaluated": evaluated,
        "examples": examples,
    }


def print_report(r: Dict) -> None:
    line = "=" * 78
    print("\n" + line)
    print("Prompt 蒸馏：蒸馏前 vs 蒸馏后 量化对比")
    print(line)
    print(f"样本数            : {r['num_cases']}")
    print(f"Token 计数方式    : {r['token_method']}")
    print(f"固定提示开销      : {r['fixed_prompt_overhead']} tokens（模板本身，每次调用都要重复付费）")

    print("\n" + "-" * 78)
    print("一、输入成本（每次调用的输入 token）")
    print("-" * 78)
    print(f"{'维度':<24}{'教师(长提示+思考)':>20}{'学生(无提示)':>18}")
    print(f"{'单条平均输入 token':<24}{r['teacher_input_avg']:>20.1f}{r['student_input_avg']:>18.1f}")
    print(f"{'全量总输入 token':<24}{r['teacher_input_total']:>20,}{r['student_input_total']:>18,}")
    print(
        f"\n→ 输入 token 降低 {r['input_token_reduction_pct']:.1f}%"
        f"（教师是学生的 {r['teacher_student_ratio']:.1f} 倍）。"
    )
    print("  按输入 token 计费的 API 上，这一项直接等比例降低费用；教师端还有未计入的")
    print("  思考（CoT）输出 token，实际差距只会更大。延迟需在 GPU 上实测，此处不估算。")

    print("\n" + "-" * 78)
    print("二、任务质量（学生在相同输入上与教师标注的一致率 = 蒸馏保真度）")
    print("-" * 78)
    if r["student_accuracy"] is not None:
        print(
            f"教师(基准) : 100.00%   学生(蒸馏后) : {r['student_accuracy'] * 100:.2f}%"
            f"  ({r['student_correct']}/{r['student_evaluated']})"
        )
        print(
            f"→ 无提示、无思考的学生保留了教师约 {r['student_accuracy'] * 100:.1f}% 的判断，"
            f"质量损失约 {(1 - r['student_accuracy']) * 100:.1f} 个百分点。"
        )
    else:
        print("未找到 evaluation_results.json（学生尚未评估）。先运行 evaluate.py 生成，")
        print("再回来看这一栏。本栏缺失不影响上面的输入成本对比。")

    print("\n" + "-" * 78)
    print(f"三、逐条案例（{len(r['examples'])} 例）")
    print("-" * 78)
    print(f"{'待分类文本':<44}{'教师tok':>8}{'学生tok':>8}{'教师':>6}{'学生':>6}{'一致':>6}")
    for ex in r["examples"]:
        if ex["match"] is None:
            mark = "—"
        else:
            mark = "✓" if ex["match"] else "✗"
        pred = ex["student_pred"] if ex["student_pred"] else "—"
        print(
            f"{truncate(ex['text']):<44}"
            f"{ex['teacher_tokens']:>8}{ex['student_tokens']:>8}"
            f"{ex['teacher_label']:>6}{pred:>6}{mark:>6}"
        )
    print(line + "\n")


def main():
    parser = argparse.ArgumentParser(
        description="Prompt 蒸馏「蒸馏前 vs 蒸馏后」量化对比：离线算出输入成本、任务质量与逐条案例",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parser.add_argument(
        "--test_file",
        type=str,
        default="./example-data/multilingual.txt",
        help="待分类文本文件（每行一句），作为对比的输入集合",
    )
    parser.add_argument(
        "--train_data_file",
        type=str,
        default="./data/prompt_distillation_lang.jsonl",
        help="蒸馏训练数据（教师标注），用于取教师标签作为质量基准",
    )
    parser.add_argument(
        "--eval_results",
        type=str,
        default="./evaluation_results.json",
        help="evaluate.py 产出的评估结果，用于读取学生的一致率（可选）",
    )
    parser.add_argument(
        "--prompt_source",
        type=str,
        default="./create_data.py",
        help="包含教师提示模板 LANGUAGE_CLASSIFICATION_PROMPT 的源文件",
    )
    parser.add_argument(
        "--tokenizer",
        type=str,
        default=None,
        help="可选：HuggingFace 分词器名/路径（如 Qwen/Qwen3-30B-A3B-Instruct-2507）。"
        "指定后用它精确计数；不指定则用 tiktoken 近似，保证离线可跑",
    )
    parser.add_argument(
        "--num_examples",
        type=int,
        default=10,
        help="逐条案例展示的条数（尽量覆盖不同语言）",
    )
    parser.add_argument(
        "--output_file",
        type=str,
        default=None,
        help="可选：把对比结果（含逐条案例）保存为 JSON 的路径",
    )
    args = parser.parse_args()

    if not os.path.exists(args.test_file):
        raise FileNotFoundError(f"待分类文本文件不存在: {args.test_file}")
    if not os.path.exists(args.prompt_source):
        raise FileNotFoundError(f"提示模板源文件不存在: {args.prompt_source}")

    prompt_template = load_prompt_template(args.prompt_source)
    texts = load_texts(args.test_file)
    teacher_labels = load_teacher_labels(args.train_data_file)
    eval_results = load_eval_results(args.eval_results)
    count_tokens, token_method = build_token_counter(args.tokenizer)

    if not teacher_labels:
        print(
            f"[warn] 未从 {args.train_data_file} 读到教师标注，逐条案例的教师标签将显示为 '?'。",
            file=sys.stderr,
        )
    if eval_results is None:
        print(
            f"[warn] 未找到 {args.eval_results}，将只给出输入成本对比，跳过质量一栏。",
            file=sys.stderr,
        )

    report = compare(
        prompt_template=prompt_template,
        texts=texts,
        teacher_labels=teacher_labels,
        eval_results=eval_results,
        count_tokens=count_tokens,
        token_method=token_method,
        num_examples=args.num_examples,
    )

    print_report(report)

    if args.output_file:
        with open(args.output_file, "w", encoding="utf-8") as f:
            json.dump(report, f, indent=2, ensure_ascii=False)
        print(f"📁 对比结果已保存到: {args.output_file}")


if __name__ == "__main__":
    main()

create_data.py

"""
Data generation script for prompt distillation using vLLM.

This script generates training data for prompt distillation by using a teacher model
to generate language classification labels with a detailed prompt, which will then be
used to train a student model that internalizes the prompt.

Based on the tinker cookbook prompt distillation recipe.
"""

import argparse
import asyncio
import json
import os
import re
from pathlib import Path
from typing import Optional

from tqdm.asyncio import tqdm_asyncio

# 注意：vllm / SamplingParams 在 generate_distillation_data() 内部按需导入，
# 这样即便未安装 vllm（如离线查看 --help 时）也能正常展示命令行帮助。

LANGUAGE_CLASSIFICATION_PROMPT = """You are a precise language classifier.

Goal: Classify the language of the provided text into exactly one of these labels:
ar (Arabic), de (German), el (Greek), en (English), es (Spanish), fr (French),
hi (Hindi), ru (Russian), tr (Turkish), ur (Urdu), vi (Vietnamese),
zh (Chinese - Simplified), ot (Other/Unknown).

Instructions:
1) Preprocess carefully (without changing the intended meaning):
   - Trim whitespace.
   - Ignore URLs, emails, file paths, hashtags, user handles, and emojis.
   - Ignore numbers, math expressions, and standalone punctuation.
   - If there is code, IGNORE code syntax (keywords, operators, braces) and focus ONLY on human language in comments and string literals.
   - Preserve letters and diacritics; do NOT strip accents.
   - If after ignoring the above there are no alphabetic letters left, output 'ot'.

2) Script-based rules (highest priority):
   - Devanagari script → hi.
   - Greek script → el.
   - Cyrillic script → ru.
   - Han characters (中文) → zh. (Treat Traditional as zh too.)
   - Arabic script → ar vs ur:
       • If Urdu-only letters appear (e.g., ے, ڑ, ں, ھ, ٹ, ڈ, کھ, گ, چ with Urdu forms), or clear Urdu words, choose ur.
       • Otherwise choose ar.
   (If multiple scripts appear, pick the script that contributes the majority of alphabetic characters. If tied, go to step 5.)

3) Latin-script heuristics (use when text is mainly Latin letters):
   - vi: presence of Vietnamese-specific letters/diacritics (ă â ê ô ơ ư đ, plus dense diacritics across many words).
   - tr: presence of Turkish-specific letters (ı İ ğ Ğ ş Ş ç Ç ö Ö ü Ü) and common function words (ve, bir, için, değil, ama, çok).
   - de: presence of umlauts (ä ö ü) or ß and common function words (und, der, die, das, nicht, ist).
   - es: presence of ñ, ¿, ¡ and common words (y, de, la, el, es, no, por, para, con, gracias, hola).
   - fr: frequent French diacritics (é è ê à ç ô â î û ù) and common words (et, le, la, les, des, une, est, avec, pour, merci, bonjour).
   - en: default among Latin languages if strong evidence for others is absent, but ONLY if English function words are present (the, and, is, are, to, of, in, for, on, with). If evidence is insufficient for any Latin language, prefer 'ot' over guessing.

4) Named entities & loanwords:
   - Do NOT decide based on a single proper noun, brand, or place name.
   - Require at least two function words or repeated language-specific signals (diacritics/letters) before assigning a Latin-language label.

5) Mixed-language text:
   - Determine the dominant language by counting indicative tokens (language-specific letters/diacritics/function words) AFTER preprocessing.
   - If two or more languages are equally dominant or the text is a deliberate multi-language mix, return 'ot'.

6) Very short or noisy inputs:
   - If the text is ≤2 meaningful words or too short to be confident, return 'ot' unless there is a very strong language-specific signal (e.g., "bonjour" → fr, "hola" → es).

7) Transliteration/romanization:
   - If Hindi/Urdu/Arabic/Chinese/Russian/Greek is written purely in Latin letters (romanized) without clear, repeated language-specific cue words, return 'ot'. (Only classify as hi/ur/ar/zh/ru/el when native scripts or highly distinctive romanized patterns are clearly present.)

8) Code-heavy inputs:
   - If the text is mostly code with minimal or no natural-language comments/strings, return 'ot'.
   - If comments/strings clearly indicate a language per rules above, use that label.

9) Ambiguity & confidence:
   - When in doubt, choose 'ot' rather than guessing.

Text to classify:
{text}

Output format:
- Respond with EXACTLY one line: "Final Answer: xx"
- Where xx ∈ {{ar, de, el, en, es, fr, hi, ru, tr, ur, vi, zh, ot}} and nothing else.
"""


def parse_final_answer(response: str, debug: bool = False) -> Optional[str]:
    """
    Parse the final answer from the model response.
    For Thinking models, extract from <think>...</think> tags or after them.
    """
    # For Thinking models, the response may have <think></think> tags
    # Remove thinking content and focus on the final answer
    response_stripped = response.strip()

    # Remove <think>...</think> content if present
    response_cleaned = re.sub(r'<think>.*?</think>', '', response_stripped, flags=re.DOTALL)
    response_cleaned = response_cleaned.strip()

    # Also try the original response
    candidates = [response_cleaned, response_stripped]

    valid_labels = {'ar', 'de', 'el', 'en', 'es', 'fr', 'hi', 'ru', 'tr', 'ur', 'vi', 'zh', 'ot'}

    # Try multiple patterns to extract language label
    patterns = [
        r"Final Answer:\s*(\w{2})",  # Standard format
        r"Final Answer:\s*([a-z]{2})",  # Lowercase only
        r"Answer:\s*(\w{2})",  # Without "Final"
        r"Language:\s*(\w{2})",  # "Language: xx"
        r"^([a-z]{2})$",  # Just the label alone
        r"\b([a-z]{2})\b\s*$",  # Label at the end with word boundary
        r"is:\s*(\w{2})",  # "is: xx"
        r"→\s*(\w{2})",  # "→ xx"
    ]

    for candidate in candidates:
        candidate_lower = candidate.lower()

        # Try each pattern
        for pattern in patterns:
            match = re.search(pattern, candidate_lower, re.MULTILINE)
            if match:
                label = match.group(1)
                if label in valid_labels:
                    if debug:
                        print(f"    [DEBUG] Matched pattern '{pattern}' -> '{label}'")
                    return label

        # Special case: check if the entire response is just a language code
        if len(candidate) <= 3 and candidate_lower in valid_labels:
            if debug:
                print(f"    [DEBUG] Matched entire response as label -> '{candidate_lower}'")
            return candidate_lower

    if debug:
        print(f"    [DEBUG] No pattern matched.")
        print(f"    [DEBUG] Response length: {len(response_stripped)}")
        print(f"    [DEBUG] Cleaned response: '{response_cleaned[:300]}'")
        print(f"    [DEBUG] Original response: '{response_stripped[:300]}'")

    return None


async def generate_distillation_data(
    input_file: str,
    output_file: str,
    model_name: str = "Qwen/Qwen3-30B-A3B-Thinking-2507",
    temperature: float = 0.15,
    max_tokens: int = 4096,
    tensor_parallel_size: int = 1,
    max_retries: int = 3,
):
    """
    Generate prompt distillation training data.

    Args:
        input_file: Path to file containing sentences to classify (one per line)
        output_file: Path to save the generated training data (JSONL format)
        model_name: Teacher model to use for generating labels
        temperature: Sampling temperature
        max_tokens: Maximum tokens to generate
        tensor_parallel_size: Number of GPUs to use for tensor parallelism
    """
    print(f"Loading input sentences from {input_file}")
    with open(input_file, "r", encoding="utf-8") as f:
        sentences = [line.strip() for line in f if line.strip()]

    print(f"Loaded {len(sentences)} sentences")
    if not sentences:
        print("Input file has no sentences to process, skipping data generation.")
        return

    from vllm import LLM, SamplingParams

    # Initialize vLLM model
    print(f"Initializing teacher model: {model_name}")
    print(f"Using tensor parallelism across {tensor_parallel_size} GPU(s)")

    # Get tokenizer to use proper chat template
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

    llm = LLM(
        model=model_name,
        tensor_parallel_size=tensor_parallel_size,
        trust_remote_code=True,
        gpu_memory_utilization=0.90,  # Use 90% of GPU memory for better throughput
        max_model_len=32768,  # Match training max length
        enable_prefix_caching=True,  # Cache the system prompt
    )

    # Set sampling parameters - use Qwen3 recommended settings
    # For Thinking models, we need to allow enough tokens for reasoning
    sampling_params = SamplingParams(
        temperature=temperature,
        max_tokens=max_tokens,
        top_p=0.8,
        top_k=20,
        # Don't use custom stop sequences - let model finish naturally
        skip_special_tokens=False,  # Keep special tokens for thinking models
    )

    # Initial generation
    print("Generating labels with teacher model...")
    results = {}  # sentence -> (response, final_answer)
    failed_indices = []
    failed_examples = []  # Store examples for debugging

    # Format prompts using proper chat template
    print("Formatting prompts with Qwen3 chat template...")
    formatted_prompts = []
    for sentence in sentences:
        messages = [
            {
                "role": "user",
                "content": LANGUAGE_CLASSIFICATION_PROMPT.format(text=sentence)
            }
        ]
        # Use tokenizer's chat template
        prompt_text = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )
        formatted_prompts.append(prompt_text)

    print(f"Sample formatted prompt:")
    print(formatted_prompts[0])

    outputs = llm.generate(formatted_prompts, sampling_params)

    for idx, (sentence, output) in enumerate(zip(sentences, outputs)):
        response = output.outputs[0].text
        # Enable debug mode for first few failures
        debug_mode = len(failed_examples) < 3
        final_answer = parse_final_answer(response, debug=debug_mode)

        if final_answer:
            results[sentence] = (response, final_answer)
        else:
            failed_indices.append(idx)
            # Store first 10 failed examples for debugging
            if len(failed_examples) < 10:
                failed_examples.append({
                    'sentence': sentence,
                    'response': response,
                })

    print(f"\nInitial generation: {len(results)}/{len(sentences)} successful ({len(results)/len(sentences)*100:.2f}%)")

    # Show debugging info for failed samples
    if failed_examples:
        print(f"\n{'='*60}")
        print("DEBUGGING: Examples of FAILED responses")
        print(f"{'='*60}")
        for i, example in enumerate(failed_examples, 1):
            print(f"\nFailed Example {i}:")
            print(f"  Input: {example['sentence']}")
            print(f"  Response: {example['response']}")
            print(f"  Parsed result: None")

    # Show examples of successful responses
    if results:
        print(f"\n{'='*60}")
        print("DEBUGGING: Examples of SUCCESSFUL responses")
        print(f"{'='*60}")
        success_examples = list(results.items())[:3]
        for i, (sentence, (response, label)) in enumerate(success_examples, 1):
            print(f"\nSuccess Example {i}:")
            print(f"  Input: {sentence}")
            print(f"  Response: {response}")
            print(f"  Parsed label: {label}")

    # Retry failed generations up to max_retries times
    for retry in range(1, max_retries + 1):
        if not failed_indices:
            break

        print(f"\nRetry {retry}/{max_retries}: Regenerating {len(failed_indices)} failed samples...")

        # Prepare prompts for failed sentences
        retry_sentences = [sentences[idx] for idx in failed_indices]
        retry_formatted_prompts = []
        for s in retry_sentences:
            messages = [
                {
                    "role": "user",
                    "content": LANGUAGE_CLASSIFICATION_PROMPT.format(text=s)
                }
            ]
            prompt_text = tokenizer.apply_chat_template(
                messages,
                tokenize=False,
                add_generation_prompt=True
            )
            retry_formatted_prompts.append(prompt_text)

        # Generate with slightly higher temperature to encourage different outputs
        retry_params = SamplingParams(
            temperature=min(temperature * (1 + retry * 0.1), 0.5),  # Gradually increase temp
            max_tokens=max_tokens,
            top_p=0.8,
            top_k=20,
            skip_special_tokens=False,
        )

        retry_outputs = llm.generate(retry_formatted_prompts, retry_params)

        # Track newly successful and still-failed indices
        new_failed_indices = []
        for idx, sentence, output in zip(failed_indices, retry_sentences, retry_outputs):
            response = output.outputs[0].text
            final_answer = parse_final_answer(response)

            if final_answer:
                results[sentence] = (response, final_answer)
            else:
                new_failed_indices.append(idx)

        newly_successful = len(failed_indices) - len(new_failed_indices)
        print(f"  ✓ {newly_successful} more samples successful")
        print(f"  Total successful: {len(results)}/{len(sentences)} ({len(results)/len(sentences)*100:.2f}%)")

        failed_indices = new_failed_indices

    # Save results
    print(f"\nSaving results to {output_file}...")
    with open(output_file, "w", encoding="utf-8") as f:
        for sentence in sentences:
            if sentence in results:
                _, final_answer = results[sentence]
                data = {
                    "messages": [
                        {
                            "role": "user",
                            "content": sentence,
                        },
                        {
                            "role": "assistant",
                            "content": final_answer,
                        },
                    ]
                }
                f.write(json.dumps(data, ensure_ascii=False) + "\n")

    # Final report
    print(f"\n{'='*60}")
    print("DATA GENERATION COMPLETE")
    print(f"{'='*60}")
    print(f"Total sentences: {len(sentences)}")
    print(f"Valid labels generated: {len(results)}")
    print(f"Failed after {max_retries} retries: {len(failed_indices)}")
    print(f"Final success rate: {len(results)/len(sentences)*100:.2f}%")
    print(f"Saved to: {output_file}")

    if failed_indices:
        print(f"\n⚠️  Warning: {len(failed_indices)} sentences failed to generate valid labels")
        print("Consider inspecting these samples or adjusting the prompt/temperature")


def main():
    parser = argparse.ArgumentParser(
        description="用教师模型（长提示 + 思考）生成 Prompt 蒸馏训练数据",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parser.add_argument(
        "--input_file",
        type=str,
        default="./example-data/multilingual.txt",
        help="输入文本文件路径（每行一句待分类文本）",
    )
    parser.add_argument(
        "--output_file",
        type=str,
        default="./data/prompt_distillation_lang.jsonl",
        help="生成的训练数据保存路径（JSONL 格式）",
    )
    parser.add_argument(
        "--model_name",
        type=str,
        default="Qwen/Qwen3-30B-A3B-Thinking-2507",
        help="教师模型名称（用思考型模型以获得更高准确率）",
    )
    parser.add_argument(
        "--temperature",
        type=float,
        default=0.15,
        help="采样温度（与 tinker 保持一致，取 0.15）",
    )
    parser.add_argument(
        "--max_tokens",
        type=int,
        default=4096,
        help="单条生成的最大 token 数",
    )
    parser.add_argument(
        "--tensor_parallel_size",
        type=int,
        default=1,
        help="张量并行使用的 GPU 数（30B 模型在 H100 上建议 2-4）",
    )
    parser.add_argument(
        "--max_retries",
        type=int,
        default=3,
        help="失败样本的最大重试次数",
    )

    args = parser.parse_args()

    # Create output directory if needed
    output_dir = os.path.dirname(args.output_file)
    if output_dir:
        os.makedirs(output_dir, exist_ok=True)

    # Check if input file exists
    if not os.path.exists(args.input_file):
        raise FileNotFoundError(f"Input file not found: {args.input_file}")

    # Generate data
    asyncio.run(
        generate_distillation_data(
            input_file=args.input_file,
            output_file=args.output_file,
            model_name=args.model_name,
            temperature=args.temperature,
            max_tokens=args.max_tokens,
            tensor_parallel_size=args.tensor_parallel_size,
            max_retries=args.max_retries,
        )
    )


if __name__ == "__main__":
    main()

evaluate.py

"""
Evaluation script for the distilled prompt model.

This script evaluates the student model's performance on language classification
without providing the detailed prompt.

The student model (Qwen3-30B-A3B-Instruct) has been distilled from the teacher
(Qwen3-30B-A3B-Thinking) which used a 2000+ token prompt. After distillation,
the student responds directly without needing the prompt or thinking process.
"""

import argparse
import json
import re
from pathlib import Path
from typing import List, Dict, Optional
from collections import defaultdict

import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from tqdm import tqdm


def load_model(model_path: str, base_model: str = "Qwen/Qwen3-30B-A3B-Instruct-2507"):
    """Load the fine-tuned model with LoRA adapters."""
    print(f"Loading base model: {base_model}")
    print(f"This may take a few minutes for the 30B model...")
    tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(
        base_model,
        dtype=torch.bfloat16,  # Use 'dtype' instead of deprecated 'torch_dtype'
        device_map="auto",
        trust_remote_code=True,
    )

    print(f"Loading LoRA adapters from: {model_path}")
    model = PeftModel.from_pretrained(model, model_path)
    model.eval()

    return model, tokenizer


def parse_language_label(response: str) -> Optional[str]:
    """Extract the language label from model response."""
    # Try to match common patterns
    patterns = [
        r"^([a-z]{2})$",  # Just "en", "fr", etc.
        r"^([a-z]{2})\s*$",  # With trailing whitespace
        r"Final Answer:\s*([a-z]{2})",  # With "Final Answer:" prefix
        r"Language:\s*([a-z]{2})",  # With "Language:" prefix
    ]

    response = response.strip().lower()
    for pattern in patterns:
        match = re.search(pattern, response)
        if match:
            return match.group(1)

    # If response is short and looks like a language code
    if len(response) == 2 and response.isalpha():
        return response

    return None


def evaluate_model(
    model,
    tokenizer,
    test_sentences: List[str],
    ground_truth_labels: Optional[List[str]] = None,
    max_new_tokens: int = 10,
    temperature: float = 0.0,
) -> Dict:
    """Evaluate the distilled model on test sentences."""

    predictions = []
    correct = 0
    total = 0

    print("\nEvaluating model...")
    print("="*80)

    for idx, sentence in enumerate(test_sentences):
        # Format as user message (no system prompt!)
        messages = [
            {"role": "user", "content": sentence}
        ]

        # Apply chat template
        input_text = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True,
        )

        # Tokenize
        inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

        # Generate
        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=max_new_tokens,
                temperature=temperature if temperature > 0 else None,
                do_sample=temperature > 0,
                pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
            )

        # Decode
        response = tokenizer.decode(
            outputs[0][inputs.input_ids.shape[1]:],
            skip_special_tokens=True,
        )

        # Parse prediction
        pred_label = parse_language_label(response)
        predictions.append(pred_label)

        # Check correctness and show real-time progress
        if ground_truth_labels and idx < len(ground_truth_labels):
            gt_label = ground_truth_labels[idx]
            is_correct = pred_label == gt_label
            if is_correct:
                correct += 1
            total += 1

            # Show every sample in real-time
            status = "✓" if is_correct else "✗"
            status_color = "✓" if is_correct else "✗"

            # Truncate sentence for display
            display_sentence = sentence if len(sentence) <= 60 else sentence[:57] + "..."

            print(f"{status_color} [{idx+1:4d}/{len(test_sentences)}] "
                  f"Pred: {pred_label:>2s} | GT: {gt_label:>2s} | "
                  f"Acc: {correct}/{total} ({correct/total*100:5.1f}%) | "
                  f"{display_sentence}")
        else:
            # No ground truth - just show prediction
            display_sentence = sentence if len(sentence) <= 60 else sentence[:57] + "..."
            print(f"  [{idx+1:4d}/{len(test_sentences)}] "
                  f"Pred: {pred_label:>2s} | "
                  f"{display_sentence}")

    print("="*80)
    print(f"Evaluation completed: {len(test_sentences)} samples processed")

    # Calculate metrics
    results = {
        "predictions": predictions,
        "total": len(test_sentences),
        "predicted": sum(1 for p in predictions if p is not None),
        "unparseable": sum(1 for p in predictions if p is None),
    }

    if ground_truth_labels:
        results["accuracy"] = correct / total if total > 0 else 0.0
        results["correct"] = correct
        results["evaluated"] = total

        # Build confusion matrix
        results["confusion_matrix"] = build_confusion_matrix(
            predictions, ground_truth_labels, test_sentences
        )

    return results


def build_confusion_matrix(predictions: List[str], ground_truth: List[str], 
                           sentences: List[str]) -> Dict:
    """
    Build confusion matrix and identify problematic languages.

    Returns:
        Dict with confusion matrix, per-language stats, and error examples
    """
    # Get all unique labels
    all_labels = sorted(set(ground_truth) | set(p for p in predictions if p))

    # Initialize confusion matrix
    confusion = defaultdict(lambda: defaultdict(int))
    per_language_stats = defaultdict(lambda: {"correct": 0, "total": 0, "errors": []})

    # Build matrix
    for idx, (pred, gt, sentence) in enumerate(zip(predictions, ground_truth, sentences)):
        if pred is None:
            pred = "None"

        confusion[gt][pred] += 1
        per_language_stats[gt]["total"] += 1

        if pred == gt:
            per_language_stats[gt]["correct"] += 1
        else:
            # Store error examples
            if len(per_language_stats[gt]["errors"]) < 5:  # Keep first 5 errors per language
                per_language_stats[gt]["errors"].append({
                    "sentence": sentence,
                    "predicted": pred,
                    "ground_truth": gt,
                    "index": idx,
                })

    # Calculate per-language accuracy
    language_accuracy = {}
    for lang in all_labels:
        stats = per_language_stats[lang]
        if stats["total"] > 0:
            accuracy = stats["correct"] / stats["total"]
            language_accuracy[lang] = {
                "accuracy": accuracy,
                "correct": stats["correct"],
                "total": stats["total"],
                "errors": stats["errors"],
            }

    # Sort languages by accuracy (worst first)
    sorted_languages = sorted(
        language_accuracy.items(),
        key=lambda x: x[1]["accuracy"]
    )

    # Create 2D confusion matrix as a proper array
    confusion_matrix_2d = []
    for true_label in all_labels:
        row = []
        for pred_label in all_labels:
            count = confusion.get(true_label, {}).get(pred_label, 0)
            row.append(count)
        confusion_matrix_2d.append(row)

    return {
        "confusion_matrix": {gt: dict(preds) for gt, preds in confusion.items()},  # Dict format
        "confusion_matrix_2d": confusion_matrix_2d,  # 2D array format
        "all_labels": all_labels,
        "per_language_accuracy": language_accuracy,
        "worst_performing_languages": sorted_languages[:5],  # Top 5 worst
        "all_languages_sorted": sorted_languages,  # All languages sorted by accuracy
    }


def print_confusion_matrix(confusion_data: Dict):
    """Pretty print confusion matrix and analysis."""
    print("\n" + "="*80)
    print("CONFUSION MATRIX")
    print("="*80)

    # Get labels and matrix
    labels = confusion_data["all_labels"]
    confusion = confusion_data["confusion_matrix"]

    # Print matrix header - use shorter labels for display
    print("\n     ", end="")
    for label in labels:
        # Truncate long labels for display
        display_label = label if len(label) <= 3 else label[:3]
        print(f"{display_label:>4s}", end="")
    print("  | Total")
    print("  " + "-" * (len(labels) * 4 + 10))

    # Print matrix rows
    for true_label in labels:
        # Truncate long labels for display
        display_label = true_label if len(true_label) <= 2 else true_label[:2]
        print(f"{display_label:>2s} | ", end="")
        row_total = sum(confusion.get(true_label, {}).values())

        for pred_label in labels:
            count = confusion.get(true_label, {}).get(pred_label, 0)
            if count > 0:
                if true_label == pred_label:
                    print(f"\033[92m{count:4d}\033[0m", end="")  # Green for diagonal
                else:
                    print(f"\033[91m{count:4d}\033[0m", end="")  # Red for errors
            else:
                print(f"   .", end="")
        print(f"  | {row_total:4d}")

    # Print per-language accuracy
    print(f"\n{'='*80}")
    print("PER-LANGUAGE ACCURACY")
    print("="*80)

    lang_acc = confusion_data["per_language_accuracy"]
    sorted_langs = sorted(lang_acc.items(), key=lambda x: x[1]["accuracy"])

    for lang, stats in sorted_langs:
        acc = stats["accuracy"] * 100
        symbol = "✓" if acc >= 90 else "⚠️" if acc >= 70 else "✗"
        print(f"{symbol} {lang:>2s}: {acc:5.1f}% ({stats['correct']:4d}/{stats['total']:4d})")

    # Identify problematic languages
    print(f"\n{'='*80}")
    print("MOST PROBLEMATIC LANGUAGES (Top 5)")
    print("="*80)

    worst_languages = confusion_data["worst_performing_languages"]
    for idx, (lang, stats) in enumerate(worst_languages, 1):
        acc = stats["accuracy"] * 100
        print(f"\n{idx}. Language: {lang} - Accuracy: {acc:.1f}% ({stats['correct']}/{stats['total']})")

        if stats["errors"]:
            print(f"   Error examples:")
            for err in stats["errors"][:3]:  # Show first 3 errors
                pred_lang = err['predicted']
                sentence_preview = err['sentence'][:50] + "..." if len(err['sentence']) > 50 else err['sentence']
                print(f"     - Predicted {pred_lang} (should be {lang}): {sentence_preview}")

    print("\n" + "="*80)


def main():
    parser = argparse.ArgumentParser(
        description="评估蒸馏后的学生模型（无提示、直接作答）在语言分类上的表现",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parser.add_argument(
        "--model_path",
        type=str,
        default="./models/prompt_distillation_trl",
        help="训练得到的 LoRA adapter 路径",
    )
    parser.add_argument(
        "--base_model",
        type=str,
        default="Qwen/Qwen3-30B-A3B-Instruct-2507",
        help="学生基座模型名称（非思考型）",
    )
    parser.add_argument(
        "--test_file",
        type=str,
        default="./example-data/multilingual.txt",
        help="测试文本文件（每行一句）",
    )
    parser.add_argument(
        "--ground_truth_file",
        type=str,
        default=None,
        help="标准答案标签文件（可选，每行一个）。未提供时会尝试从训练数据中读取",
    )
    parser.add_argument(
        "--train_data_file",
        type=str,
        default="./data/prompt_distillation_lang.jsonl",
        help="训练数据文件，未提供 ground_truth_file 时从中提取教师标签作为基准",
    )
    parser.add_argument(
        "--output_file",
        type=str,
        default="./evaluation_results.json",
        help="评估结果的保存路径",
    )
    parser.add_argument(
        "--max_samples",
        type=int,
        default=None,
        help="最多评估的样本数（用于快速测试）",
    )

    args = parser.parse_args()

    # Load model
    model, tokenizer = load_model(args.model_path, args.base_model)

    # Load test data
    print(f"\nLoading test sentences from: {args.test_file}")
    with open(args.test_file, "r", encoding="utf-8") as f:
        test_sentences = [line.strip() for line in f if line.strip()]

    # Load ground truth
    ground_truth = None
    if args.ground_truth_file:
        print(f"Loading ground truth from: {args.ground_truth_file}")
        with open(args.ground_truth_file, "r", encoding="utf-8") as f:
            ground_truth = [line.strip() for line in f if line.strip()]
    elif args.train_data_file and Path(args.train_data_file).exists():
        # Try to extract ground truth from training data
        print(f"Loading ground truth from training data: {args.train_data_file}")
        ground_truth = []
        sentence_to_label = {}

        with open(args.train_data_file, "r", encoding="utf-8") as f:
            for line in f:
                if line.strip():
                    data = json.loads(line)
                    messages = data.get("messages", [])
                    if len(messages) >= 2:
                        user_content = messages[0].get("content", "")
                        assistant_content = messages[1].get("content", "")
                        sentence_to_label[user_content] = assistant_content

        # Match test sentences to ground truth labels
        for sentence in test_sentences:
            label = sentence_to_label.get(sentence)
            ground_truth.append(label if label else "?")

        matched = sum(1 for gt in ground_truth if gt != "?")
        print(f"Matched {matched}/{len(test_sentences)} test sentences to training data")

        if matched == 0:
            print("⚠️  Warning: No ground truth matched. Results will not show accuracy.")
            ground_truth = None

    # Limit samples if requested
    if args.max_samples:
        test_sentences = test_sentences[:args.max_samples]
        if ground_truth:
            ground_truth = ground_truth[:args.max_samples]
        print(f"Limiting evaluation to {args.max_samples} samples")

    print(f"Total test sentences: {len(test_sentences)}")

    # Evaluate
    results = evaluate_model(
        model=model,
        tokenizer=tokenizer,
        test_sentences=test_sentences,
        ground_truth_labels=ground_truth,
    )

    # Print confusion matrix analysis
    if "confusion_matrix" in results:
        print_confusion_matrix(results["confusion_matrix"])

    # Print summary
    print("\n" + "="*60)
    print("EVALUATION SUMMARY")
    print("="*60)
    print(f"Model: {args.base_model}")
    print(f"Adapter: {args.model_path}")
    print(f"\nPerformance:")
    print(f"  Total samples: {results['total']}")
    print(f"  Successfully predicted: {results['predicted']}")
    print(f"  Unparseable responses: {results['unparseable']}")
    print(f"  Parse rate: {results['predicted']/results['total']*100:.2f}%")

    if "accuracy" in results:
        print(f"\n  Overall Accuracy: {results['accuracy']*100:.2f}%")
        print(f"  Correct: {results['correct']}/{results['evaluated']}")
        print(f"\n💡 The model responds directly without the 2000+ token prompt!")

    # Save comprehensive results to JSON
    output = {
        "model_path": args.model_path,
        "base_model": args.base_model,
        "test_file": args.test_file,
        "train_data_file": args.train_data_file,
        "timestamp": str(Path(args.model_path).stat().st_mtime) if Path(args.model_path).exists() else None,
        "summary": {
            "total_samples": results["total"],
            "predicted": results["predicted"],
            "unparseable": results["unparseable"],
            "parse_rate": results["predicted"]/results["total"] if results["total"] > 0 else 0,
        },
        "predictions": results["predictions"],
    }

    # Add accuracy metrics if available
    if "accuracy" in results:
        output["summary"]["accuracy"] = results["accuracy"]
        output["summary"]["correct"] = results["correct"]
        output["summary"]["evaluated"] = results["evaluated"]

    # Add confusion matrix and language analysis
    if "confusion_matrix" in results:
        cm_data = results["confusion_matrix"]
        output["confusion_matrix"] = {
            "matrix_dict": cm_data["confusion_matrix"],  # Dict format for readability
            "matrix_2d": cm_data["confusion_matrix_2d"],  # 2D array for analysis
            "labels": cm_data["all_labels"],  # Label order for the 2D matrix
        }

        # Save ALL languages with their full statistics
        output["all_languages_accuracy"] = {
            lang: {
                "accuracy": stats["accuracy"],
                "correct": stats["correct"],
                "total": stats["total"],
                "error_examples": stats["errors"],
            }
            for lang, stats in cm_data["all_languages_sorted"]
        }

        # Also save per-language accuracy (same data, different format)
        output["per_language_accuracy"] = {
            lang: {
                "accuracy": stats["accuracy"],
                "correct": stats["correct"],
                "total": stats["total"],
                "error_examples": stats["errors"],
            }
            for lang, stats in cm_data["per_language_accuracy"].items()
        }

        # Save top 5 worst for quick reference
        output["worst_performing_languages"] = [
            {
                "language": lang,
                "accuracy": stats["accuracy"],
                "correct": stats["correct"],
                "total": stats["total"],
                "error_examples": stats["errors"],
            }
            for lang, stats in cm_data["worst_performing_languages"]
        ]

    # Save to file
    with open(args.output_file, "w", encoding="utf-8") as f:
        json.dump(output, f, indent=2, ensure_ascii=False)
    print(f"\n📁 Complete results saved to: {args.output_file}")
    print(f"   Includes: predictions, confusion matrix, per-language stats, error examples")


if __name__ == "__main__":
    main()

test_compare_empty.py

from compare import compare


def test_compare_empty_texts():
    result = compare(
        prompt_template="Classify: {text}",
        texts=[],
        teacher_labels={},
        eval_results=None,
        count_tokens=lambda s: max(1, len(s) // 4),
        token_method="test",
        num_examples=3,
    )
    assert result["teacher_input_avg"] == 0.0
    assert result["student_input_avg"] == 0.0
    assert result["input_token_reduction_pct"] == 0.0

test_create_data_empty.py

import asyncio

from create_data import generate_distillation_data


def test_generate_empty_input_file(tmp_path):
    # 空输入文件（0 个句子）：应干净地直接返回，不加载模型、不抛 IndexError/ZeroDivisionError
    input_file = tmp_path / "empty.txt"
    input_file.write_text("", encoding="utf-8")
    output_file = tmp_path / "out.jsonl"

    result = asyncio.run(
        generate_distillation_data(
            input_file=str(input_file),
            output_file=str(output_file),
            model_name="stub",
        )
    )
    assert result is None
    assert not output_file.exists()


def test_generate_blank_lines_only(tmp_path):
    # 只有空白行的文件同样视为空
    input_file = tmp_path / "blank.txt"
    input_file.write_text("\n   \n\t\n", encoding="utf-8")
    output_file = tmp_path / "out.jsonl"

    result = asyncio.run(
        generate_distillation_data(
            input_file=str(input_file),
            output_file=str(output_file),
            model_name="stub",
        )
    )
    assert result is None
    assert not output_file.exists()

train_sft_trl.py

"""
Prompt Distillation Training using Hugging Face TRL

This script trains a student model using the TRL SFTTrainer instead of verl.
TRL is more widely used, better documented, and easier to work with.

Based on the same prompt distillation methodology but using standard HF tools.
"""

import argparse
import json
import os
from pathlib import Path

import torch
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

# 注意：trl 的 SFTTrainer / SFTConfig 在 train_model() 内部按需导入，
# 这样即便未安装 trl（如离线查看 --help 时）也能正常展示命令行帮助。


def load_jsonl_dataset(file_path: str) -> Dataset:
    """
    Load training data from JSONL file.

    Args:
        file_path: Path to JSONL file with messages format

    Returns:
        Dataset: Hugging Face Dataset object
    """
    local_rank = int(os.environ.get("LOCAL_RANK", 0))

    if local_rank == 0:
        print(f"Loading dataset from: {file_path}")

    data = []
    with open(file_path, "r", encoding="utf-8") as f:
        for line in f:
            if line.strip():
                data.append(json.loads(line))

    if local_rank == 0:
        print(f"Loaded {len(data)} training examples")

        # Show sample
        if data:
            print(f"\nSample data:")
            print(f"  Messages: {data[0]['messages']}")

    # Convert to HF Dataset
    dataset = Dataset.from_list(data)

    return dataset


def prepare_model_and_tokenizer(model_name: str, use_lora: bool = True, 
                                lora_rank: int = 32, lora_alpha: int = 16):
    """
    Load model and tokenizer, optionally with LoRA.

    Args:
        model_name: Model name or path
        use_lora: Whether to use LoRA for efficient training
        lora_rank: LoRA rank
        lora_alpha: LoRA alpha parameter

    Returns:
        tuple: (model, tokenizer, peft_config or None)
    """
    local_rank = int(os.environ.get("LOCAL_RANK", 0))

    if local_rank == 0:
        print(f"\n{'='*80}")
        print(f"Loading Model and Tokenizer")
        print(f"{'='*80}")
        print(f"Model: {model_name}")
        print(f"LoRA: {'Enabled' if use_lora else 'Disabled'}")

    # Load tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

    # Set pad token if not exists
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
        tokenizer.pad_token_id = tokenizer.eos_token_id

    if local_rank == 0:
        print(f"Tokenizer loaded: vocab_size={len(tokenizer)}")

    # Load model
    # Note: Don't use device_map='auto' in distributed training - let DDP/FSDP handle device placement
    model_kwargs = {
        "torch_dtype": torch.bfloat16,
        "trust_remote_code": True,
        "use_cache": False,  # Disable for training
    }

    # Only use device_map for single GPU (non-distributed)
    if local_rank == -1 or int(os.environ.get("WORLD_SIZE", "1")) == 1:
        model_kwargs["device_map"] = "auto"

    if local_rank == 0:
        print(f"Loading model (this may take a few minutes)...")
    model = AutoModelForCausalLM.from_pretrained(model_name, **model_kwargs)

    if local_rank == 0:
        print(f"Model loaded successfully!")
        print(f"  Parameters: {model.num_parameters() / 1e9:.2f}B")

    # Configure LoRA if enabled
    peft_config = None
    if use_lora:
        if local_rank == 0:
            print(f"\nConfiguring LoRA:")
            print(f"  Rank: {lora_rank}")
            print(f"  Alpha: {lora_alpha}")

        peft_config = LoraConfig(
            r=lora_rank,
            lora_alpha=lora_alpha,
            target_modules="all-linear",
            lora_dropout=0.0,
            bias="none",
            task_type="CAUSAL_LM",
        )

        model = get_peft_model(model, peft_config)

        # Print trainable parameters
        if local_rank == 0:
            trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
            total_params = sum(p.numel() for p in model.parameters())
            trainable_percent = 100 * trainable_params / total_params

            print(f"\nTrainable Parameters:")
            print(f"  Trainable: {trainable_params:,} ({trainable_percent:.2f}%)")
            print(f"  Total: {total_params:,}")

    return model, tokenizer, peft_config


def train_model(
    model,
    tokenizer,
    train_dataset,
    output_dir: str,
    num_train_epochs: int = 1,
    per_device_train_batch_size: int = 4,
    gradient_accumulation_steps: int = 4,
    learning_rate: float = 2e-4,
    max_length: int = 2048,
    warmup_ratio: float = 0.03,
    logging_steps: int = 1,
    save_strategy: str = "epoch",
    lr_scheduler_type: str = "cosine_with_min_lr",
    report_to: str = "wandb",
    run_name: str = None,
):
    """
    Train the model using TRL SFTTrainer.

    Hyperparameters are based on the OpenAI Cookbook gpt-oss-20b example,
    which provides good defaults for efficient fine-tuning.

    Args:
        model: The model to train
        tokenizer: The tokenizer
        train_dataset: Training dataset
        output_dir: Output directory for checkpoints
        num_train_epochs: Number of training epochs (default: 1, matching OpenAI)
        per_device_train_batch_size: Batch size per device (default: 4, matching OpenAI)
        gradient_accumulation_steps: Gradient accumulation steps (default: 4, matching OpenAI)
        learning_rate: Learning rate (default: 2e-4, matching OpenAI)
        max_length: Maximum sequence length (default: 2048, matching OpenAI)
        warmup_ratio: Warmup ratio (default: 0.03, matching OpenAI)
        logging_steps: Steps between logging (default: 1, matching OpenAI)
        save_strategy: When to save checkpoints
        lr_scheduler_type: Learning rate scheduler type (default: cosine_with_min_lr, matching OpenAI)
        report_to: Where to report metrics (default: wandb)
        run_name: Custom run name for logging (default: auto-generated)

    Returns:
        SFTTrainer: The trained trainer object
    """
    from trl import SFTTrainer, SFTConfig

    local_rank = int(os.environ.get("LOCAL_RANK", 0))

    if local_rank == 0:
        print(f"\n{'='*80}")
        print(f"Training Configuration")
        print(f"{'='*80}")

    # Calculate effective batch size
    world_size = torch.cuda.device_count() if torch.cuda.is_available() else 1
    effective_batch_size = per_device_train_batch_size * gradient_accumulation_steps * world_size

    # Detect distributed mode
    distributed_mode = "Single GPU"
    if int(os.environ.get("WORLD_SIZE", "1")) > 1:
        if os.environ.get("ACCELERATE_USE_FSDP", "false").lower() == "true":
            distributed_mode = "FSDP (Fully Sharded Data Parallel)"
        else:
            distributed_mode = "DDP (Distributed Data Parallel)"

    if local_rank == 0:
        print(f"Training Parameters:")
        print(f"  Output directory: {output_dir}")
        print(f"  Distributed mode: {distributed_mode}")
        print(f"  Epochs: {num_train_epochs}")
        print(f"  Per-device batch size: {per_device_train_batch_size}")
        print(f"  Gradient accumulation steps: {gradient_accumulation_steps}")
        print(f"  Number of GPUs: {world_size}")
        print(f"  Effective batch size: {effective_batch_size}")
        print(f"  Learning rate: {learning_rate}")
        print(f"  LR scheduler: {lr_scheduler_type}")
        print(f"  Warmup ratio: {warmup_ratio}")
        print(f"  Max length: {max_length}")
        print(f"  Logging steps: {logging_steps}")
        print(f"  Save strategy: {save_strategy}")

        # Show memory advantage for FSDP
        if "FSDP" in distributed_mode:
            print(f"\n  💡 FSDP Mode: Each GPU holds ~{100/world_size:.1f}% of the model")

    # Training configuration (matching OpenAI Cookbook gpt-oss-20b example)
    training_args = SFTConfig(
        output_dir=output_dir,
        num_train_epochs=num_train_epochs,
        per_device_train_batch_size=per_device_train_batch_size,
        gradient_accumulation_steps=gradient_accumulation_steps,
        learning_rate=learning_rate,
        max_length=max_length,  # Note: Use max_length, not max_seq_length
        warmup_ratio=warmup_ratio,
        lr_scheduler_type=lr_scheduler_type,
        lr_scheduler_kwargs={"min_lr_rate": 0.1} if lr_scheduler_type == "cosine_with_min_lr" else {},
        logging_steps=logging_steps,
        save_strategy=save_strategy,
        save_total_limit=2,  # Keep only last 2 checkpoints
        gradient_checkpointing=True,  # Save memory (matching OpenAI)
        bf16=torch.cuda.is_available(),  # Use bfloat16 if available
        logging_first_step=True,
        report_to=report_to,  # wandb, tensorboard, or none
        run_name=run_name or f"prompt-distillation-{num_train_epochs}epoch",
        remove_unused_columns=False,
        dataset_text_field="",  # We'll use formatting function
        dataset_kwargs={
            "skip_prepare_dataset": False,
        },
    )

    if local_rank == 0:
        if report_to == "wandb":
            print(f"\n  📊 Logging to Weights & Biases (wandb)")
            print(f"     Run name: {run_name or f'prompt-distillation-{num_train_epochs}epoch'}")
            print(f"     View at: https://wandb.ai")
        elif report_to == "tensorboard":
            print(f"\n  📊 Logging to TensorBoard")
            print(f"     View with: tensorboard --logdir {output_dir}")
        else:
            print(f"\n  📊 Logging disabled (report_to=none)")

    # Initialize trainer
    if local_rank == 0:
        print(f"\n{'='*80}")
        print(f"Initializing SFTTrainer")
        print(f"{'='*80}")

    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        processing_class=tokenizer,
        formatting_func=lambda x: tokenizer.apply_chat_template(
            x["messages"],
            tokenize=False,
            add_generation_prompt=False,
        ),
    )

    # Start training
    if local_rank == 0:
        print(f"\n{'='*80}")
        print(f"Starting Training")
        print(f"{'='*80}")

    trainer.train()

    if local_rank == 0:
        print(f"\n{'='*80}")
        print(f"Training Complete!")
        print(f"{'='*80}")

    return trainer


def main():
    parser = argparse.ArgumentParser(
        description="用 Hugging Face TRL 训练 Prompt 蒸馏学生模型（无提示直接作答）",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )

    # Data arguments
    parser.add_argument(
        "--train_file",
        type=str,
        default="./data/prompt_distillation_lang.jsonl",
        help="训练数据路径（JSONL 格式）",
    )

    # Model arguments
    parser.add_argument(
        "--model_name",
        type=str,
        default="Qwen/Qwen3-30B-A3B-Instruct-2507",
        help="学生基座模型名称或路径（用于蒸馏的非思考型模型）",
    )
    parser.add_argument(
        "--output_dir",
        type=str,
        default="./models/prompt_distillation_trl",
        help="模型 checkpoint 的输出目录",
    )

    # LoRA arguments
    parser.add_argument(
        "--use_lora",
        action="store_true",
        default=True,
        help="使用 LoRA 做参数高效微调",
    )
    parser.add_argument(
        "--lora_rank",
        type=int,
        default=32,
        help="LoRA rank（默认 32，与 tinker 一致）",
    )
    parser.add_argument(
        "--lora_alpha",
        type=int,
        default=16,
        help="LoRA alpha 参数（默认 16）",
    )

    # Training arguments
    parser.add_argument(
        "--num_train_epochs",
        type=int,
        default=1,
        help="训练轮数（默认 1，与 OpenAI 教程一致）",
    )
    parser.add_argument(
        "--per_device_train_batch_size",
        type=int,
        default=4,
        help="每张 GPU 的批次大小（默认 4，与 OpenAI 教程一致）",
    )
    parser.add_argument(
        "--gradient_accumulation_steps",
        type=int,
        default=4,
        help="梯度累积步数（默认 4，与 OpenAI 教程一致）",
    )
    parser.add_argument(
        "--learning_rate",
        type=float,
        default=2e-4,
        help="学习率（默认 2e-4，与 OpenAI 教程一致）",
    )
    parser.add_argument(
        "--max_length",
        type=int,
        default=2048,
        help="最大序列长度（默认 2048，与 OpenAI 教程一致）",
    )
    parser.add_argument(
        "--warmup_ratio",
        type=float,
        default=0.03,
        help="warmup 比例（默认 0.03，与 OpenAI 教程一致）",
    )
    parser.add_argument(
        "--lr_scheduler_type",
        type=str,
        default="cosine_with_min_lr",
        help="学习率调度器类型（默认 cosine_with_min_lr，与 OpenAI 教程一致）",
    )

    # Logging arguments
    parser.add_argument(
        "--report_to",
        type=str,
        default="wandb",
        choices=["wandb", "tensorboard", "none"],
        help="训练指标上报目标（默认 wandb）",
    )
    parser.add_argument(
        "--run_name",
        type=str,
        default=None,
        help="wandb 运行名称（未提供则自动生成）",
    )

    args = parser.parse_args()

    # Print configuration
    print(f"{'='*80}")
    print(f"Prompt Distillation Training with TRL")
    print(f"{'='*80}")
    print(f"Configuration:")
    print(f"  Train file: {args.train_file}")
    print(f"  Model: {args.model_name}")
    print(f"  Output dir: {args.output_dir}")
    print(f"  LoRA: {args.use_lora}")
    if args.use_lora:
        print(f"    - Rank: {args.lora_rank}")
        print(f"    - Alpha: {args.lora_alpha}")
    print(f"  Epochs: {args.num_train_epochs}")
    print(f"  Batch size: {args.per_device_train_batch_size}")
    print(f"  Gradient accumulation: {args.gradient_accumulation_steps}")
    print(f"  Learning rate: {args.learning_rate}")
    print(f"  Max length: {args.max_length}")
    print(f"  LR scheduler: {args.lr_scheduler_type}")
    print(f"  Warmup ratio: {args.warmup_ratio}")
    print(f"{'='*80}\n")

    # Check if training file exists
    if not os.path.exists(args.train_file):
        raise FileNotFoundError(f"Training file not found: {args.train_file}")

    # Create output directory
    os.makedirs(args.output_dir, exist_ok=True)

    # Load dataset
    train_dataset = load_jsonl_dataset(args.train_file)

    # Load model and tokenizer
    model, tokenizer, peft_config = prepare_model_and_tokenizer(
        args.model_name,
        use_lora=args.use_lora,
        lora_rank=args.lora_rank,
        lora_alpha=args.lora_alpha,
    )

    # Train model
    trainer = train_model(
        model=model,
        tokenizer=tokenizer,
        train_dataset=train_dataset,
        output_dir=args.output_dir,
        num_train_epochs=args.num_train_epochs,
        per_device_train_batch_size=args.per_device_train_batch_size,
        gradient_accumulation_steps=args.gradient_accumulation_steps,
        learning_rate=args.learning_rate,
        max_length=args.max_length,
        warmup_ratio=args.warmup_ratio,
        lr_scheduler_type=args.lr_scheduler_type,
        report_to=args.report_to,
        run_name=args.run_name,
    )

    # Save final model (only main process)
    local_rank = int(os.environ.get("LOCAL_RANK", 0))

    if local_rank == 0:
        print(f"\nSaving final model to: {args.output_dir}")
    trainer.save_model(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)

    if local_rank == 0:
        print(f"\n{'='*80}")
        print(f"Training Complete!")
        print(f"{'='*80}")
        print(f"Model saved to: {args.output_dir}")
        print(f"\nTo use the model:")
        print(f"  from transformers import AutoModelForCausalLM, AutoTokenizer")
        print(f"  from peft import PeftModel")
        print(f"  ")
        print(f"  tokenizer = AutoTokenizer.from_pretrained('{args.output_dir}')")
        print(f"  model = AutoModelForCausalLM.from_pretrained('{args.model_name}')")
        print(f"  model = PeftModel.from_pretrained(model, '{args.output_dir}')")
        print(f"{'='*80}\n")


if __name__ == "__main__":
    main()

create_data_h100x8.sh

#!/bin/bash
# Parallel data generation script for H100x8 GPU setup
# This uses ALL 8 GPUs by running 2 instances in parallel

set -e

echo "=========================================="
echo "Parallel Data Generation for H100x8"
echo "=========================================="
echo ""

# Configuration
INPUT_FILE=${1:-"./example-data/multilingual.txt"}
OUTPUT_FILE=${2:-"./data/prompt_distillation_lang.jsonl"}
MODEL_NAME="Qwen/Qwen3-30B-A3B-Thinking-2507"

echo "Configuration:"
echo "  Input file: $INPUT_FILE"
echo "  Output file: $OUTPUT_FILE"
echo "  Model: $MODEL_NAME"
echo "  Strategy: 2 parallel instances, each using TP=4"
echo ""

if [ ! -f "$INPUT_FILE" ]; then
    echo "❌ Input file not found: $INPUT_FILE"
    exit 1
fi

# Check if output file exists
if [ -f "$OUTPUT_FILE" ]; then
    echo "⚠️  Output file already exists: $OUTPUT_FILE"
    read -p "Do you want to overwrite it? (y/N): " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
        echo "Aborted."
        exit 0
    fi
    rm "$OUTPUT_FILE"
fi

# Create temp directory for split files
TEMP_DIR="./data/temp_$$"
mkdir -p "$TEMP_DIR"

echo "Splitting dataset into 2 parts..."
TOTAL_LINES=$(wc -l < "$INPUT_FILE")
HALF_LINES=$((TOTAL_LINES / 2))

head -n $HALF_LINES "$INPUT_FILE" > "$TEMP_DIR/part1.txt"
tail -n +$((HALF_LINES + 1)) "$INPUT_FILE" > "$TEMP_DIR/part2.txt"

PART1_LINES=$(wc -l < "$TEMP_DIR/part1.txt")
PART2_LINES=$(wc -l < "$TEMP_DIR/part2.txt")

echo "  Part 1: $PART1_LINES lines (GPU 0-3)"
echo "  Part 2: $PART2_LINES lines (GPU 4-7)"
echo ""

# Setup signal handler to kill both processes on Ctrl+C
cleanup() {
    echo ""
    echo "🛑 Caught interrupt signal (Ctrl+C)"
    echo "Killing both processes..."
    if [ ! -z "$PID1" ] && kill -0 $PID1 2>/dev/null; then
        echo "  Killing Instance 1 (PID $PID1)..."
        kill -TERM $PID1 2>/dev/null || true
    fi
    if [ ! -z "$PID2" ] && kill -0 $PID2 2>/dev/null; then
        echo "  Killing Instance 2 (PID $PID2)..."
        kill -TERM $PID2 2>/dev/null || true
    fi
    # Wait a moment for graceful shutdown
    sleep 2
    # Force kill if still running
    if [ ! -z "$PID1" ] && kill -0 $PID1 2>/dev/null; then
        echo "  Force killing Instance 1..."
        kill -9 $PID1 2>/dev/null || true
    fi
    if [ ! -z "$PID2" ] && kill -0 $PID2 2>/dev/null; then
        echo "  Force killing Instance 2..."
        kill -9 $PID2 2>/dev/null || true
    fi
    # Cleanup temp files
    echo "  Cleaning up temporary files..."
    rm -rf "$TEMP_DIR"
    echo "✅ Cleanup complete"
    exit 130
}

trap cleanup SIGINT SIGTERM

# Run both instances in parallel
echo "Starting parallel data generation..."
echo ""

# Instance 1: GPU 0-3
echo "Starting Instance 1 on GPUs 0-3..."
CUDA_VISIBLE_DEVICES=0,1,2,3 python create_data.py \
    --input_file "$TEMP_DIR/part1.txt" \
    --output_file "$TEMP_DIR/part1.jsonl" \
    --model_name "$MODEL_NAME" \
    --temperature 0.15 \
    --tensor_parallel_size 4 \
    --max_retries 3 \
    &
PID1=$!

echo "Instance 1 is running (PID: $PID1)"

# Instance 2: GPU 4-7
echo "Starting Instance 2 on GPUs 4-7..."
CUDA_VISIBLE_DEVICES=4,5,6,7 python create_data.py \
    --input_file "$TEMP_DIR/part2.txt" \
    --output_file "$TEMP_DIR/part2.jsonl" \
    --model_name "$MODEL_NAME" \
    --temperature 0.15 \
    --tensor_parallel_size 4 \
    --max_retries 3 \
    &
PID2=$!

echo "Instance 2 is running (PID: $PID2)"
echo ""
echo "Both instances are running. You can monitor GPU usage with:"
echo "  watch -n 1 nvidia-smi"
echo ""

# Wait for both to complete
echo "Waiting for both instances to complete..."
echo "  Instance 1 (PID $PID1): GPU 0-3"
echo "  Instance 2 (PID $PID2): GPU 4-7"
echo ""

wait $PID1
STATUS1=$?
echo ""
echo "Instance 1 completed with status: $STATUS1"
if [ $STATUS1 -ne 0 ]; then
    echo "Instance 1 log:"
    cat "$TEMP_DIR/instance1.log" | tail -50
fi

wait $PID2
STATUS2=$?
echo ""
echo "Instance 2 completed with status: $STATUS2"
if [ $STATUS2 -ne 0 ]; then
    echo "Instance 2 log:"
    cat "$TEMP_DIR/instance2.log" | tail -50
fi

echo ""

# Check if both succeeded
if [ $STATUS1 -ne 0 ] || [ $STATUS2 -ne 0 ]; then
    echo "❌ One or both instances failed!"
    echo "   Instance 1 status: $STATUS1"
    echo "   Instance 2 status: $STATUS2"
    rm -rf "$TEMP_DIR"
    exit 1
fi

# Combine results
echo "Combining results..."
cat "$TEMP_DIR/part1.jsonl" "$TEMP_DIR/part2.jsonl" > "$OUTPUT_FILE"

# Show statistics
PART1_COUNT=$(wc -l < "$TEMP_DIR/part1.jsonl")
PART2_COUNT=$(wc -l < "$TEMP_DIR/part2.jsonl")
TOTAL_COUNT=$((PART1_COUNT + PART2_COUNT))

echo ""
echo "=========================================="
echo "✅ Parallel data generation complete!"
echo "=========================================="
echo "Part 1: $PART1_COUNT / $PART1_LINES samples ($(awk "BEGIN {printf \"%.2f\", $PART1_COUNT/$PART1_LINES*100}")%)"
echo "Part 2: $PART2_COUNT / $PART2_LINES samples ($(awk "BEGIN {printf \"%.2f\", $PART2_COUNT/$PART2_LINES*100}")%)"
echo "Total:  $TOTAL_COUNT / $TOTAL_LINES samples ($(awk "BEGIN {printf \"%.2f\", $TOTAL_COUNT/$TOTAL_LINES*100}")%)"
echo ""
echo "Output: $OUTPUT_FILE"
echo ""

# Cleanup
rm -rf "$TEMP_DIR"

echo "All 8 GPUs were utilized! 🚀"

train_trl.sh

#!/bin/bash
# Training script using Hugging Face TRL (more common and easier to use than verl)
#
# This script uses the widely-adopted TRL SFTTrainer instead of verl.
# Benefits:
# - Works directly with JSONL (no parquet conversion needed)
# - Better documentation and community support
# - Simpler setup and fewer dependencies
# - Standard Hugging Face workflow

set -x

# Default configuration
MODEL_NAME=${1:-"Qwen/Qwen3-30B-A3B-Instruct-2507"}
OUTPUT_DIR=${2:-"./models/prompt_distillation_trl"}
TRAIN_FILE=${3:-"./data/prompt_distillation_lang.jsonl"}

echo "============================================"
echo "Prompt Distillation Training with TRL"
echo "============================================"
echo "Model: $MODEL_NAME"
echo "Output: $OUTPUT_DIR"
echo "Train file: $TRAIN_FILE"
echo "============================================"
echo ""

# Check if training file exists
if [ ! -f "$TRAIN_FILE" ]; then
    echo "❌ Training file not found: $TRAIN_FILE"
    echo "Please run data generation first:"
    echo "  python create_data.py"
    exit 1
fi

# Run training with OpenAI-style hyperparameters
python train_sft_trl.py \
    --model_name "$MODEL_NAME" \
    --output_dir "$OUTPUT_DIR" \
    --train_file "$TRAIN_FILE" \
    --use_lora \
    --lora_rank 32 \
    --lora_alpha 16 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --learning_rate 2e-4 \
    --max_length 2048 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type cosine_with_min_lr

echo ""
echo "============================================"
echo "✅ Training Complete!"
echo "============================================"
echo "Model saved to: $OUTPUT_DIR"

evaluation_results.json

{
  "model_path": "./models/prompt_distillation_trl",
  "base_model": "Qwen/Qwen3-30B-A3B-Instruct-2507",
  "test_file": "./example-data/multilingual.txt",
  "train_data_file": "./data/prompt_distillation_lang.jsonl",
  "timestamp": "1759556861.0279787",
  "summary": {
    "total_samples": 2100,
    "predicted": 2100,
    "unparseable": 0,
    "parse_rate": 1.0,
    "accuracy": 0.9519047619047619,
    "correct": 1999,
    "evaluated": 2100
  },
  "predictions": [
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "hi",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "en",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "hi",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "hi",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "hi",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "es",
    "fr",
    "hi",
    "ru",
    "ot",
    "ot",
    "tr",
    "ur",
    "vi",
    "zh",
    "ar",
    "ru",
    "de",
    "el",
    "en",
    "ar",
    "ur",
    "ar",
    "ur",
    "ar",
    "ur",
    "ar",
    "ur",
    "ar",
    "ur",
    "el",
    "el",
    "el",
    "el",
    "el",
    "el",
    "el",
    "el",
    "el",
    "el",
    "ru",
    "ru",
    "ru",
    "ru",
    "ru",
    "ru",
    "ru",
    "ru",
    "ru",
    "ru",
    "zh",
    "zh",
    "zh",
    "zh",
    "zh",
    "zh",
    "zh",
    "zh",
    "zh",
    "zh",
    "fr",
    "fr",
    "fr",
    "es",
    "es",
    "es",
    "de",
    "de",
    "de",
    "tr",
    "tr",
    "tr",
    "vi",
    "vi",
    "vi",
    "en",
    "en",
    "en",
    "en",
    "en",
    "en",
    "en",
    "en",
    "en",
    "en",
    "hi",
    "hi",
    "hi",
    "hi",
    "hi",
    "ur",
    "ar",
    "ur",
    "ar",
    "ur",
    "en",
    "en",
    "fr",
    "es",
    "zh",
    "de",
    "tr",
    "vi",
    "ru",
    "zh",
    "en",
    "zh",
    "ru",
    "en",
    "zh",
    "fr",
    "de",
    "ru",
    "vi",
    "hi",
    "en",
    "en",
    "fr",
    "es",
    "hi"
  ],
  "confusion_matrix": {
    "matrix_dict": {
      "ar": {
        "ar": 141,
        "ur": 5
      },
      "ru": {
        "ru": 279
      },
      "de": {
        "de": 135,
        "tr": 1,
        "es": 1
      },
      "el": {
        "el": 144,
        "ot": 10,
        "hi": 3
      },
      "en": {
        "en": 146,
        "es": 3
      },
      "fr": {
        "fr": 139,
        "tr": 3,
        "en": 1
      },
      "hi": {
        "hi": 171,
        "ot": 39,
        "zh": 1
      },
      "ot": {
        "ot": 169,
        "hi": 14,
        "en": 4,
        "zh": 3,
        "ru": 1,
        "de": 1,
        "vi": 1
      },
      "ur": {
        "ur": 133,
        "hi": 1
      },
      "vi": {
        "vi": 134
      },
      "zh": {
        "zh": 143,
        "ot": 1
      },
      "es": {
        "es": 133,
        "de": 3
      },
      "tr": {
        "tr": 132
      },
      "unknown": {
        "vi": 3,
        "tr": 1,
        "ot": 1
      }
    },
    "matrix_2d": [
      [
        141,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        5,
        0,
        0
      ],
      [
        0,
        135,
        0,
        0,
        1,
        0,
        0,
        0,
        0,
        1,
        0,
        0,
        0,
        0
      ],
      [
        0,
        0,
        144,
        0,
        0,
        0,
        3,
        10,
        0,
        0,
        0,
        0,
        0,
        0
      ],
      [
        0,
        0,
        0,
        146,
        3,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ],
      [
        0,
        3,
        0,
        0,
        133,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ],
      [
        0,
        0,
        0,
        1,
        0,
        139,
        0,
        0,
        0,
        3,
        0,
        0,
        0,
        0
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        171,
        39,
        0,
        0,
        0,
        0,
        0,
        1
      ],
      [
        0,
        1,
        0,
        4,
        0,
        0,
        14,
        169,
        1,
        0,
        0,
        0,
        1,
        3
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        279,
        0,
        0,
        0,
        0,
        0
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        132,
        0,
        0,
        0,
        0
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        1,
        0,
        1,
        0,
        0,
        3,
        0
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        1,
        0,
        0,
        0,
        0,
        133,
        0,
        0
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        134,
        0
      ],
      [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        1,
        0,
        0,
        0,
        0,
        0,
        143
      ]
    ],
    "labels": [
      "ar",
      "de",
      "el",
      "en",
      "es",
      "fr",
      "hi",
      "ot",
      "ru",
      "tr",
      "unknown",
      "ur",
      "vi",
      "zh"
    ]
  },
  "all_languages_accuracy": {
    "unknown": {
      "accuracy": 0.0,
      "correct": 0,
      "total": 5,
      "error_examples": [
        {
          "sentence": "Vì vậy, cô ấy giống như, à nhìn đi, trong mong vào công ty như vậy.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1078
        },
        {
          "sentence": "Thế là, à, tôi, ừ, dù sao, ừ, ừ, đây là ba, ừ, phi công U2, ừ, văn phòng Tổng thống Kennedy ở Washington với Đại tướng May.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1138
        },
        {
          "sentence": "1880'li bir tarihte doğdu, 188 gibi, sanırım 1889'du, galiba doğduğu zaman buydu.",
          "predicted": "tr",
          "ground_truth": "unknown",
          "index": 1466
        },
        {
          "sentence": "Nilipata maagizo ya kwenda Del Rio, TX, hivyo wakati nilipofika huko, niligundua nilipaswa kwenda kwa Laughlin Air Force Base.",
          "predicted": "ot",
          "ground_truth": "unknown",
          "index": 1494
        },
        {
          "sentence": "Và tôi đã đến, tôi đã đến Washington D.C. và tôi đã không đi thẳng đến, uh, chỗ đó, uh, nơi họ bảo tôi đến trong các đơn hàng.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1858
        }
      ]
    },
    "hi": {
      "accuracy": 0.8104265402843602,
      "correct": 171,
      "total": 211,
      "error_examples": [
        {
          "sentence": "และเขาพูดว่า, ม่าม๊า ผมอยู่บ้าน",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 10
        },
        {
          "sentence": "มันมีอีกมากที่คุณสามารถพูดคุยเกี่ยวกับสิ่งนั้น ฉันจะข้ามไปละกัน",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 70
        },
        {
          "sentence": "และฉันก็แบบว่าตอบตกลงและมันก็เท่านั่น!",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 130
        },
        {
          "sentence": "ฉันก็เลยไม่แน่ใจจริงๆ ว่าทำไม",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 145
        },
        {
          "sentence": "และฉันได้ใส่ เอ่อ, ห้ากองออกมาจาก ยู2",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 205
        }
      ]
    },
    "ot": {
      "accuracy": 0.8756476683937824,
      "correct": 169,
      "total": 193,
      "error_examples": [
        {
          "sentence": "ฉันไม่รู้ว่าฉันไปเพื่ออะไรหรือเพื่อสิ่งใด ดังนั้นแค่รายงานเกี่ยวกับสถานที่ที่ระบุในวอชิงตัน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 85
        },
        {
          "sentence": "วันนี้เขาจะพูดคุยกับเราเกี่ยวกับ Third SS, U2 Quick และ Blackbird",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 340
        },
        {
          "sentence": "เธอกล่าวว่ามีน้ำตาไหลออกมาจากตาของเธอ และเธอกล่าวว่าโจมาปรากฏตัวที่ชานบ้าน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 385
        },
        {
          "sentence": "เมื่อฉันดึง เมื่อเขาดึงกระโจมเพื่อให้ฉันดึงเขาออกไป เขาชี้ไปที่อุปกรณ์ทั้งสองด้านซ้ายมือของเครื่องบินที่ละลายจริง ๆ ระหว่างการบิน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 505
        },
        {
          "sentence": "แต่ทันทีทันใดนั้น เราก็ถูกเรียกออกไปดูว่ามีอะไรบินอยู่",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 790
        }
      ]
    },
    "el": {
      "accuracy": 0.9171974522292994,
      "correct": 144,
      "total": 157,
      "error_examples": [
        {
          "sentence": "ดี, ฉันไม่ได้คิดอะไรเกี่ยวกับเรื่องนี้, แต่ฉันก็ผิดหวัง, และ, ฉันก็กลับไปคุยกับเขาอีกครั้ง",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 25
        },
        {
          "sentence": "พวกเขาบอกฉันว่าเขาจะเรียกคน ๆ หนึ่งเข้ามาในตอนท้ายให้ฉันพบ",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 55
        },
        {
          "sentence": "และย่าเคยเล่าเรื่องเกี่ยวที่น้องสาวของเธอและสามีของน้องสาวได้ตัดสินใจว่าพวกเขาจะย้ายไปที่เมืองออร์กัสต้าและมองว่าสีขาว",
          "predicted": "hi",
          "ground_truth": "el",
          "index": 250
        },
        {
          "sentence": "แล้วความจริงก็คือ เธอเป็นความสว่าง!",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 775
        },
        {
          "sentence": "ยายของฉันเคยเล่าเรื่องต่าง ๆ มากมายเกี่ยวกับปีเธอที่เติบโตขึ้นมา และโดยเฉพาะอย่างยิ่งเธอเคยพูดเรื่องครอบครัวของเธอและเล่าว่าครอบครัวเธอเป็นอย่างไรในช่วงเวลานั้น",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 1015
        }
      ]
    },
    "ar": {
      "accuracy": 0.9657534246575342,
      "correct": 141,
      "total": 146,
      "error_examples": [
        {
          "sentence": "U2 (یو 2) کی پرواز شروع کرنے یا پریشر سوٹ کے ساتھ پرواز کرنے سے بھی پہلے انہیں اونچائی کے چیمبر کی کئی سواریوں/پروازوں سے گزرنا پڑتا ہے.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1242
        },
        {
          "sentence": "'پچتھر سال میں یہ پہلی بار ہوا ہے کہ' ٹی ایکس آۂین نے فوج کو ووٹ دی، ' ٹی ایکس' سفیر ہوتے ہوے  ' ٹی ایکس' سفیر چاھۂے تھے۔",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1287
        },
        {
          "sentence": "میرا مطلب یہ تھا کہ پوری بات.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1557
        },
        {
          "sentence": "وہ کیوبا کے بحران کا واحد زخمی تھا، اور uh، کیسر uh، وہ تصاویر ملی اور واشنگٹن میں اینڈریو ایئر فورس کی طرف براہ راست اڑ گئے.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1767
        },
        {
          "sentence": "خوش آمدید",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 2007
        }
      ]
    },
    "fr": {
      "accuracy": 0.972027972027972,
      "correct": 139,
      "total": 143,
      "error_examples": [
        {
          "sentence": "Ve Anne, evdeyim dedi.",
          "predicted": "tr",
          "ground_truth": "fr",
          "index": 11
        },
        {
          "sentence": "Nereye gittiklerini bilmiyorduk.",
          "predicted": "tr",
          "ground_truth": "fr",
          "index": 1376
        },
        {
          "sentence": "Nereye gittiklerini kimse bilmiyordu.",
          "predicted": "tr",
          "ground_truth": "fr",
          "index": 1661
        },
        {
          "sentence": "Well, there's nobody there to help me.",
          "predicted": "en",
          "ground_truth": "fr",
          "index": 1744
        }
      ]
    },
    "es": {
      "accuracy": 0.9779411764705882,
      "correct": 133,
      "total": 136,
      "error_examples": [
        {
          "sentence": "„Und ich dachte, das wäre ein Privileg, und das ist es immer noch, das ist esi mmer noch. Ich war der einzige 229 Ex-O, was mein AFCF Air Force-Karriere-Feld war.“",
          "predicted": "de",
          "ground_truth": "es",
          "index": 32
        },
        {
          "sentence": "Nun es kam soweit, dass zwei oder drei Flugzeuge pro Woche ankamen und ich nicht wusste, wohin sie flogen.",
          "predicted": "de",
          "ground_truth": "es",
          "index": 452
        },
        {
          "sentence": "Es sind 30 oder 40 U2 Flugzeuge und wir haben begonnen Chinesische und britische Piloten, praktisch Piloten von Verbündeten auf der ganzen Welt, in ihnen zu trainieren",
          "predicted": "de",
          "ground_truth": "es",
          "index": 1052
        }
      ]
    },
    "en": {
      "accuracy": 0.9798657718120806,
      "correct": 146,
      "total": 149,
      "error_examples": [
        {
          "sentence": "Y él dijo: Mamá, estoy en casa.",
          "predicted": "es",
          "ground_truth": "en",
          "index": 5
        },
        {
          "sentence": "Era en lo que teníamos a Rudolph Anderson, una formulación de tres U2.",
          "predicted": "es",
          "ground_truth": "en",
          "index": 695
        },
        {
          "sentence": "Así que, dice Bueno busca, busca esto en esta y esta empresa.",
          "predicted": "es",
          "ground_truth": "en",
          "index": 1070
        }
      ]
    },
    "de": {
      "accuracy": 0.9854014598540146,
      "correct": 135,
      "total": 137,
      "error_examples": [
        {
          "sentence": "Kariyerimin 16 yılını Özel Faaliyetler'de harcamama sıradışı denebilir.",
          "predicted": "tr",
          "ground_truth": "de",
          "index": 866
        },
        {
          "sentence": "Y entonces, una vez tienes todo introducido, puedes seguir a partir de ahí.",
          "predicted": "es",
          "ground_truth": "de",
          "index": 1160
        }
      ]
    },
    "ur": {
      "accuracy": 0.9925373134328358,
      "correct": 133,
      "total": 134,
      "error_examples": [
        {
          "sentence": "และแน่นอนว่า Androv Gromikov ไม่ได้ตอบอะไร แต่เรามีข้อมูลทั้งหมดจากภาพยนตร์ว่า U2 ที่ใช้ไปแล้ว",
          "predicted": "hi",
          "ground_truth": "ur",
          "index": 370
        }
      ]
    },
    "zh": {
      "accuracy": 0.9930555555555556,
      "correct": 143,
      "total": 144,
      "error_examples": [
        {
          "sentence": "แต่พวกมันถูกแบ่งประมาณว่าใครเป็นลูกจ้างซึ่งทำงานในท้องนาและใครเป็นคนทำงานบ้าน มันเป็นประเภทของ--",
          "predicted": "ot",
          "ground_truth": "zh",
          "index": 310
        }
      ]
    },
    "ru": {
      "accuracy": 1.0,
      "correct": 279,
      "total": 279,
      "error_examples": []
    },
    "tr": {
      "accuracy": 1.0,
      "correct": 132,
      "total": 132,
      "error_examples": []
    },
    "vi": {
      "accuracy": 1.0,
      "correct": 134,
      "total": 134,
      "error_examples": []
    }
  },
  "per_language_accuracy": {
    "ar": {
      "accuracy": 0.9657534246575342,
      "correct": 141,
      "total": 146,
      "error_examples": [
        {
          "sentence": "U2 (یو 2) کی پرواز شروع کرنے یا پریشر سوٹ کے ساتھ پرواز کرنے سے بھی پہلے انہیں اونچائی کے چیمبر کی کئی سواریوں/پروازوں سے گزرنا پڑتا ہے.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1242
        },
        {
          "sentence": "'پچتھر سال میں یہ پہلی بار ہوا ہے کہ' ٹی ایکس آۂین نے فوج کو ووٹ دی، ' ٹی ایکس' سفیر ہوتے ہوے  ' ٹی ایکس' سفیر چاھۂے تھے۔",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1287
        },
        {
          "sentence": "میرا مطلب یہ تھا کہ پوری بات.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1557
        },
        {
          "sentence": "وہ کیوبا کے بحران کا واحد زخمی تھا، اور uh، کیسر uh، وہ تصاویر ملی اور واشنگٹن میں اینڈریو ایئر فورس کی طرف براہ راست اڑ گئے.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1767
        },
        {
          "sentence": "خوش آمدید",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 2007
        }
      ]
    },
    "de": {
      "accuracy": 0.9854014598540146,
      "correct": 135,
      "total": 137,
      "error_examples": [
        {
          "sentence": "Kariyerimin 16 yılını Özel Faaliyetler'de harcamama sıradışı denebilir.",
          "predicted": "tr",
          "ground_truth": "de",
          "index": 866
        },
        {
          "sentence": "Y entonces, una vez tienes todo introducido, puedes seguir a partir de ahí.",
          "predicted": "es",
          "ground_truth": "de",
          "index": 1160
        }
      ]
    },
    "el": {
      "accuracy": 0.9171974522292994,
      "correct": 144,
      "total": 157,
      "error_examples": [
        {
          "sentence": "ดี, ฉันไม่ได้คิดอะไรเกี่ยวกับเรื่องนี้, แต่ฉันก็ผิดหวัง, และ, ฉันก็กลับไปคุยกับเขาอีกครั้ง",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 25
        },
        {
          "sentence": "พวกเขาบอกฉันว่าเขาจะเรียกคน ๆ หนึ่งเข้ามาในตอนท้ายให้ฉันพบ",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 55
        },
        {
          "sentence": "และย่าเคยเล่าเรื่องเกี่ยวที่น้องสาวของเธอและสามีของน้องสาวได้ตัดสินใจว่าพวกเขาจะย้ายไปที่เมืองออร์กัสต้าและมองว่าสีขาว",
          "predicted": "hi",
          "ground_truth": "el",
          "index": 250
        },
        {
          "sentence": "แล้วความจริงก็คือ เธอเป็นความสว่าง!",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 775
        },
        {
          "sentence": "ยายของฉันเคยเล่าเรื่องต่าง ๆ มากมายเกี่ยวกับปีเธอที่เติบโตขึ้นมา และโดยเฉพาะอย่างยิ่งเธอเคยพูดเรื่องครอบครัวของเธอและเล่าว่าครอบครัวเธอเป็นอย่างไรในช่วงเวลานั้น",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 1015
        }
      ]
    },
    "en": {
      "accuracy": 0.9798657718120806,
      "correct": 146,
      "total": 149,
      "error_examples": [
        {
          "sentence": "Y él dijo: Mamá, estoy en casa.",
          "predicted": "es",
          "ground_truth": "en",
          "index": 5
        },
        {
          "sentence": "Era en lo que teníamos a Rudolph Anderson, una formulación de tres U2.",
          "predicted": "es",
          "ground_truth": "en",
          "index": 695
        },
        {
          "sentence": "Así que, dice Bueno busca, busca esto en esta y esta empresa.",
          "predicted": "es",
          "ground_truth": "en",
          "index": 1070
        }
      ]
    },
    "es": {
      "accuracy": 0.9779411764705882,
      "correct": 133,
      "total": 136,
      "error_examples": [
        {
          "sentence": "„Und ich dachte, das wäre ein Privileg, und das ist es immer noch, das ist esi mmer noch. Ich war der einzige 229 Ex-O, was mein AFCF Air Force-Karriere-Feld war.“",
          "predicted": "de",
          "ground_truth": "es",
          "index": 32
        },
        {
          "sentence": "Nun es kam soweit, dass zwei oder drei Flugzeuge pro Woche ankamen und ich nicht wusste, wohin sie flogen.",
          "predicted": "de",
          "ground_truth": "es",
          "index": 452
        },
        {
          "sentence": "Es sind 30 oder 40 U2 Flugzeuge und wir haben begonnen Chinesische und britische Piloten, praktisch Piloten von Verbündeten auf der ganzen Welt, in ihnen zu trainieren",
          "predicted": "de",
          "ground_truth": "es",
          "index": 1052
        }
      ]
    },
    "fr": {
      "accuracy": 0.972027972027972,
      "correct": 139,
      "total": 143,
      "error_examples": [
        {
          "sentence": "Ve Anne, evdeyim dedi.",
          "predicted": "tr",
          "ground_truth": "fr",
          "index": 11
        },
        {
          "sentence": "Nereye gittiklerini bilmiyorduk.",
          "predicted": "tr",
          "ground_truth": "fr",
          "index": 1376
        },
        {
          "sentence": "Nereye gittiklerini kimse bilmiyordu.",
          "predicted": "tr",
          "ground_truth": "fr",
          "index": 1661
        },
        {
          "sentence": "Well, there's nobody there to help me.",
          "predicted": "en",
          "ground_truth": "fr",
          "index": 1744
        }
      ]
    },
    "hi": {
      "accuracy": 0.8104265402843602,
      "correct": 171,
      "total": 211,
      "error_examples": [
        {
          "sentence": "และเขาพูดว่า, ม่าม๊า ผมอยู่บ้าน",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 10
        },
        {
          "sentence": "มันมีอีกมากที่คุณสามารถพูดคุยเกี่ยวกับสิ่งนั้น ฉันจะข้ามไปละกัน",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 70
        },
        {
          "sentence": "และฉันก็แบบว่าตอบตกลงและมันก็เท่านั่น!",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 130
        },
        {
          "sentence": "ฉันก็เลยไม่แน่ใจจริงๆ ว่าทำไม",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 145
        },
        {
          "sentence": "และฉันได้ใส่ เอ่อ, ห้ากองออกมาจาก ยู2",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 205
        }
      ]
    },
    "ot": {
      "accuracy": 0.8756476683937824,
      "correct": 169,
      "total": 193,
      "error_examples": [
        {
          "sentence": "ฉันไม่รู้ว่าฉันไปเพื่ออะไรหรือเพื่อสิ่งใด ดังนั้นแค่รายงานเกี่ยวกับสถานที่ที่ระบุในวอชิงตัน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 85
        },
        {
          "sentence": "วันนี้เขาจะพูดคุยกับเราเกี่ยวกับ Third SS, U2 Quick และ Blackbird",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 340
        },
        {
          "sentence": "เธอกล่าวว่ามีน้ำตาไหลออกมาจากตาของเธอ และเธอกล่าวว่าโจมาปรากฏตัวที่ชานบ้าน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 385
        },
        {
          "sentence": "เมื่อฉันดึง เมื่อเขาดึงกระโจมเพื่อให้ฉันดึงเขาออกไป เขาชี้ไปที่อุปกรณ์ทั้งสองด้านซ้ายมือของเครื่องบินที่ละลายจริง ๆ ระหว่างการบิน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 505
        },
        {
          "sentence": "แต่ทันทีทันใดนั้น เราก็ถูกเรียกออกไปดูว่ามีอะไรบินอยู่",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 790
        }
      ]
    },
    "ru": {
      "accuracy": 1.0,
      "correct": 279,
      "total": 279,
      "error_examples": []
    },
    "tr": {
      "accuracy": 1.0,
      "correct": 132,
      "total": 132,
      "error_examples": []
    },
    "unknown": {
      "accuracy": 0.0,
      "correct": 0,
      "total": 5,
      "error_examples": [
        {
          "sentence": "Vì vậy, cô ấy giống như, à nhìn đi, trong mong vào công ty như vậy.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1078
        },
        {
          "sentence": "Thế là, à, tôi, ừ, dù sao, ừ, ừ, đây là ba, ừ, phi công U2, ừ, văn phòng Tổng thống Kennedy ở Washington với Đại tướng May.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1138
        },
        {
          "sentence": "1880'li bir tarihte doğdu, 188 gibi, sanırım 1889'du, galiba doğduğu zaman buydu.",
          "predicted": "tr",
          "ground_truth": "unknown",
          "index": 1466
        },
        {
          "sentence": "Nilipata maagizo ya kwenda Del Rio, TX, hivyo wakati nilipofika huko, niligundua nilipaswa kwenda kwa Laughlin Air Force Base.",
          "predicted": "ot",
          "ground_truth": "unknown",
          "index": 1494
        },
        {
          "sentence": "Và tôi đã đến, tôi đã đến Washington D.C. và tôi đã không đi thẳng đến, uh, chỗ đó, uh, nơi họ bảo tôi đến trong các đơn hàng.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1858
        }
      ]
    },
    "ur": {
      "accuracy": 0.9925373134328358,
      "correct": 133,
      "total": 134,
      "error_examples": [
        {
          "sentence": "และแน่นอนว่า Androv Gromikov ไม่ได้ตอบอะไร แต่เรามีข้อมูลทั้งหมดจากภาพยนตร์ว่า U2 ที่ใช้ไปแล้ว",
          "predicted": "hi",
          "ground_truth": "ur",
          "index": 370
        }
      ]
    },
    "vi": {
      "accuracy": 1.0,
      "correct": 134,
      "total": 134,
      "error_examples": []
    },
    "zh": {
      "accuracy": 0.9930555555555556,
      "correct": 143,
      "total": 144,
      "error_examples": [
        {
          "sentence": "แต่พวกมันถูกแบ่งประมาณว่าใครเป็นลูกจ้างซึ่งทำงานในท้องนาและใครเป็นคนทำงานบ้าน มันเป็นประเภทของ--",
          "predicted": "ot",
          "ground_truth": "zh",
          "index": 310
        }
      ]
    }
  },
  "worst_performing_languages": [
    {
      "language": "unknown",
      "accuracy": 0.0,
      "correct": 0,
      "total": 5,
      "error_examples": [
        {
          "sentence": "Vì vậy, cô ấy giống như, à nhìn đi, trong mong vào công ty như vậy.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1078
        },
        {
          "sentence": "Thế là, à, tôi, ừ, dù sao, ừ, ừ, đây là ba, ừ, phi công U2, ừ, văn phòng Tổng thống Kennedy ở Washington với Đại tướng May.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1138
        },
        {
          "sentence": "1880'li bir tarihte doğdu, 188 gibi, sanırım 1889'du, galiba doğduğu zaman buydu.",
          "predicted": "tr",
          "ground_truth": "unknown",
          "index": 1466
        },
        {
          "sentence": "Nilipata maagizo ya kwenda Del Rio, TX, hivyo wakati nilipofika huko, niligundua nilipaswa kwenda kwa Laughlin Air Force Base.",
          "predicted": "ot",
          "ground_truth": "unknown",
          "index": 1494
        },
        {
          "sentence": "Và tôi đã đến, tôi đã đến Washington D.C. và tôi đã không đi thẳng đến, uh, chỗ đó, uh, nơi họ bảo tôi đến trong các đơn hàng.",
          "predicted": "vi",
          "ground_truth": "unknown",
          "index": 1858
        }
      ]
    },
    {
      "language": "hi",
      "accuracy": 0.8104265402843602,
      "correct": 171,
      "total": 211,
      "error_examples": [
        {
          "sentence": "และเขาพูดว่า, ม่าม๊า ผมอยู่บ้าน",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 10
        },
        {
          "sentence": "มันมีอีกมากที่คุณสามารถพูดคุยเกี่ยวกับสิ่งนั้น ฉันจะข้ามไปละกัน",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 70
        },
        {
          "sentence": "และฉันก็แบบว่าตอบตกลงและมันก็เท่านั่น!",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 130
        },
        {
          "sentence": "ฉันก็เลยไม่แน่ใจจริงๆ ว่าทำไม",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 145
        },
        {
          "sentence": "และฉันได้ใส่ เอ่อ, ห้ากองออกมาจาก ยู2",
          "predicted": "ot",
          "ground_truth": "hi",
          "index": 205
        }
      ]
    },
    {
      "language": "ot",
      "accuracy": 0.8756476683937824,
      "correct": 169,
      "total": 193,
      "error_examples": [
        {
          "sentence": "ฉันไม่รู้ว่าฉันไปเพื่ออะไรหรือเพื่อสิ่งใด ดังนั้นแค่รายงานเกี่ยวกับสถานที่ที่ระบุในวอชิงตัน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 85
        },
        {
          "sentence": "วันนี้เขาจะพูดคุยกับเราเกี่ยวกับ Third SS, U2 Quick และ Blackbird",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 340
        },
        {
          "sentence": "เธอกล่าวว่ามีน้ำตาไหลออกมาจากตาของเธอ และเธอกล่าวว่าโจมาปรากฏตัวที่ชานบ้าน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 385
        },
        {
          "sentence": "เมื่อฉันดึง เมื่อเขาดึงกระโจมเพื่อให้ฉันดึงเขาออกไป เขาชี้ไปที่อุปกรณ์ทั้งสองด้านซ้ายมือของเครื่องบินที่ละลายจริง ๆ ระหว่างการบิน",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 505
        },
        {
          "sentence": "แต่ทันทีทันใดนั้น เราก็ถูกเรียกออกไปดูว่ามีอะไรบินอยู่",
          "predicted": "hi",
          "ground_truth": "ot",
          "index": 790
        }
      ]
    },
    {
      "language": "el",
      "accuracy": 0.9171974522292994,
      "correct": 144,
      "total": 157,
      "error_examples": [
        {
          "sentence": "ดี, ฉันไม่ได้คิดอะไรเกี่ยวกับเรื่องนี้, แต่ฉันก็ผิดหวัง, และ, ฉันก็กลับไปคุยกับเขาอีกครั้ง",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 25
        },
        {
          "sentence": "พวกเขาบอกฉันว่าเขาจะเรียกคน ๆ หนึ่งเข้ามาในตอนท้ายให้ฉันพบ",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 55
        },
        {
          "sentence": "และย่าเคยเล่าเรื่องเกี่ยวที่น้องสาวของเธอและสามีของน้องสาวได้ตัดสินใจว่าพวกเขาจะย้ายไปที่เมืองออร์กัสต้าและมองว่าสีขาว",
          "predicted": "hi",
          "ground_truth": "el",
          "index": 250
        },
        {
          "sentence": "แล้วความจริงก็คือ เธอเป็นความสว่าง!",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 775
        },
        {
          "sentence": "ยายของฉันเคยเล่าเรื่องต่าง ๆ มากมายเกี่ยวกับปีเธอที่เติบโตขึ้นมา และโดยเฉพาะอย่างยิ่งเธอเคยพูดเรื่องครอบครัวของเธอและเล่าว่าครอบครัวเธอเป็นอย่างไรในช่วงเวลานั้น",
          "predicted": "ot",
          "ground_truth": "el",
          "index": 1015
        }
      ]
    },
    {
      "language": "ar",
      "accuracy": 0.9657534246575342,
      "correct": 141,
      "total": 146,
      "error_examples": [
        {
          "sentence": "U2 (یو 2) کی پرواز شروع کرنے یا پریشر سوٹ کے ساتھ پرواز کرنے سے بھی پہلے انہیں اونچائی کے چیمبر کی کئی سواریوں/پروازوں سے گزرنا پڑتا ہے.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1242
        },
        {
          "sentence": "'پچتھر سال میں یہ پہلی بار ہوا ہے کہ' ٹی ایکس آۂین نے فوج کو ووٹ دی، ' ٹی ایکس' سفیر ہوتے ہوے  ' ٹی ایکس' سفیر چاھۂے تھے۔",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1287
        },
        {
          "sentence": "میرا مطلب یہ تھا کہ پوری بات.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1557
        },
        {
          "sentence": "وہ کیوبا کے بحران کا واحد زخمی تھا، اور uh، کیسر uh، وہ تصاویر ملی اور واشنگٹن میں اینڈریو ایئر فورس کی طرف براہ راست اڑ گئے.",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 1767
        },
        {
          "sentence": "خوش آمدید",
          "predicted": "ur",
          "ground_truth": "ar",
          "index": 2007
        }
      ]
    }
  ]
}