使用 PEFT 和 Unsloth Fine-tuning LLM

# 使用 PEFT 和 Unsloth Fine-tuning LLM ## 載入 Base Model 我們下載 4-bit Mistral 7b 的模型並透過 unsloth 的 **`FastLanguageModel`** 類別載入。 ```python from unsloth import FastLanguageModel max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False. # 4bit pre quantized models we support for 4x faster downloading + no OOMs. fourbit_models = [ "unsloth/mistral-7b-bnb-4bit", "unsloth/mistral-7b-instruct-v0.2-bnb-4bit", "unsloth/llama-2-7b-bnb-4bit", "unsloth/llama-2-13b-bnb-4bit", "unsloth/codellama-34b-bnb-4bit", "unsloth/tinyllama-bnb-4bit", ] # More models at https://huggingface.co/unsloth model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/mistral-7b-bnb-4bit", # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, attn_implementation="flash_attention_2", # 使用 Flash Attention-2 # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf ) ``` 我們查看一下 base model 本身 tokenizer 的 **`bos_token`**、**`eos_token`** 和 **`pad_token`** 分別是什麼。 ```python tokenizer.bos_token, tokenizer.eos_token, tokenizer.pad_token # ('<s>', '</s>', '<unk>') tokenizer.bos_token_id, tokenizer.eos_token_id, tokenizer.pad_token_id # (1, 2, 0) ``` ## 加入 LoRA adapters 我們只需要訓練 LoRA 模組的參數，佔不到所有參數的 10%。 ```python model = FastLanguageModel.get_peft_model( model, r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized use_gradient_checkpointing = True, random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ ) ``` ## 資料準備我們使用 `ChatML` 格式進行對話風格的微調。這邊示範 ShareGPT 風格的 [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) 資料集。`ChatML` 呈現多輪對話，如下圖所示： ``` <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user What's the capital of France?<|im_end|> <|im_start|>assistant Paris.<|im_end|> ``` :::success :notebook: Note 若要僅訓練 completions（忽略使用者的輸入），請閱讀 [Train on completions only](https://hackmd.io/@6j0OMC7UQbGqQLfUg9pauA/H1CYpAK3T) 的章節。 ::: 我們使用 unsloth 自家的 **`get_chat_template`** 函數來取得正確的聊天模板。此支援 `zephyr`、`chatml`、`mistral`、`llama`、`alpaca`、`vicuna`、`vicuna_old` 和 `unsloth`。通常需要訓練 **`<|im_start|>`** 和 **`<|im_end|>`**。我們將 **`<|im_end|>`** 映射為 EOS token，並保留 **`<|im_start|>`** 不變。這不需要對額外的 token 進行額外的訓練。 :bulb:【Note】注意 ShareGPT 使用 `{"from": " human", "value" : "Hi"}` 而不是 `{"role": "user", "content" : "Hi"}`，因此我們使用 **`mapping`** 來映射它。 ```python from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to </s> instead ) def formatting_prompts_func(examples): convos = examples["conversations"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } pass from datasets import load_dataset dataset = load_dataset("teknium/OpenHermes-2.5", split = "train") dataset = dataset.map(formatting_prompts_func, batched = True,) ``` 現在 tokenizer 的 **`eos_token`** 變成 **`<|im_end|>`**，但 **`eos_token_id`** 依舊維持本來的數值，也代表著不需要額外訓練新的 token。 ```python tokenizer.bos_token, tokenizer.eos_token, tokenizer.pad_token # ('<s>', '<|im_end|>', '<unk>') tokenizer.bos_token_id, tokenizer.eos_token_id, tokenizer.pad_token_id # (1, 2, 0) ``` :::danger :no_entry: 嚴重在訓練之前要確保 tokenizer 的 **`eos_token`** 是否和 **`pad_token`** 不一樣，如果使用 **`SFTTrainer`** 進行訓練且參數設定 **`packing=False`** 及 **`data_collator=None`**，則 **`data_collator`** 預設使用 **`transformers.DataCollatorForLanguageModeling`**，而它不會計算 **`pad_token`** 的 loss，因此 **`eos_token`** 和 **`pad_token`** 一樣，就意味著模型學不到生成 **`eos_token`**。 ::: 接下來，讓我們透過印出第一筆來看看 `ChatML` 格式是如何運作的。 ```python dataset["conversations"][1] # [{'from': 'human', # 'value': 'In analytical chemistry, what is the principle behind the use of an internal standard in quantitative analysis?\nA. It compensates for variations in sample preparation and instrumental response.\nB. It enhances the sensitivity of the analytical method.\nC. It reduces the detection limit of the analytical method.\nD. It increases the resolution between analyte peaks in chromatography.\nE. None of the above.', # 'weight': None}, # {'from': 'gpt', # 'value': 'A. It compensates for variations in sample preparation and instrumental response.', # 'weight': None}] print(dataset[1]["text"]) # <|im_start|>user # In analytical chemistry, what is the principle behind the use of an internal standard in quantitative analysis? # A. It compensates for variations in sample preparation and instrumental response. # B. It enhances the sensitivity of the analytical method. # C. It reduces the detection limit of the analytical method. # D. It increases the resolution between analyte peaks in chromatography. # E. None of the above.<|im_end|> # <|im_start|>assistant # A. It compensates for variations in sample preparation and instrumental response.<|im_end|> ``` ## 訓練模型現在讓我們來使用 Huggingface TRL 的 **`SFTTrainer`**！[TRL SFT 文件](https://huggingface.co/docs/trl/sft_trainer)。Unsloth 也支持 TRL 的 **`DPOTrainer`**！ ```python import torch from trl import SFTTrainer from transformers import TrainingArguments trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. neftune_noise_alpha = 5, args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, learning_rate = 2e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", ), ) ``` :::warning :warning: 警告確定 tokenizer 的 **`eos_token`** 和 **`pad_token`** 不一樣之後，要記得傳入 tokenizer 到 **`SFTTrainer`**，如果沒有且 tokenizer 本身沒有設定 **`pad_token`**，則 **`SFTTrainer`** 會將 **`pad_token`** 設定和 **`eos_token`** 一樣，導致不會訓練 **`eos_token`**。 ::: ```python trainer_stats = trainer.train() # {'loss': 1.7715, 'learning_rate': 4e-05, 'epoch': 0.0} # {'loss': 1.6014, 'learning_rate': 8e-05, 'epoch': 0.0} # {'loss': 1.2659, 'learning_rate': 0.00012, 'epoch': 0.0} # {'loss': 1.4734, 'learning_rate': 0.00016, 'epoch': 0.0} # {'loss': 1.6183, 'learning_rate': 0.0002, 'epoch': 0.0} # {'loss': 1.3259, 'learning_rate': 0.00019636363636363636, 'epoch': 0.0} # {'loss': 1.2349, 'learning_rate': 0.00019272727272727274, 'epoch': 0.0} # {'loss': 1.403, 'learning_rate': 0.0001890909090909091, 'epoch': 0.0} # 13%|██████ | 8/60 [00:35<03:27, 4.00s/it] ``` ## Inference 讓我們來運行模型吧！由於我們使用的是 `ChatML`，因此請使用 **`apply_chat_template`** 並將 **`add_generation_prompt`** 設為 **`True`** 進行推理。 ```python from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to </s> instead ) FastLanguageModel.for_inference(model) # Enable native 2x faster inference messages = [ {"from": "human", "value": "Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"}, ] inputs = tokenizer.apply_chat_template( messages, tokenize = True, add_generation_prompt = True, # Must add for generation return_tensors = "pt", ).to("cuda") outputs = model.generate(input_ids = inputs, max_new_tokens = 64, use_cache = True) tokenizer.batch_decode(outputs) ``` 輸出結果： ``` ['<|im_start|>user\nNext number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|> \n<|im_start|>assistant\nThe next number in the Fibonacci sequence is 13.<|im_end|>'] ``` 您也可以使用 **`TextStreamer`** 進行持續的推論 - 這樣您可以逐個查看生成的 token，而不是等待整個過程！ ```python FastLanguageModel.for_inference(model) # Enable native 2x faster inference messages = [ {"from": "human", "value": "Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"}, ] inputs = tokenizer.apply_chat_template( messages, tokenize = True, add_generation_prompt = True, # Must add for generation return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True) ``` 輸出結果： ``` <|im_start|>user Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|> <|im_start|>assistant The next number in the Fibonacci sequence is 13.<|im_end|> ``` ## 儲存、載入微調模型若要將最終模型儲存為 LoRA adapters，請使用 Huggingface 的 **`push_to_hub`** 進行線上儲存，或使用 **`save_pretrained`** 進行本機儲存。 :bulb:【Note】這僅保存 LoRA adapters，而不是完整模型。要儲存到 16 位元或 GGUF，請向下捲動！ ```python model.save_pretrained("lora_model") # Local saving # model.push_to_hub("your_name/lora_model", token = "...") # Online saving ``` 現在我們載入剛剛儲存用於推理的 LoRA adapters。 ```python from unsloth import FastLanguageModel from unsloth.chat_templates import get_chat_template max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False. model, tokenizer = FastLanguageModel.from_pretrained( model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, attn_implementation="flash_attention_2", ) FastLanguageModel.for_inference(model) # Enable native 2x faster inference tokenizer = get_chat_template( tokenizer, chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style map_eos_token = True, # Maps <|im_end|> to </s> instead ) messages = [ {"from": "human", "value": "What is a famous tall tower in Paris?"}, ] inputs = tokenizer.apply_chat_template( messages, tokenize = True, add_generation_prompt = True, # Must add for generation return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True) ``` 輸出結果： ``` <|im_start|>user What is a famous tall tower in Paris?<|im_end|> <|im_start|>assistant The Eiffel Tower is a famous tall tower in Paris, France. It was built in 1889 as the entrance arch to the 1889 World's Fair and is named after its designer, Gustave Eiffel. The tower is 324 meters (1,063 feet) tall and is one of the most recognizable symbols of Paris and France. It is located on the Champ de Mars, a large public park in the 7th arrondissement of Paris.<|im_end|> ``` ## Saving to float16 for VLLM 我們也直接支援儲存為 `float16`。選擇 **`merged_16bit`** 以儲存為 `float16`，或選擇 **`merged_4bit`** 以儲存為 `int4`。我們也允許作為後備方案使用 **`lora`** adapters。使用 **`push_to_hub_merged`** 上傳到您的 Hugging Face 帳號！您可以前往 https://huggingface.co/settings/tokens 取得您的個人 tokens。 ```python # Merge to 16bit model.save_pretrained_merged("model_16bit", tokenizer, save_method = "merged_16bit",) model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "") # Merge to 4bit model.save_pretrained_merged("model_4bit", tokenizer, save_method = "merged_4bit",) model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_4bit", token = "") # Just LoRA adapters model.save_pretrained_merged("model_lora", tokenizer, save_method = "lora",) model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "") ``` :notebook: 使用 **`save_pretrained_merged()`** 函數可以傳入 tokenizer，它會一起儲存模型和 tokenizer。現在我們載入剛剛 merge 好的 16-bit 模型並進行 inference，查看輸出結果是否一致。 ```python from unsloth import FastLanguageModel max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False. model, tokenizer = FastLanguageModel.from_pretrained( model_name = "model_16bit", # YOUR MODEL YOU USED FOR TRAINING max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, attn_implementation="flash_attention_2", ) FastLanguageModel.for_inference(model) messages = [ {"from": "human", "value": "Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"}, ] inputs = tokenizer.apply_chat_template( messages, tokenize = True, add_generation_prompt = True, # Must add for generation return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True) ``` 輸出結果： ``` <|im_start|>user Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|> <|im_start|>assistant The next number in the Fibonacci sequence is 13.<|im_end|> ``` 我們可以看到輸出的結果一模一樣，接著就可以拿它在 vLLM 進行部署。 :::warning :warning: 警告使用 **`FastLanguageModel.from_pretrained()`** 載入 16-bit 模型進行 inference 時，要特別小心 **`load_in_4bit`** 參數要設為 **`False`**，由於 LoRA 的 weight 已經合併到 base model，如果再進行 4-bit 量化會損失準確率，導致 inference 行為和原先不一致。 :::

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.