# 使用 PEFT 和 Unsloth Fine-tuning LLM
## 載入 Base Model
我們下載 4-bit Mistral 7b 的模型並透過 unsloth 的 **`FastLanguageModel`** 類別載入。
```python
from unsloth import FastLanguageModel
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
"unsloth/mistral-7b-bnb-4bit",
"unsloth/mistral-7b-instruct-v0.2-bnb-4bit",
"unsloth/llama-2-7b-bnb-4bit",
"unsloth/llama-2-13b-bnb-4bit",
"unsloth/codellama-34b-bnb-4bit",
"unsloth/tinyllama-bnb-4bit",
] # More models at https://huggingface.co/unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/mistral-7b-bnb-4bit", # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
attn_implementation="flash_attention_2", # 使用 Flash Attention-2
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
```
我們查看一下 base model 本身 tokenizer 的 **`bos_token`**、**`eos_token`** 和 **`pad_token`** 分別是什麼。
```python
tokenizer.bos_token, tokenizer.eos_token, tokenizer.pad_token
# ('<s>', '</s>', '<unk>')
tokenizer.bos_token_id, tokenizer.eos_token_id, tokenizer.pad_token_id
# (1, 2, 0)
```
## 加入 LoRA adapters
我們只需要訓練 LoRA 模組的參數，佔不到所有參數的 10%。
```python
model = FastLanguageModel.get_peft_model(
model,
r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 16,
lora_dropout = 0, # Supports any, but = 0 is optimized
bias = "none", # Supports any, but = "none" is optimized
use_gradient_checkpointing = True,
random_state = 3407,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)
```
## 資料準備
我們使用 `ChatML` 格式進行對話風格的微調。這邊示範 ShareGPT 風格的 [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) 資料集。`ChatML` 呈現多輪對話，如下圖所示：
```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What's the capital of France?<|im_end|>
<|im_start|>assistant
Paris.<|im_end|>
```
:::success
:notebook: Note
若要僅訓練 completions（忽略使用者的輸入），請閱讀 [Train on completions only](https://hackmd.io/@6j0OMC7UQbGqQLfUg9pauA/H1CYpAK3T) 的章節。
:::
我們使用 unsloth 自家的 **`get_chat_template`** 函數來取得正確的聊天模板。此支援 `zephyr`、`chatml`、`mistral`、`llama`、`alpaca`、`vicuna`、`vicuna_old` 和 `unsloth`。
通常需要訓練 **`<|im_start|>`** 和 **`<|im_end|>`**。我們將 **`<|im_end|>`** 映射為 EOS token，並保留 **`<|im_start|>`** 不變。這不需要對額外的 token 進行額外的訓練。
:bulb:【Note】注意 ShareGPT 使用 `{"from": " human", "value" : "Hi"}` 而不是 `{"role": "user", "content" : "Hi"}`，因此我們使用 **`mapping`** 來映射它。
```python
from unsloth.chat_templates import get_chat_template
tokenizer = get_chat_template(
tokenizer,
chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth
mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style
map_eos_token = True, # Maps <|im_end|> to </s> instead
)
def formatting_prompts_func(examples):
convos = examples["conversations"]
texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]
return { "text" : texts, }
pass
from datasets import load_dataset
dataset = load_dataset("teknium/OpenHermes-2.5", split = "train")
dataset = dataset.map(formatting_prompts_func, batched = True,)
```
現在 tokenizer 的 **`eos_token`** 變成 **`<|im_end|>`**，但 **`eos_token_id`** 依舊維持本來的數值，也代表著不需要額外訓練新的 token。
```python
tokenizer.bos_token, tokenizer.eos_token, tokenizer.pad_token
# ('<s>', '<|im_end|>', '<unk>')
tokenizer.bos_token_id, tokenizer.eos_token_id, tokenizer.pad_token_id
# (1, 2, 0)
```
:::danger
:no_entry: 嚴重
在訓練之前要確保 tokenizer 的 **`eos_token`** 是否和 **`pad_token`** 不一樣，如果使用 **`SFTTrainer`** 進行訓練且參數設定 **`packing=False`** 及 **`data_collator=None`**，則 **`data_collator`** 預設使用 **`transformers.DataCollatorForLanguageModeling`**，而它不會計算 **`pad_token`** 的 loss，因此 **`eos_token`** 和 **`pad_token`** 一樣，就意味著模型學不到生成 **`eos_token`**。
:::
接下來，讓我們透過印出第一筆來看看 `ChatML` 格式是如何運作的。
```python
dataset["conversations"][1]
# [{'from': 'human',
# 'value': 'In analytical chemistry, what is the principle behind the use of an internal standard in quantitative analysis?\nA. It compensates for variations in sample preparation and instrumental response.\nB. It enhances the sensitivity of the analytical method.\nC. It reduces the detection limit of the analytical method.\nD. It increases the resolution between analyte peaks in chromatography.\nE. None of the above.',
# 'weight': None},
# {'from': 'gpt',
# 'value': 'A. It compensates for variations in sample preparation and instrumental response.',
# 'weight': None}]
print(dataset[1]["text"])
# <|im_start|>user
# In analytical chemistry, what is the principle behind the use of an internal standard in quantitative analysis?
# A. It compensates for variations in sample preparation and instrumental response.
# B. It enhances the sensitivity of the analytical method.
# C. It reduces the detection limit of the analytical method.
# D. It increases the resolution between analyte peaks in chromatography.
# E. None of the above.<|im_end|>
# <|im_start|>assistant
# A. It compensates for variations in sample preparation and instrumental response.<|im_end|>
```
## 訓練模型
現在讓我們來使用 Huggingface TRL 的 **`SFTTrainer`**！[TRL SFT 文件](https://huggingface.co/docs/trl/sft_trainer)。Unsloth 也支持 TRL 的 **`DPOTrainer`**！
```python
import torch
from trl import SFTTrainer
from transformers import TrainingArguments
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
packing = False, # Can make training 5x faster for short sequences.
neftune_noise_alpha = 5,
args = TrainingArguments(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_steps = 5,
max_steps = 60,
learning_rate = 2e-4,
fp16 = not torch.cuda.is_bf16_supported(),
bf16 = torch.cuda.is_bf16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "outputs",
),
)
```
:::warning
:warning: 警告
確定 tokenizer 的 **`eos_token`** 和 **`pad_token`** 不一樣之後，要記得傳入 tokenizer 到 **`SFTTrainer`**，如果沒有且 tokenizer 本身沒有設定 **`pad_token`**，則 **`SFTTrainer`** 會將 **`pad_token`** 設定和 **`eos_token`** 一樣，導致不會訓練 **`eos_token`**。
:::
```python
trainer_stats = trainer.train()
# {'loss': 1.7715, 'learning_rate': 4e-05, 'epoch': 0.0}
# {'loss': 1.6014, 'learning_rate': 8e-05, 'epoch': 0.0}
# {'loss': 1.2659, 'learning_rate': 0.00012, 'epoch': 0.0}
# {'loss': 1.4734, 'learning_rate': 0.00016, 'epoch': 0.0}
# {'loss': 1.6183, 'learning_rate': 0.0002, 'epoch': 0.0}
# {'loss': 1.3259, 'learning_rate': 0.00019636363636363636, 'epoch': 0.0}
# {'loss': 1.2349, 'learning_rate': 0.00019272727272727274, 'epoch': 0.0}
# {'loss': 1.403, 'learning_rate': 0.0001890909090909091, 'epoch': 0.0}
# 13%|██████ | 8/60 [00:35<03:27, 4.00s/it]
```
## Inference
讓我們來運行模型吧！由於我們使用的是 `ChatML`，因此請使用 **`apply_chat_template`** 並將 **`add_generation_prompt`** 設為 **`True`** 進行推理。
```python
from unsloth.chat_templates import get_chat_template
tokenizer = get_chat_template(
tokenizer,
chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth
mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style
map_eos_token = True, # Maps <|im_end|> to </s> instead
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
messages = [
{"from": "human", "value": "Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize = True,
add_generation_prompt = True, # Must add for generation
return_tensors = "pt",
).to("cuda")
outputs = model.generate(input_ids = inputs, max_new_tokens = 64, use_cache = True)
tokenizer.batch_decode(outputs)
```
輸出結果：
```
['<|im_start|>user\nNext number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|> \n<|im_start|>assistant\nThe next number in the Fibonacci sequence is 13.<|im_end|>']
```
您也可以使用 **`TextStreamer`** 進行持續的推論 - 這樣您可以逐個查看生成的 token，而不是等待整個過程！
```python
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
messages = [
{"from": "human", "value": "Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize = True,
add_generation_prompt = True, # Must add for generation
return_tensors = "pt",
).to("cuda")
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True)
```
輸出結果：
```
<|im_start|>user
Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|>
<|im_start|>assistant
The next number in the Fibonacci sequence is 13.<|im_end|>
```
## 儲存、載入微調模型
若要將最終模型儲存為 LoRA adapters，請使用 Huggingface 的 **`push_to_hub`** 進行線上儲存，或使用 **`save_pretrained`** 進行本機儲存。
:bulb:【Note】這僅保存 LoRA adapters，而不是完整模型。要儲存到 16 位元或 GGUF，請向下捲動！
```python
model.save_pretrained("lora_model") # Local saving
# model.push_to_hub("your_name/lora_model", token = "...") # Online saving
```
現在我們載入剛剛儲存用於推理的 LoRA adapters。
```python
from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
attn_implementation="flash_attention_2",
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
tokenizer = get_chat_template(
tokenizer,
chat_template = "chatml", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth
mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"}, # ShareGPT style
map_eos_token = True, # Maps <|im_end|> to </s> instead
)
messages = [
{"from": "human", "value": "What is a famous tall tower in Paris?"},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize = True,
add_generation_prompt = True, # Must add for generation
return_tensors = "pt",
).to("cuda")
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True)
```
輸出結果：
```
<|im_start|>user
What is a famous tall tower in Paris?<|im_end|>
<|im_start|>assistant
The Eiffel Tower is a famous tall tower in Paris, France. It was built in 1889 as the entrance arch to the 1889 World's Fair and is named after its designer, Gustave Eiffel. The tower is 324 meters (1,063 feet) tall and is one of the most recognizable symbols of Paris and France. It is located on the Champ de Mars, a large public park in the 7th arrondissement of Paris.<|im_end|>
```
## Saving to float16 for VLLM
我們也直接支援儲存為 `float16`。選擇 **`merged_16bit`** 以儲存為 `float16`，或選擇 **`merged_4bit`** 以儲存為 `int4`。我們也允許作為後備方案使用 **`lora`** adapters。
使用 **`push_to_hub_merged`** 上傳到您的 Hugging Face 帳號！您可以前往 https://huggingface.co/settings/tokens 取得您的個人 tokens。
```python
# Merge to 16bit
model.save_pretrained_merged("model_16bit", tokenizer, save_method = "merged_16bit",)
model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "")
# Merge to 4bit
model.save_pretrained_merged("model_4bit", tokenizer, save_method = "merged_4bit",)
model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_4bit", token = "")
# Just LoRA adapters
model.save_pretrained_merged("model_lora", tokenizer, save_method = "lora",)
model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "")
```
:notebook: 使用 **`save_pretrained_merged()`** 函數可以傳入 tokenizer，它會一起儲存模型和 tokenizer。
現在我們載入剛剛 merge 好的 16-bit 模型並進行 inference，查看輸出結果是否一致。
```python
from unsloth import FastLanguageModel
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False.
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "model_16bit", # YOUR MODEL YOU USED FOR TRAINING
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
attn_implementation="flash_attention_2",
)
FastLanguageModel.for_inference(model)
messages = [
{"from": "human", "value": "Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize = True,
add_generation_prompt = True, # Must add for generation
return_tensors = "pt",
).to("cuda")
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True)
```
輸出結果：
```
<|im_start|>user
Next number of the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|im_end|>
<|im_start|>assistant
The next number in the Fibonacci sequence is 13.<|im_end|>
```
我們可以看到輸出的結果一模一樣，接著就可以拿它在 vLLM 進行部署。
:::warning
:warning: 警告
使用 **`FastLanguageModel.from_pretrained()`** 載入 16-bit 模型進行 inference 時，要特別小心 **`load_in_4bit`** 參數要設為 **`False`**，由於 LoRA 的 weight 已經合併到 base model，如果再進行 4-bit 量化會損失準確率，導致 inference 行為和原先不一致。
:::
Sign in with Wallet
Connect another wallet