# Team 2002! ⚽️
Chugai x Prescient Design
Yukiko - Ryouichi - Mariko
Stephen - Keunwoo
## Goal
- Predict success of clinical trials
## Data
- HINT dataset: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9024011/
- Repo: https://github.com/futianfan/clinical-trial-outcome-prediction
> Various public data sources can provide vital information for predicting the trial outcome. For example, the ClinicalTrials.gov database (publicly available at https://clinicaltrials.gov/) has 369,700 historical clinical trials, including important information about those trials. In addition, we utilize the standard medical codes of the diseases and their natural language descriptions through the National Institutes of Health website (publicly available at https://clinicaltables.nlm.nih.gov/). The DrugBank database (publicly available at https://www.drugbank.ca/) contains the biochemical description of many drugs, which allows for the computational modeling of drug molecules.6
- AACT: https://aact.ctti-clinicaltrials.org/learn_more
- only ClinicalTrials.gov
clinical trials status to use:
- Terminated: The study has stopped early and will not start again. Participants are no longer being examined or treated. **<- Failure**
- Completed: The study has ended normally, and participants are no longer being examined or treated (that is, the last participant's last visit has occurred). **<- Success**
other status:
- Not yet recruiting: The study has not started recruiting participants.
- Recruiting: The study is currently recruiting participants.
- Enrolling by invitation: The study is selecting its participants from a population, or group of people, decided on by the researchers in advance. These studies are not open to everyone who meets the eligibility criteria but only to people in that particular population, who are specifically invited to participate.
- Active, not recruiting: The study is ongoing, and participants are receiving an intervention or being examined, but potential participants are not currently being recruited or enrolled.
- Suspended: The study has stopped early but may start again.
- Withdrawn: The study stopped early, before enrolling its first participant.
- Unknown: A study on ClinicalTrials.gov whose last known status was recruiting; not yet recruiting; or active, not recruiting but that has passed its completion date, and the status has not been last verified within the past 2 years.
### Processed Data
https://drive.google.com/drive/folders/1xFrOUeN9mgh253tdwLZ_j7EdNEfztd8j
Filename: `raw_data.csv`
Using colums as input_text:
- ['diseases', 'icdcodes', 'drugs', 'smiless', 'criteria']
## Example finetuning code using Huggingface tools
This is an example working code that performs finetuning Prescient-LM models.
```python
"""
# Finetuning Prescient-LM with Alpaca dataset and HuggingFace Transformers
## Prerequisites
- Software: Install prescient-lm with `-U` to get the latest versions of the dependencies (peft, transformers, ..)
- Model: Convert the model to the HuggingFace format first. See "text-generation-with-Huggingface" notebook for details.
- Data: `wget https://raw.githubusercontent.com/gururise/AlpacaDataCleaned/main/alpaca_data_cleaned_archive.json`
- Hardware (as used during test)
- For 0.5B, x1 V100 GPU (or better) with 16GB memory;
- For 7B
- (Fully) Finetuning: xN A100 GPUs (or better) with 80GB memory each. (Reduce the batch size if you have less memory.)
- Quantized LoRA:
- Minimal requirement: x1 V100 (16GB): Takes 45 hrs for 3 epochs (with batch_size=1.)
- Recommended: x1 A100 (80GB): Takes 14 hrs for 3 epochs.
## Note
- This code is based on the custom argparser from Huggingface. See [the TrainingArguments doc](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) to see what arguments are available.
As in the script below, you can simply pass the arguments to the `Trainer` class on CLI.
## 1. Script for (full) fine-tuning
**0.5B Model**
- Single-GPU (tested on V100)
- Required to convert Prescient-LM model to the HF format first. (See `text-generation-with-Huggingface.ipynb`)
On Bash shell:
python finetuning_alpaca_huggingface.py \
--model_name=0.5B-latest-latest \
--data_path="alpaca_data_cleaned_archive.json" \
--output_dir="/data/prescient-lm/experiment/checkpointing/0.5B-finetune-alpaca" \
--per_device_train_batch_size 4
**7B Model**
- Assuming 4 a100s are available:
- Required to convert Prescient-LM model to the HF format first. (See `text-generation-with-Huggingface.ipynb`)
On Bash Shell:
export OMP_NUM_THREADS=1
torchrun --nproc_per_node=4 --master_port=9987 \
finetuning_alpaca_huggingface.py \
--model_name=7B-latest \
--data_path="alpaca_data_cleaned_archive.json" \
--output_dir="output-HF-finetuned/7B-nonQ-finetune" \
--per_device_train_batch_size 1 \
--bf16 True \
--tf32 True \
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer'
## 2. Script for LoRA fine-tuning
**0.5B Model**
python finetuning_alpaca_huggingface.py \
--model_name=0.5B-latest \
--data_path="alpaca_data_cleaned_archive.json" \
--output_dir="output-HF-finetuned/0.5B-nonQ-lora" \
--per_device_train_batch_size 2 \
--use_lora True
**7B Model**
export OMP_NUM_THREADS=1
torchrun --nproc_per_node=4 --master_port=9987 \
finetuning_alpaca_huggingface.py \
--model_name=7B-latest \
--data_path="alpaca_data_cleaned_archive.json" \
--output_dir="output-HF-finetuned/7B-nonQ-lora" \
--per_device_train_batch_size 2 \
--use_lora True \
--bf16 True \
--tf32 True
## 3. Script for Quantized LoRA
**0.5B Model**
python finetuning_alpaca_huggingface.py \
--model_name=0.5B-latest \
--data_path="alpaca_data_cleaned_archive.json" \
--output_dir="output-HF-finetuned/0.5B-8bitQ-lora" \
--per_device_train_batch_size 2 \
--use_lora True \
--load_in_4bit True
**7B Model**
python finetuning_alpaca_huggingface.py \
--model_name=7B-latest \
--data_path="alpaca_data_cleaned_archive.json" \
--output_dir="output-HF-finetuned/7B-8bitQ-lora" \
--per_device_train_batch_size 2 \
--use_lora True \
--load_in_4bit True \
--per_device_train_batch_size 1
## After training
See `generate_with_alpaca_huggingface.py`.
"""
import json
import copy
import logging
from dataclasses import dataclass, field
from typing import Dict, Optional, Sequence
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch
import transformers
from torch.utils.data import Dataset
from transformers import Trainer, BitsAndBytesConfig
import prescient_lm as plm
IGNORE_INDEX = -100 # This is the value we use to mask the "question" part (or user instruction). Timesteps replaced with this value will be ignored during the loss calculation.
DEFAULT_PAD_TOKEN = "<pad>"
DEFAULT_EOS_TOKEN = "</s>"
DEFAULT_BOS_TOKEN = "<s>"
DEFAULT_UNK_TOKEN = "<unk>"
# This is the Alpaca-style prompt format for instruction finetuning.
PROMPT_DICT = {
"prompt_input": (
f"{DEFAULT_BOS_TOKEN}Below is an instruction that describes a task, paired with an input that provides further context. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
),
"prompt_no_input": (
f"{DEFAULT_BOS_TOKEN}Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{instruction}\n\n### Response:"
),
}
# data: list of dict, keys are
# - "report_text" (=merged from many columns to create text)
# - "outcome" (=status)
PROMPT_DICT = {
"prompt_input": (
f"{DEFAULT_BOS_TOKEN}Below is a clinical trial report record."
"\n\n"
"{report_text}"
"\n"
"Please estimate the success rate of this clinical trial.\n\n"
""
) # later, we append f"The success rate is {probability}"
}
@dataclass
class ModelArguments:
model_name: Optional[str] = field(default="0.5B-latest", metadata={"help": "Model name from plm_api.list_hf_models()"})
use_lora: Optional[bool] = field(default=False)
load_in_4bit: Optional[bool] = field(default=False)
torch_dtype: Optional[str] = field(default="auto", metadata={"help": "torch dtype. passed to `from_pretrained()."})
@dataclass
class DataArguments:
data_path: str = field(default=None, metadata={"help": "Path to the training data."})
@dataclass
class TrainingArguments(transformers.TrainingArguments):
cache_dir: Optional[str] = field(default=None)
optim: str = field(default="adamw_torch")
model_max_length: int = field(
default=512,
metadata={"help": "Maximum sequence length. Sequences will be right padded (and possibly truncated)."},
)
gradient_accumulation_steps: int = field(
default=64,
metadata={"help": "Number of updates steps to accumulate before performing a backward/update pass."},
)
num_train_epochs: float = field(default=3.0, metadata={"help": "Total number of training epochs to perform."})
learning_rate: float = field(default=2e-5, metadata={"help": "The initial learning rate for AdamW."})
lr_scheduler_type: str = field(default="cosine", metadata={"help": "The type of scheduler."})
save_strategy: str = field(default="no")
@dataclass
class MiscArguments:
use_bos_token: Optional[bool] = field(default=False)
def print_trainable_parameters(model):
"""
Prints the number of trainable parameters in the model.
"""
trainable_params = 0
all_param = 0
for _, param in model.named_parameters():
all_param += param.numel()
if param.requires_grad:
trainable_params += param.numel()
print(
f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
)
def smart_tokenizer_and_embedding_resize(
special_tokens_dict: Dict,
tokenizer: transformers.PreTrainedTokenizer,
model: transformers.PreTrainedModel,
):
"""Resize tokenizer and embedding.
Note: This is the unoptimized version that may make your embedding size not be divisible by 64.
"""
num_new_tokens = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
if num_new_tokens > 0:
input_embeddings = model.get_input_embeddings().weight.data
output_embeddings = model.get_output_embeddings().weight.data
input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
output_embeddings_avg = output_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings[-num_new_tokens:] = input_embeddings_avg
output_embeddings[-num_new_tokens:] = output_embeddings_avg
def _tokenize_fn(strings: Sequence[str], tokenizer: transformers.PreTrainedTokenizer) -> Dict:
"""Tokenize a list of strings.
This function constructs a dictionary of input_ids, labels, input_ids_lens, labels_lens based on the tokenizer's model_max_length.
"""
tokenized_list = [
tokenizer(
text,
return_tensors="pt",
padding="longest",
max_length=tokenizer.model_max_length,
truncation=True,
)
for text in strings
]
input_ids = labels = [tokenized.input_ids[0] for tokenized in tokenized_list]
input_ids_lens = labels_lens = [
tokenized.input_ids.ne(tokenizer.pad_token_id).sum().item() for tokenized in tokenized_list
]
return dict(
input_ids=input_ids,
labels=labels,
input_ids_lens=input_ids_lens,
labels_lens=labels_lens,
)
def preprocess(
sources: Sequence[str],
targets: Sequence[str],
tokenizer: transformers.PreTrainedTokenizer,
) -> Dict:
"""Preprocess the data by tokenizing by masking the beginnings of the target sequence (labels)."""
examples = [s + t for s, t in zip(sources, targets)]
examples_tokenized, sources_tokenized = [_tokenize_fn(strings, tokenizer) for strings in (examples, sources)]
input_ids = examples_tokenized["input_ids"]
labels = copy.deepcopy(input_ids)
for label, source_len in zip(labels, sources_tokenized["input_ids_lens"]):
label[:source_len] = IGNORE_INDEX
return dict(input_ids=input_ids, labels=labels)
class SupervisedDataset(Dataset):
"""Dataset for supervised fine-tuning.
Parameters
----------
data_path : str
Path to the training data (json file) - Alpaca instruction.
tokenizer : transformers.PreTrainedTokenizer
Tokenizer used in the model.
"""
def __init__(self, data_path: str, tokenizer: transformers.PreTrainedTokenizer):
super(SupervisedDataset, self).__init__()
logging.warning("Loading data...")
with open(data_path, "r") as f:
list_data_dict = json.load(f)
logging.warning("Formatting inputs...")
prompt_input, prompt_no_input = PROMPT_DICT["prompt_input"], PROMPT_DICT["prompt_no_input"]
sources = [
prompt_input.format_map(example) if example.get("input", "") else prompt_no_input.format_map(example)
for example in list_data_dict
]
targets = [f"{example['output']}{tokenizer.eos_token}" for example in list_data_dict]
logging.warning("Tokenizing inputs... This may take some time...")
data_dict = preprocess(sources, targets, tokenizer)
self.input_ids = data_dict["input_ids"]
self.labels = data_dict["labels"]
def __len__(self):
return len(self.input_ids)
def __getitem__(self, i) -> Dict[str, torch.Tensor]:
"""
Returns
-------
A dictionary with the following keys:
- input_ids (:obj:`torch.Tensor` of shape :obj:`[sequence_length]`):
Indices of input sequence tokens in the vocabulary.
BOS token + prompt + EOS token + Padding tokens
- labels (:obj:`torch.Tensor` of shape :obj:`[sequence_length]`):
Indices of labels in the vocabulary.
Masking tokens (IGNORE_INDEX) + output + EOS token + Padding tokens
"""
return dict(input_ids=self.input_ids[i], labels=self.labels[i])
@dataclass
class DataCollatorForSupervisedDataset(object):
"""Collate examples for supervised fine-tuning."""
tokenizer: transformers.PreTrainedTokenizer
def __call__(self, instances: Sequence[Dict]) -> Dict[str, torch.Tensor]:
input_ids, labels = tuple([instance[key] for instance in instances] for key in ("input_ids", "labels"))
input_ids = torch.nn.utils.rnn.pad_sequence(
input_ids, batch_first=True, padding_value=self.tokenizer.pad_token_id
)
labels = torch.nn.utils.rnn.pad_sequence(labels, batch_first=True, padding_value=IGNORE_INDEX)
return dict(
input_ids=input_ids,
labels=labels,
attention_mask=input_ids.ne(self.tokenizer.pad_token_id),
)
def make_supervised_data_module(tokenizer: transformers.PreTrainedTokenizer, data_args) -> Dict:
"""Make dataset and collator for supervised fine-tuning."""
train_dataset = SupervisedDataset(tokenizer=tokenizer, data_path=data_args.data_path)
data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
return dict(train_dataset=train_dataset, eval_dataset=None, data_collator=data_collator)
def train():
parser = transformers.HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
plmapi = plm.PlmApi()
model, tokenizer = plmapi.load_hf_model(model_args.model_name,
quantization_config=BitsAndBytesConfig(
load_in_4bit=model_args.load_in_4bit, ),
torch_dtype=model_args.torch_dtype
) # HF model and TokenizerFast
tokenizer.model_max_length = 2048
if model_args.load_in_4bit:
if not model_args.use_lora:
raise ValueError("LoRA must be enabled when loading in 8bit.")
logging.info('Preparing the model (freezing base model params)..')
model = prepare_model_for_kbit_training(model) # now the quantized base model is frozen.
if model_args.use_lora:
model.save_pretrained(save_directory=training_args.output_dir)
# the basemodel is saved now. (config.json, generation_config.json, pytorch_model.bin`)
# this is needed because when lora is enabled, `trainer.save_model` saves the adaptor model only.
peft_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, peft_config)
# now the model is with the LoRA "adaptor" model.
print_trainable_parameters(model) # if LoRA, expect only adaptor model params to be trainable.
special_tokens_dict = dict()
if tokenizer.pad_token is None:
special_tokens_dict["pad_token"] = DEFAULT_PAD_TOKEN
if tokenizer.eos_token is None:
special_tokens_dict["eos_token"] = DEFAULT_EOS_TOKEN
if tokenizer.bos_token is None:
special_tokens_dict["bos_token"] = DEFAULT_BOS_TOKEN
if tokenizer.unk_token is None:
special_tokens_dict["unk_token"] = DEFAULT_UNK_TOKEN
smart_tokenizer_and_embedding_resize(
special_tokens_dict=special_tokens_dict,
tokenizer=tokenizer,
model=model,
)
adam = torch.optim.AdamW(model.parameters(), lr=training_args.learning_rate)
data_module = make_supervised_data_module(tokenizer=tokenizer, data_args=data_args)
trainer = Trainer(model=model, tokenizer=tokenizer, args=training_args,
optimizers=(adam, transformers.get_constant_schedule(adam)),
**data_module)
trainer.train()
trainer.save_model(output_dir=training_args.output_dir)
# todo: With Qlora, sometimes it fails to pickle TrainingArguments. I had no time to debug it yet.
# But the models are saved correctly as pickling `trainer.args` is done at the very end of the process.
# if finetuning (no LoRA), now there are
# [config.json, generation_config.json, pytorch_model.bin,
# special_tokens_map.json, tokenizer_config.json, tokenizer.json, training_args.bin] in the `output_dir`.
#
# if LoRA, now there are
# [adapter_config.json, adapter_model.bin, config.json, generation_config.json, pytorch_model.bin,
# special_tokens_map.json, tokenizer_config.json, tokenizer.json, training_args.bin] in the `output_dir`.
# QLoRa is the same, with additional `README.md`.
if __name__ == "__main__":
train()
```
Data Preprocessing
## Conver csv to json
Usage
```
$ python reformat.py hogehoge.csv
```
```python=
# reformat.py
if __name__ == '__main__':
import sys
import pandas as pd
import json
# コマンドライン引数から入力ファイルのパスを受け取る
# get filepath
input_file = sys.argv[1]
# load dataset by pandas
df = pd.read_csv(input_file)
output_file = input_file.replace('.csv', '.json')
# カラムを抽出
# extract colmuns
diseases = df['diseases'].tolist()
icdcodes = df['icdcodes'].tolist()
drugs = df['drugs'].tolist()
smiless = df['smiless'].tolist()
criteria = df['criteria'].tolist()
status = df['status'].tolist()
# re-format to jsonl
json_data = []
for i in range(len(diseases)):
json_data.append({
"input_text": "disease: {}, icdcodes: {}, drugs: {}, smiless: {}, criteria: {}".format(diseases[i], icdcodes[i], drugs[i], smiless[i], criteria[i]),
"output": status[i]
})
# JSON ファイルに出力
# output json file
with open(output_file, 'w') as f:
json.dump(json_data, f)
```
Google Drive
Gist
Clipboard
Sign in with Wallet
