How to run docker on a DPO model

[tamanna-mostafa]

1. I fine-tuned mistral 7b model with preference data (32k).
2. Then I ran DPO on the fine tuned model with 12k data.
   This is the command I used to run docker:

accelerate launch --config_file ./accelerate_configs/ds_zero3.yaml rlhf_dpo.py \
--model_name_or_path="/mnt/efs/data/tammosta/files_t/output_sft_32k" \
--output_dir="/mnt/efs/data/tammosta/files_t/DPO_output_mistral_32k" \
--data_path="/mnt/efs/data/tammosta/files_t/DPO_data_rbs_clean_AIF.json" \
--use_lamma2_peft_config False \
--beta 0.1 \
--optimizer_type adamw_hf \
--learning_rate 1e-6 \
--warmup_steps 50 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 8 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_r 8 \
--max_prompt_length 2048 \
--max_length 4096 \
--num_train_epochs 4 \
--logging_steps 20 \
--save_steps 100 \
--save_total_limit 8 \
--eval_steps 50 \
--gradient_checkpointing True \
--report_to "wandb"

3. Now, I need to run inference on the DPO model.
   I ran the following commands for this:

model=/data/DPO_output_mistral_32k
volume=/mnt/efs/data/tammosta/files_t:/data
num_shard=8
docker run --gpus all --shm-size 1g -p 172.31.8.218:80:80 -v $volume ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id $model --num-shard $num_shard --max-input-length 4095 --max-total-tokens 12000

However, the docker failed to initialize the model with the following error:

OSError: /data/DPO_output_mistral_32k does not appear to have a file named config.json. Checkout ' https://huggingface.co//data/DPO_output_mistral_32k/None ' for available files.

Does anyone know how to create/find the config.json file?
I'll highly appreciate any help.

[Narsil]

I think you could just copy the original config from the original repo over (with tokenizer etc..).