2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Current SDK version is 0.17.0
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Configure stats pid to 33493
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Loading settings from /root/mistral-finetune/wandb/settings
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
2024-05-25 20:26:13,701 WARNING MainThread:33493 [wandb_setup.py:_flush():76] Could not find program at -m train
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m train'}
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_setup.py:_flush():76] Applying login settings: {}
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_init.py:_log_setup():520] Logging user logs to /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/logs/debug.log
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_init.py:_log_setup():521] Logging internal logs to /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/logs/debug-internal.log
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_init.py:init():560] calling init triggers
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_init.py:init():567] wandb.init called with sweep_config: {}
config: {'data': {'data': '', 'shuffle': False, 'instruct_data': '/root/data/mol_instructions_train.jsonl', 'eval_instruct_data': '', 'instruct': {'shuffle': True, 'dynamic_chunk_fn_call': True}}, 'model_id_or_path': '/root/mistral_models/7B-v0.3', 'run_dir': '/root/mistral-finetune/runs2', 'optim': {'lr': 6e-05, 'weight_decay': 0.1, 'pct_start': 0.05}, 'seed': 0, 'num_microbatches': 1, 'seq_len': 32768, 'batch_size': 1, 'max_norm': 1.0, 'max_steps': 500, 'log_freq': 1, 'ckpt_freq': 100, 'ckpt_only_lora': False, 'no_ckpt': False, 'num_ckpt_keep': 3, 'eval_freq': 100, 'no_eval': True, 'checkpoint': True, 'world_size': 1, 'wandb': {'project': 'CHEMISTral7b-ft', 'offline': False, 'key': 'aaf77f83a4e316f6a8b47fa975ab6b5e73c7c8df', 'run_name': 'run1'}, 'mlflow': {'tracking_uri': None, 'experiment_name': None}, 'lora': {'enable': True, 'rank': 64, 'dropout': 0.0, 'scaling': 2.0}}
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_init.py:init():610] starting backend
2024-05-25 20:26:13,701 INFO    MainThread:33493 [wandb_init.py:init():614] setting up manager
2024-05-25 20:26:13,702 INFO    MainThread:33493 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-05-25 20:26:13,702 INFO    MainThread:33493 [wandb_init.py:init():622] backend started and connected
2024-05-25 20:26:13,704 INFO    MainThread:33493 [wandb_init.py:init():711] updated telemetry
2024-05-25 20:26:13,706 INFO    MainThread:33493 [wandb_init.py:init():744] communicating run to backend with 90.0 second timeout
2024-05-25 20:26:14,054 INFO    MainThread:33493 [wandb_run.py:_on_init():2396] communicating current version
2024-05-25 20:26:14,086 INFO    MainThread:33493 [wandb_run.py:_on_init():2405] got version response 
2024-05-25 20:26:14,087 INFO    MainThread:33493 [wandb_init.py:init():795] starting run threads in backend
2024-05-25 20:26:15,147 INFO    MainThread:33493 [wandb_run.py:_console_start():2374] atexit reg
2024-05-25 20:26:15,147 INFO    MainThread:33493 [wandb_run.py:_redirect():2229] redirect: wrap_raw
2024-05-25 20:26:15,148 INFO    MainThread:33493 [wandb_run.py:_redirect():2294] Wrapping output streams.
2024-05-25 20:26:15,148 INFO    MainThread:33493 [wandb_run.py:_redirect():2319] Redirects installed.
2024-05-25 20:26:15,148 INFO    MainThread:33493 [wandb_init.py:init():838] run started, returning control to user process
2024-05-25 21:29:34,196 INFO    MainThread:33493 [wandb_run.py:_finish():2103] finishing run artsy/CHEMISTral7b-ft/sapihyp4
2024-05-25 21:29:34,196 INFO    MainThread:33493 [wandb_run.py:_atexit_cleanup():2343] got exitcode: 0
2024-05-25 21:29:34,196 INFO    MainThread:33493 [wandb_run.py:_restore():2326] restore
2024-05-25 21:29:34,196 INFO    MainThread:33493 [wandb_run.py:_restore():2332] restore done
2024-05-25 21:29:38,445 INFO    MainThread:33493 [wandb_run.py:_footer_history_summary_info():3994] rendering history
2024-05-25 21:29:38,445 INFO    MainThread:33493 [wandb_run.py:_footer_history_summary_info():4026] rendering summary
2024-05-25 21:29:38,448 INFO    MainThread:33493 [wandb_run.py:_footer_sync_info():3953] logging synced files