CODEWITHSUNDEEP
pythonhyperparameterswandb
kkgarg
kkgarg

Reputation: 1376

wandb.wandb_agent - ERROR - Detected 5 failed runs in a row, shutting down

While trying to setup wandb, I am facing the following error:

wandb: WARNING Calling wandb.login() after wandb.init() has no effect.                                                                                                            
2021-07-23 19:19:32,639 - wandb.wandb_agent - INFO - Running runs: []                                                                                                             
2021-07-23 19:19:32,824 - wandb.wandb_agent - INFO - Agent received command: run                                                                                                  
2021-07-23 19:19:32,825 - wandb.wandb_agent - INFO - Agent starting run with config:                                                                                              
        lr: 0.01                                                                                                                                                                  
        optimizer: Adam                                                                                                                                                           
2021-07-23 19:19:32,826 - wandb.wandb_agent - INFO - About to run command: /usr/bin/env python  --lr=0.01 --optimizer=Adam                                                        
/home/ubuntu/anaconda3/envs/pytorch_latest_p37/bin/python: can't find '__main__' module in ''                                                                                     
2021-07-23 19:19:37,945 - wandb.wandb_agent - INFO - Running runs: ['e8ff7j11']                                                                                                   
2021-07-23 19:19:37,946 - wandb.wandb_agent - INFO - Cleaning up finished run: e8ff7j11

------4 more runs for different hyperparamters-------

2021-07-23 19:19:59,139 - wandb.wandb_agent - ERROR - Detected 5 failed runs in a row, shutting down.
2021-07-23 19:19:59,139 - wandb.wandb_agent - INFO - To change this value set WANDB_AGENT_MAX_INITIAL_FAILURES=val
wandb: Terminating and syncing runs. Press ctrl-c to kill.

Code:

base_config.py

class base_config:
    def __init__(self):
        self.epochs = 10
        self.sweep_config = {
            'method': 'grid',
            'metric': {
              'name': 'val_F1@M',
              'goal': 'maximize'  
            },
            'parameters': {
                'lr': {
                    'values': [1e-2, 1e-3, 1e-4]
                },
                'optimizer': {
                    'values': ['Adam', 'SM3']
                },
            }
        }
        self.config_defaults = {
            'lr': 1e-2,
            'optimizer': 'Adam',
        }

train.py

import wandb
def run(args, config):
    # wandb.log()

def run_and_collect_results(args, config):
    wandb.init(config=config['config_defaults'])
    config.update({k: v for k, v in wandb.config.items()})
    run(args, config)

if __name__ == '__main__':
    # load config from config file
    # load args
    sweep_id = wandb.sweep(config['sweep_config'], project="Pytorch-sweeps")
    wandb.agent(sweep_id, run_and_collect_results(args, config))

I am not sure what is the correct way to write the agent for wandb. The current code ends up with logs like python --lr=0.01 --optimizer=Adam . The file name seems missing. In that case, would I need to write wandb.agent in a separate file or use CLI interface? I was expecting the behavior that wandb.agent would call function run_and_collect_results for different hyperaparameters.

Upvotes: 1

Views: 2071

Answers (2)

lolftw
lolftw

Reputation: 35

The issue is that wandb.agent() is only recommended to be called from Jupyter notebooks and not raw python scripts. The recommended action for python scripts is using a .yaml configuration file and running wandb agent script_id from the command line.

Here is an official comment from the W&B github: https://github.com/wandb/client/issues/2282#issuecomment-860906921

Documentation: https://docs.wandb.ai/guides/sweeps/quickstart

Upvotes: 1

Artyom Hakobyan
Artyom Hakobyan

Reputation: 11

I believe the issue is in your wandb.agent() function call. It's supposed to get a function (name only) as an argument, so the agent knows which function to call with different arguments.

But you are passing run_and_collect_results(args, config)

Instead of passing the function name you are actually calling the function (have parentheses and arguments after the function name), and passing the result value.

Upvotes: 1

Related Questions