Cloud integration and automatisation

GCP

GCP is used for two things

  • Storing model weights and training info

  • Storing pre-processed datasets (to avoid re-processing them)

This can be setup as follows

  1. Connect to GCP using the GCP command line. This will store credentials in a GCP-specific location, and then nugets will be able to use them. If you are running as normal user, the command is

    $ gcloud auth

    If running on a google cloud machine, there is an automated way to do the login. The command is :

    $ gcloud activate-service-account

  2. Store appropriate bucket names in the config file (see Config file)

Weights and biases

To use the weights and biases integration, you need to fill out three variables in the config file (see Config file).

wandb_key: [the api key for weights and biases]
wandb_project: [the project name for weights and biases]

Once that is done, the project will use W&B for 2 things:

  1. logging training runs.

  2. sweeps (see next paragraph)

Hyperparameter sweeps

They are run through weights and biases, by using the W&B Sweep API

Sweeps in weights and biases rely on two things (I would recommend using yaml and cli, in a tracked git directory, it looks easier to keep track of)

  1. a config for the sweep (that is passed to the agent). Note that we use nested parameters, the configuration structure for the hyperparameters is the same one as the one in training-run-config

  2. an controller that orchestrates the sweeps. This can be the W&B cloud, and it can be run through the wandb sweep command line (see here)

  3. workers (also called agents), which are processes that are running on the actual machines with GPUs. The workers get info from the agent on the actual training passes to run, and run them.

Workers can be started with

./nugets.sh wandb_agent $SWEEP_ID

where $SWEEP_ID is the sweep id returned by the wandb sweep cli.