Overview ↗
noOriginal Documentation
Documentation Index#
Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt Use this file to discover all available pages before exploring further.
Learn how to create configuration files for sweeps.
A W&B Sweep combines a strategy for exploring hyperparameter values with the code that evaluates them. The strategy can be as simple as trying every option or as complex as Bayesian Optimization and Hyperband (BOHB).
Define a sweep configuration either in a Python dictionary or a YAML file. How you define your sweep configuration depends on how you want to manage your sweep.
Define your sweep configuration in a YAML file if you want to initialize a sweep and start a sweep agent from the command line. Define your sweep in a Python dictionary if you initialize a sweep and start a sweep entirely within a Python script or notebook.
The following guide describes how to format your sweep configuration. See Sweep configuration options for a comprehensive list of top-level sweep configuration keys.
Basic structure#
Both sweep configuration format options (YAML and Python dictionary) utilize key-value pairs and nested structures.
Use top-level keys within your sweep configuration to define qualities of your sweep search such as the name of the sweep (name key), the parameters to search through (parameters key), the methodology to search the parameter space (method key), and more.
For example, the following code snippets show the same sweep configuration defined within a YAML file and within a Python dictionary. Within the sweep configuration there are five top level keys specified: program, name, method, metric and parameters.
Define a sweep configuration in a YAML file if you want to manage sweeps interactively from the command line (CLI)
program: train.py
name: sweepdemo
method: bayes
metric:
goal: minimize
name: validation_loss
parameters:
learning_rate:
min: 0.0001
max: 0.1
batch_size:
values: [16, 32, 64]
epochs:
values: [5, 10, 15]
optimizer:
values: ["adam", "sgd"]
```
<span class="tab-end"></span>
<span class="tab-start" data-tab-title="Python script or notebook"></span>
Define a sweep in a Python dictionary data structure if you define training algorithm in a Python script or notebook.
The following code snippet stores a sweep configuration in a variable named `sweep_configuration`:
```python
sweep_configuration = {
"name": "sweepdemo",
"method": "bayes",
"metric": {"goal": "minimize", "name": "validation_loss"},
"parameters": {
"learning_rate": {"min": 0.0001, "max": 0.1},
"batch_size": {"values": [16, 32, 64]},
"epochs": {"values": [5, 10, 15]},
"optimizer": {"values": ["adam", "sgd"]},
},
}
```
<span class="tab-end"></span>
<span class="tab-group-end"></span>
Within the top level `parameters` key, the following keys are nested: `learning_rate`, `batch_size`, `epoch`, and `optimizer`. For each of the nested keys you specify, you can provide one or more values, a distribution, a probability, and more. For more information, see the [parameters](./sweep-config-keys#parameters) section in [Sweep configuration options](./sweep-config-keys).
## Double nested parameters
Sweep configurations support nested parameters. To define a nested parameter, include an additional `parameters` key under the top-level parameter name.
The following example shows a sweep configuration with three nested parameters: `nested_category_1`, `nested_category_2`, and `nested_category_3`. Each nested parameter includes two additional parameters: `momentum` and `weight_decay`.
<span class="callout-start" data-callout-type="note"></span>
`nested_category_1`, `nested_category_2`, and `nested_category_3` are placeholders. Replace them with names that fit your use case.
<span class="callout-end"></span>
The following code snippets show how to define nested parameters in both a YAML file and a Python dictionary.
<span class="tab-group-start"></span>
<span class="tab-start" data-tab-title="CLI"></span>
```yaml
program: sweep_nest.py
name: nested_sweep
method: random
metric:
name: loss
goal: minimize
parameters:
optimizer:
values: ['adam', 'sgd']
fc_layer_size:
values: [128, 256, 512]
dropout:
values: [0.3, 0.4, 0.5]
epochs:
value: 1
learning_rate:
distribution: uniform
min: 0
max: 0.1
batch_size:
distribution: q_log_uniform_values
q: 8
min: 32
max: 256
nested_category_1:
parameters:
momentum:
distribution: uniform
min: 0.0
max: 0.9
weight_decay:
values: [0.0001, 0.0005, 0.001]
nested_category_2:
parameters:
momentum:
distribution: uniform
min: 0.0
max: 0.9
weight_decay:
values: [0.1, 0.2, 0.3]
nested_category_3:
parameters:
momentum:
distribution: uniform
min: 0.5
max: 0.7
weight_decay:
values: [0.2, 0.3, 0.4]
```
<span class="tab-end"></span>
<span class="tab-start" data-tab-title="Python script or notebook"></span>
```python
{
"program": "sweep_nest.py",
"name": "nested_sweep",
"method": "random",
"metric": {
"name": "loss",
"goal": "minimize"
},
"parameters": {
"optimizer": {
"values": ["adam", "sgd"]
},
"fc_layer_size": {
"values": [128, 256, 512]
},
"dropout": {
"values": [0.3, 0.4, 0.5]
},
"epochs": {
"value": 1
},
"learning_rate": {
"distribution": "uniform",
"min": 0,
"max": 0.1
},
"batch_size": {
"distribution": "q_log_uniform_values",
"q": 8,
"min": 32,
"max": 256
},
"nested_category_1": {
"parameters": {
"momentum": {
"distribution": "uniform",
"min": 0.0,
"max": 0.9
},
"weight_decay": {
"values": [0.0001, 0.0005, 0.001]
}
}
},
"nested_category_2": {
"parameters": {
"momentum": {
"distribution": "uniform",
"min": 0.0,
"max": 0.9
},
"weight_decay": {
"values": [0.1, 0.2, 0.3]
}
}
},
"nested_category_3": {
"parameters": {
"momentum": {
"distribution": "uniform",
"min": 0.5,
"max": 0.7
},
"weight_decay": {
"values": [0.2, 0.3, 0.4]
}
}
}
}
}
```
<span class="tab-end"></span>
<span class="tab-group-end"></span>
{/* For example, the following code snippets show a sweep config both in a YAML config file and a Python script. */}
<span class="callout-start" data-callout-type="warning"></span>
Nested parameters defined in sweep configuration overwrite keys specified in a W\&B run configuration.
As an example, suppose you have `train.py` script that initializes a run with a nested default:
```python
def main():
with wandb.init(config={"nested_param": {"manual_key": 1}}) as run:
# Your training code hereYour sweep configuration defines nested parameters under a top-level "parameters" key:
sweep_configuration = {
"method": "grid",
"metric": {"name": "score", "goal": "minimize"},
"parameters": {
"top_level_param": {"value": 0},
"nested_param": {
"parameters": {
"learning_rate": {"value": 0.01},
"double_nested_param": {
"parameters": {"x": {"value": 0.9}, "y": {"value": 0.8}}
},
}
},
},
}
sweep_id = wandb.sweep(sweep=sweep_configuration, project="<project>")
wandb.agent(sweep_id, function=main, count=4)During a sweep run, run.config["nested_param"] reflects the subtree defined by the
sweep (learning_rate, double_nested_param) config and does not include manual_key defined
in wandb.init(config=...).
Sweep configuration template#
The following template shows how you can configure parameters and specify search constraints. Replace hyperparameter_name with the name of your hyperparameter and any values enclosed in <>.
program: <insert>
method: <insert>
parameter:
hyperparameter_name0:
value: 0
hyperparameter_name1:
values: [0, 0, 0]
hyperparameter_name:
distribution: <insert>
value: <insert>
hyperparameter_name2:
distribution: <insert>
min: <insert>
max: <insert>
q: <insert>
hyperparameter_name3:
distribution: <insert>
values:
- <list_of_values>
- <list_of_values>
- <list_of_values>
early_terminate:
type: hyperband
s: 0
eta: 0
max_iter: 0
command:
- ${Command macro}
- ${Command macro}
- ${Command macro}
- ${Command macro} To express a numeric value using scientific notation, add the YAML !!float operator, which casts the value to a floating point number. For example, min: !!float 1e-5. See Command example.
Sweep configuration examples#
program: train.py
method: random
metric:
goal: minimize
name: loss
parameters:
batch_size:
distribution: q_log_uniform_values
max: 256
min: 32
q: 8
dropout:
values: [0.3, 0.4, 0.5]
epochs:
value: 1
fc_layer_size:
values: [128, 256, 512]
learning_rate:
distribution: uniform
max: 0.1
min: 0
optimizer:
values: ["adam", "sgd"]
```
<span class="tab-end"></span>
<span class="tab-start" data-tab-title="Python script or notebook"></span>
```python
sweep_config = {
"method": "random",
"metric": {"goal": "minimize", "name": "loss"},
"parameters": {
"batch_size": {
"distribution": "q_log_uniform_values",
"max": 256,
"min": 32,
"q": 8,
},
"dropout": {"values": [0.3, 0.4, 0.5]},
"epochs": {"value": 1},
"fc_layer_size": {"values": [128, 256, 512]},
"learning_rate": {"distribution": "uniform", "max": 0.1, "min": 0},
"optimizer": {"values": ["adam", "sgd"]},
},
}
```
<span class="tab-end"></span>
<span class="tab-group-end"></span>
### Bayes hyperband example
```yaml
program: train.py
method: bayes
metric:
goal: minimize
name: val_loss
parameters:
dropout:
values: [0.15, 0.2, 0.25, 0.3, 0.4]
hidden_layer_size:
values: [96, 128, 148]
layer_1_size:
values: [10, 12, 14, 16, 18, 20]
layer_2_size:
values: [24, 28, 32, 36, 40, 44]
learn_rate:
values: [0.001, 0.01, 0.003]
decay:
values: [1e-5, 1e-6, 1e-7]
momentum:
values: [0.8, 0.9, 0.95]
epochs:
value: 27
early_terminate:
type: hyperband
s: 2
eta: 3
max_iter: 27The proceeding tabs show how to specify either a minimum or maximum number of iterations for early_terminate:
The brackets for this example are: [3, 3*eta, 3*eta*eta, 3*eta*eta*eta], which equals [3, 9, 27, 81].
early_terminate:
type: hyperband
min_iter: 3
```
<span class="tab-end"></span>
<span class="tab-start" data-tab-title="Minimum number of iterations"></span>
The brackets for this example are `[27/eta, 27/eta/eta]`, which equals `[9, 3]`.
```yaml
early_terminate:
type: hyperband
max_iter: 27
s: 2
```
<span class="tab-end"></span>
<span class="tab-group-end"></span>
### Macro and custom command arguments example
For more complex command line arguments, you can use macros to pass environment variables, the Python interpreter, and additional arguments. [W\&B supports pre defined macros](./sweep-config-keys#command-macros) and custom command line arguments that you can specify in your sweep configuration.
For example, the following sweep configuration (`sweep.yaml`) defines a command that runs a Python script (`run.py`) with the `${env}`, `${interpreter}`, and `${program}` macros replaced with the appropriate values when the sweep runs.
The `--batch_size=${batch_size}`, `--test=True`, and `--optimizer=${optimizer}` arguments use custom macros to pass the values of the `batch_size`, `test`, and `optimizer` parameters defined in the sweep configuration.
```yaml
program: run.py
method: random
metric:
name: validation_loss
parameters:
learning_rate:
min: 0.0001
max: 0.1
command:
- ${env}
- ${interpreter}
- ${program}
- "--batch_size=${batch_size}"
- "--optimizer=${optimizer}"
- "--test=True"The associated Python script (run.py) can then parse these command line arguments using the argparse module.
# run.py
import wandb
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--batch_size', type=int)
parser.add_argument('--optimizer', type=str, choices=['adam', 'sgd'], required=True)
parser.add_argument('--test', type=str2bool, default=False)
args = parser.parse_args()
# Initialize a W&B Run
with wandb.init('test-project') as run:
run.log({'validation_loss':1})See the Command macros section in Sweep configuration options for a list of pre-defined macros you can use in your sweep configuration.
Boolean arguments#
The argparse module does not support boolean arguments by default. To define a boolean argument, you can use the action parameter or use a custom function to convert the string representation of the boolean value to a boolean type.
As an example, you can use the following code snippet to define a boolean argument. Pass store_true or store_false as an argument to ArgumentParser.
import wandb
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--test', action='store_true')
args = parser.parse_args()
args.test # This will be True if --test is passed, otherwise FalseYou can also define a custom function to convert the string representation of the boolean value to a boolean type. For example, the following code snippet defines the str2bool function, which converts a string to a boolean value.
def str2bool(v: str) -> bool:
"""Convert a string to a boolean. This is required because
argparse does not support boolean arguments by default.
"""
if isinstance(v, bool):
return v
return v.lower() in ('yes', 'true', 't', '1')