Skip to content

Experiments Commands

This command group provides facilities for performing actions on experiments.

The following commands can be used together with ml experiments:

list

Lists the experiments under the specified project.

The following flags are available with the ml experiments list command:

  • --project, -p

    Project Id.

    If you do not specify a project:

    • If there is only one, MissingLink will use that.
    • If there is more than one project, MissingLink will prompt you to choose a project from those found.

logs

Lets you view the execution logs of an experiment that is running in a remote location.

The following flags are available with the ml experiments logs command:

  • --project, -p

    Project Id.

  • --experiment, -e

    Experiment Id.

  • --disable-colors

    Disables colors in the console.


update-chart

Injects a custom chart into a given experiment.

The charts can be single line charts (having a single Y value per X point) or multipoint charts (having multiple Y values per X point). The experiment can be identified by the project and experiment Ids or by model weights hash.

The following flags are available with the ml experiments update-chart command:

  • --weights-hash, -wh SHAI1 HEX

    Hexadecimal sha1 hash of the model's weights.

    Not required if --project and --experiment are provided.

  • --project, -p INT

    Project Id to find experiments.

    Use ml projects list to find your project Ids.

    Not required if --weights-hash is provided.

  • --experiment, -e INT

    The experiment Id.

    Use ml experiments list to find your experiment Ids.

    Not required if --weights-hash is provided.

  • --chart-name, -c STR

    The name of the chart. The name is used to identify the chart against different experiments and through the same experiment.

    Required.

  • --chart-x-name, -cxn STR

    Display name for chart's X axis.

    Optional.

  • --chart-y-name, -cyn JSON STR

    Display name for chart's Y axis.

    Optional.

  • --chart-scope, -cs [test | validation | train]

    Scope type. Defaults to test.

  • --chart-type, -ct [line]

    Graph type. Defaults to line.

  • --chart-x, -cx JSON STR

    Array of m data points (X axis).

    Can be Strings, Integers or Floats.

    Required.

  • --chart-y, -cy JSON STR

    Array/matrix of m data values.

    Can be either an array m of Integers/Floats or a matrix (m arrays having n Ints/Floats each), a full matrix describing the values of every chart in every data point.

    Required.

Examples

Send a single point chart for experiment 1 in project 123:

ml experiments update-chart  --experiment 1 \
    --chart-name "my chart name" --chart-scope train \
    --chart-x '[0.1,0.5,0.8]' --chart-y '[0.9,0.5,0.2]' \
    --chart-x-name "X" --chart-y-name "Y"

Send a multipoint chart for the experiment associated with weights hash aa918d88a21eedbf6:

ml experiments update-chart --weights-hash aa918d88a21eedbf6  \
    --chart-name ililil --chart-scope train  \
    --chart-x '[4,8,12]' --chart-y '[[1,2,4],[2,4,8],[3,6,12]]' \
    --chart-x-name "Precision" --chart-y-name "Recall"

update-metrics

Adds metrics to an experiment that has already run.

There are two ways to identify the experiment:

  • Specify both --project_id and --experiment_id options.

  • Specify the --weights-hash option.

    When the model's weights hash is specified, MissingLink looks up the experiment that the model belongs to.

    When both --project_id and --experiment_id are specified, the --weights-hash option is ignored.

Example

To send metrics to the fifth experiment of the project "123", run

ml experiments update_metrics --project_id 123 --experiment_id 5 --metrics '{"ex_cost": 99}'

Or assuming that the model's weights hash "324e16b5e" was generated by this experiment at certain epoch or iteration, run

ml experiments update_metrics --weights-hash 324e16b5e --metrics '{"ex_cost": 99}'

The following flags are available with the ml experiments update-metrics command:

  • --project, -p INTEGER

    Project Id.

    Use ml projects list to find your project Ids.

  • --experiment, -e INT

    Experiment Id.

    Use ml experiments list to find your experiment Ids.

  • --weights-hash, -wh SHAI1 HEX

    Hexadecimal sha1 hash of the model's weights.

  • --metrics, -m JSON STR

    Metrics of the experiment as a jsonified string. The key should be the metric name with an "ex" prefix. For example, "ex_cost". The value is the metric value in String, Float, Integer or Boolean format.

    Required.

See also

Examples

There are three levels for metrics: experiment, epoch and iteration metrics. To send experiment metrics, you can specify the project Id and experiment Id.

ml experiments update-metrics  \
    --experiment myExperimentID \
    --metrics '{"ex_cost": 99}'

If you have a model's weights hash at a certain epoch or iteration of the experiment, you can provide the hash and MissingLink will find the experiment for you. Let's say your model's weights hash is 324e16b5e, you can send the experiment metrics using the following command:

ml experiments update-metrics \
    --weights-hash 324e16b5e --metrics '{"ex_cost": 99}'
MissingLink will automatically identify the right iteration from the weights hash and inject the metrics to all relevant iterations.


update-metrics-per-iteration

Sends external metrics for a specific iteration.

The command lets you create a graph of the metrics for each iteration. The experiment and iterations are obtained from the weighted hash.

Each model's weights hash is attached to certain experiment epochs and thus can be used to send metrics that are relevant to those epochs. The model weights hash is a hexadecimal string. To calculate the weights hash of a model:

  • Calculate the sha1 strings of the weights for each layers

  • Calculate the sha1 string of the combine hashes.

    For example, the model has three layers with the layers' weight hashes: ['abc', '123', 'def'], the model weight hash is sha1('abc123def')

Warning

The same model weights hash can appear in different experiments or different epochs (for example, when running the same net twice). As such, this command will send the metrics to all these experiments/epochs that it can identify from the hash.

Example

To send metrics of the model whose hash is 324e16b5e, run

ml experiments updateMetricsPerIteration --weights-hash 324e16b5e --metrics '{"ex_cost": 99}'

The following flags are available with the ml experiments update-metrics-per-iteration command:

  • --weights-hash, -wh SHAI1 HEX

    Hexadecimal sha1 hash of the model's weights.

    Required.

  • --metrics, -m JSON STR

    Metrics of the experiment as a jsonified string. The key should be the metric name with an "ex" prefix. For example, "ex_cost". The value is the metric value in String, Float, Integer or Boolean format.

    Required.

See also

ml models get-weights-hash for getting the current weights hash.