Running Jobs on a Workstation

This topic describes the basic usage of the ml run local command to schedule jobs on a workstation using MissingLink's Resource Manager.

What is a job?

A MissingLink job is a set of one or more commands.

When you run commands on your computer or server, in addition to the file executed, you also have environment variables, a code environment, command arguments and the operating system that are all part of executing the command.

A MissingLink job is very similar. For example, when running the command:

$ echo "Hello, $NAME"

You are assuming that all of the following are involved:

• An environment variable (NAME)
• Command line arguments for echo ("Hello, $NAME")
• Your operating system for providing you with the echo command and the shell that will replace the $NAME with its value.

To better understand the principles at work in running jobs using MissingLink Resource Management, familiarize yourself with the basics of docker.

Hello World!

To illustrate, examine a simple "hello world" job:

$ ml run local --command 'echo "Hello $NAME"' --env NAME "World" --image bash 

Flags for the ml run local command

The following describes the flags that are available for configuring general settings, more advanced settings, and authentication to cloud providers. The flags are available for both the ml run local and ml run xp commands.

General flags

• --command specifies a single command to be executed.
• --env specifies a single environment variable. See Environment variables.
• --image specifies the docker image to be used for this job. In this case, all that is needed is the bash image. MissingLink provides a few images that contain practically anything you need for machine learning.
• --org and --project specify where this job should be scheduled and make sure experiments that you run from this job are assigned to the desired project in MissingLink. Both of these parameters can be set to default values so that you do not need to provide them on every command. For more information, see Setting Defaults.
• -n specifies a name for the job. If no name is provided, the name will be auto-generated.
• -np specifies a name prefix for the job. If the -n was not provided, a random string will be appended to the prefix as the job name.

Flags for more advanced settings

There are more advanced features available with the ml run local command for running jobs on your workstation. Use of the flags is described in:

• Using Recipe Files and Python Requirements Files
• Specifying Inputs and Outputs for Jobs
• Confidential Data
• Git Access Policies

Flags for authentication to cloud providers

If your data is stored in the cloud, you must authenticate the job with your cloud credentials.

For more information, see Allowing Access in Hybrid Cloud

Running Jupyter Notebook in Resource Manager

You can run Jupyter Notebook as a job within Resource Manager.

Use the following command:

$ ml run local --command 'jupyter nbconvert --to html --output /output/<target filename> --execute /code/<noteboook filename>.ipynb --ExecutePreprocessor.allow_errors=True --ExecutePreprocessor.timeout=-1' 

Setting defaults

When using Resource Management commands you can set default org and project values by running:

$ ml defaults set org MY_ORG
$ ml defaults set project MY_PROJECT

Once you have set the defaults, you can cut down the hello world example that appears above to:

$ ml run local --command 'echo "Hello $NAME"' --env NAME "World!" --image bash

From this point on, the org and project parameters do not need to be added; they are understood.