Running Jobs on a Workstation
This topic describes the basic usage of the ml run local
command to schedule jobs on a workstation using MissingLink's Resource Manager.
What is a job?
A MissingLink job is a set of one or more commands.
When you run commands on your computer or server, in addition to the file executed, you also have environment variables, a code environment, command arguments and the operating system that are all part of executing the command.
A MissingLink job is very similar. For example, when running the command:
$ echo "Hello, $NAME"
- An environment variable (
NAME
) - Command line arguments for
echo
("Hello, $NAME"
) - Your operating system for providing you with the
echo
command and the shell that will replace the$NAME
with its value.
To better understand the principles at work in running jobs using MissingLink Resource Management, familiarize yourself with the basics of docker.
Hello World!
To illustrate, examine a simple "hello world" job:
$ ml run local --command 'echo "Hello $NAME"' --env NAME "World" --image bash
Flags for the ml run local
command
The following describes the flags that are available for configuring general settings, more advanced settings, and authentication to cloud providers. The flags are available for both the ml run local
and ml run xp
commands.
General flags
--command
specifies a single command to be executed.--env
specifies a single environment variable. See Environment variables.--image
specifies the docker image to be used for this job. In this case, all that is needed is thebash
image. MissingLink provides a few images that contain practically anything you need for machine learning.--org
and--project
specify where this job should be scheduled and make sure experiments that you run from this job are assigned to the desired project in MissingLink. Both of these parameters can be set to default values so that you do not need to provide them on every command. For more information, see Setting Defaults.-n
specifies a name for the job. If no name is provided, the name will be auto-generated.-np
specifies a name prefix for the job. If the-n
was not provided, a random string will be appended to the prefix as the job name.
Flags for more advanced settings
There are more advanced features available with the ml run local
command for running jobs on your workstation. Use of the flags is described in:
- Using Recipe Files and Python Requirements Files
- Specifying Inputs and Outputs for Jobs
- Confidential Data
- Git Access Policies
Flags for authentication to cloud providers
If your data is stored in the cloud, you must authenticate the job with your cloud credentials.
For more information, see Allowing Access in Hybrid Cloud
Running Jupyter Notebook in Resource Manager
You can run Jupyter Notebook as a job within Resource Manager.
Use the following command:
$ ml run local --command 'jupyter nbconvert --to html --output /output/<target filename> --execute /code/<noteboook filename>.ipynb --ExecutePreprocessor.allow_errors=True --ExecutePreprocessor.timeout=-1'
Setting defaults
When using Resource Management commands you can set default org
and project
values by running:
$ ml defaults set org MY_ORG $ ml defaults set project MY_PROJECT
Once you have set the defaults, you can cut down the hello world
example that appears above to:
$ ml run local --command 'echo "Hello $NAME"' --env NAME "World!" --image bash
From this point on, the org
and project
parameters do not need to be added; they are understood.