Skip to content

Running Jobs in the Cloud

This topic describes the basic usage of the ml run xp command to schedule jobs in the cloud using MissingLink's Resource Manager. "Cloud" can mean either a local grid of on-premises servers, or a cloud service such as AWS.

What is a job?

A MissingLink job is a set of one or more commands.

When you run commands on your computer or server, in addition to the file executed, you also have environment variables, a code environment, command arguments and the operating system that are all part of executing the command.

A MissingLink job is very similar. For example, when running the command:

$ echo "Hello, $NAME"
You are assuming that all of the following are involved:

  • An environment variable (NAME)
  • Command line arguments for echo ("Hello, $NAME")
  • Your operating system for providing you with the echo command and the shell that will replace the $NAME with its value.


    To better understand the principles at work in running jobs using MissingLink Resource Management, familiarize yourself with the basics of docker.

Hello World!

To illustrate, examine a simple "hello world" job:

$ ml run xp --command 'echo "Hello $NAME"' --env NAME "World!" --image bash 

ml run xp Flags

All the flags for the ml run xp command are identical to those available for the ml run local command. For a description of the flags, see Flags for the ml run local command.

Running Jupyter Notebook in Resource Manager

You can run Jupyter Notebook as a job within Resource Manager.

Use the following command:

$ ml run xp --command 'jupyter nbconvert --to html --output /output/<target filename> --execute /code/<noteboook filename>.ipynb --ExecutePreprocessor.allow_errors=True --ExecutePreprocessor.timeout=-1' 

Setting defaults

When using Resource Management commands, you can set default org and project values by running:

$ ml defaults set org MY_ORG
$ ml defaults set project MY_PROJECT

Once you have set the defaults, you can cut down the hello world example that appears above to:

$ ml run xp --command 'echo "Hello $NAME"' --env NAME "World!" --image bash

From this point on, the org and project parameters do not need to be added; they are understood.