Skip to content

Running Jobs in the Cloud

This topic describes the basic usage of the ml run xp command to schedule jobs in the cloud using MissingLink's Resource Manager. "Cloud" can mean either a local grid of on-premises servers, or a cloud service such as AWS.

What is a job?

A MissingLink job is a set of one or more commands.

When you run commands on your computer or server, in addition to the file executed, you also have environment variables, a code environment, command arguments and the operating system that are all part of executing the command.

A MissingLink job is very similar. For example, when running the command:

echo "Hello, {NAME}"
You are assuming that all of the following are involved:

  • An environment variable (NAME)
  • Command line arguments for echo ("Hello, {NAME}")
  • Your operating system for providing you with the echo command and the shell that will replace the {NAME} with its value


    To better understand the principles at work in running jobs using MissingLink Resource Management, familiarize yourself with the basics of docker.

Hello World!

To illustrate, examine a simple "hello world" job:

ml run xp --command 'echo "Hello {NAME}"' --env NAME "World!" --image bash 

Setting up your environment for supporting code tracking

In order to run a new job using MissingLink's resource management, you must provide the job with a Git repository.

For instructions in configuring this connection, see Setting up Code Tracking in MissingLink.

ml run xp flags

The flags that are available for the ml run xp command are identical to those for the ml run local command (except for a few). For a description of the flags, see Flags for the ml run local command.

Running Jupyter Notebook in Resource Manager

You can run Jupyter Notebook as a job within Resource Manager.

Use the following command:

ml run xp --command 'jupyter nbconvert --to html --output /output/<target filename> --execute /code/<noteboook filename>.ipynb --ExecutePreprocessor.allow_errors=True --ExecutePreprocessor.timeout=-1' 

Setting defaults

When using Resource Management commands, you can set default org and project values by running:

ml defaults set org MY_ORG
ml defaults set project MY_PROJECT

Once you have set the defaults, you can cut down the hello world example that appears above to:

ml run xp --command 'echo "Hello {NAME}"' --env NAME "World!" --image bash

From this point on, the org and project parameters do not need to be added; they are understood.