Skip to content

Jobs Commands

The ml run``ml run command group provides facilities for running jobs.

The following commands are available with ml run:

local

Runs jobs locally.

The following flags are available with the ml run local command:

  • --link-aws (default)/--no-aws-link

    When configuring the aws-cli an .aws folder is created in your home directory. The folder contains your AWS credentials. When “linked,” the folder will be mounted in your jobs and Resource Manager, allowing them to use the credentials of the host to access AWS services.

  • --env-aws (default)/--no-aws-env

    Similar to the .aws folder, AWS allows authentication using environment variables.

  • --link-gcp (default)/--no-gcp

    Similar to AWS, GCP also has its default directory and this flag allows the folder to be mounted in your jobs and Resource Manager to call GCP APIs.

  • --link-azure (default)/--no-azure

    Similar to AWS. Azure works exactly like the link-aws and link-gcp options, mounting the .azure folder into a container. Defaults to --link-azure.

  • --org TEXT

    Organization to use.

    This flag and --project can be set to default values to save you from providing them on every command. Ensures that the experiments you run from this job are assigned to the desired project in MissingLink.

  • --project, -p INTEGER

    Project Id to hold the experiments started by the job. Ensures that the experiments you run from this job are assigned to the desired project in MissingLink.

    This flag and --org can be set to default values to save you from providing them on every command.

  • --name, -n TEXT

    Specifies a name for the job. If no name is provided, the name will be auto-generated.

  • --name-prefix, -np TEXT

    Specifies a name prefix for the job. If the --name was not provided, a random string will be appended to the prefix as the job name.

  • --image TEXT

    Docker image to use for the job. Defaults to missinglinkai/tensorflow.

  • --git-repo TEXT

    Git repository to pull the code from during the execution of the job.

  • --git-tag TEXT

    Git branch/tag for the git repository. Defaults to master. The cloned code will be available under /code.

  • --git-identity PATH

    [Secure] If provided, the path will be used as git (SSH) identity when pulling code, otherwise your default organization identity will be used.

    See Confidential Data.

  • --no-source

    Do not pass source code for this job.

  • --source-dir TEXT

    Directory that holds code and tracking repository configuration for the experiment. Configure source tracking to be able to use.

  • --command TEXT

    A single command to execute.

  • --gpu (default)/--cpu

    Specifies to run this job on a GPU or CPU.

  • --data-volume TEXT

    Data volume to clone data from.

  • --queue

    Resource queue in which to run the experiment.

  • --data-query TEXT

    Query to execute on the data volume.

  • --data-dest TEXT

    Destination folder and format for cloning data. If provided, must begin with /data.

  • --data-iterator BOOLEAN

    When set to True, data will not be cloned before the experiment and the query will be available for the SDK iterator

  • --recipe, -r RECIPE PATH

    Specifies a non-default recipe file. The recipe file is a YAML file containing flag:value pairs that allows you to specify default values for all parameters for this function. .ml_recipe.yaml is the default recipe file.

  • --save-recipe TEXT

    Saves a recipe for this call to the target file and quits, without scheduling a job. Note that the default values are not encoded into the recipe.

  • --env TEXT TEXT...

    A single environment variable to pass for the invocation in key-value format. You can use this flag multiple times.

  • --output-paths TEXT

    Paths that will be exported to the Data Management at the end of the invocation job. The paths will be available to the running code under /path_name.

    Defaults to /output.

  • --persistent-path TEXT TEXT...

    Maps a path local to the server that is running the job, as a path inside the docker.

    A persistent path is a mount from the SOURCE path on the server that is hosting the job execution to the TARGET path of the docker container that executes the job.

  • --secure-env, -se TEXT TEXT...

    [Secure] Provides additional environment variables to the job. The format is env_key env_value <!--Underscores or hyphens?>.

    See Confidential Data.

  • --docker-host TEXT

    [Secure] If docker login is needed to pull the image, login to this host.

    See Confidential Data.

  • --docker-user TEXT

    [Secure] If docker login is needed to pull the image, login with this user.

    See Confidential Data.

  • --docker-password TEXT

    [Secure] If docker login is needed to pull the image, login with this password.

    See Confidential Data.

  • --requirements-txt PATH

    Installs the pip requirements file from this path (relative to the repo). Defaults to requirements.txt.

  • --disable-colors

    Disables colors in logs.

  • --attach

    Waits before exiting the job submission and prints logs of the submitted job in the command line.

xp

Runs jobs in the cloud.

Most flags that are available for the ml run local command are also available with ml run xp.

Note

The following flags are available with ml run local but are NOT available with ml run xp:

  • --link-aws/--no-aws-link

  • --env-aws/--no-aws-env

  • --link-gcp/--no-gcp

  • --link-azure / --no-azure

logs

Attaches a log to a job after it has been submitted.

The following flags are available with the ml run logs command:

  • --org TEXT

    Organization to use.

  • -- job

    Job Id.

  • -- disable-colors

    Disables colors in logs.