Skip to content

Jobs Methods

run.xp

as_api().run.xp(org=None, project=None, image=None, git_repo=None, no_source=False, git_tag=None, source_dir=None, command=None, gpu=None, data_volume=None, queue=None, data_query=None, data_dest= None, data_iterator=None, recipe=None, save_recipe=None, env=None, output_paths=None, git_identity=None, persistent_path=None, secure_env=None, docker_host=None, docker_user=None, docker_password=None, requirements_txt=None, disable_colors=None, attach=None, name=None, name_prefix=None, shm_size=None)
Run jobs in the cloud.

Parameters

Wherever [Secure] appears in the following descriptions, see Confidential Data for more information.

  • org: Optional. String. Organization to use. This parameter and project can be set to default values to save you from providing them on every call. Ensures that the experiments you run from this job are assigned to the desired project in MissingLink.
  • project: Optional. Integer. Project Id to hold the experiments started by the job. Ensures that the experiments you run from this job are assigned to the desired project in MissingLink. This parameter and org can be set to default values to save you from providing them on every call.

  • image: Optional. String. Docker image to use for the job. Defaults to missinglinkai/tensorflow.

    • For Docker images in a private Azure Container Registry (ACR), specify the image in the following format:

      <acr_name>.azurecr.io/<image_name>:<version_tag>.

    • For Docker images in a private Amazon Elastic Container Registry (ECR), specify the image in the following format:

      <ecr_id>.dkr.ecr.<region>.amazonaws.com/<image_name>:<version_tag>

  • git_repo: Optional. String. Git repository to pull the code from during the execution of the job.

  • no_source: Optional. Boolean. Pass source code for this job? Default is False.
  • git_tag: Optional. String. Git branch/tag for the git repository. Defaults to master. The cloned code will be available under /code.
  • source_dir: Optional. String. Directory that holds code and tracking repository configuration for the experiment.

  • command: Optional. A list of strings of all the command arguments, for example:

    command = ['python', '/path/to/script/my_script.py']

  • gpu: Optional. Boolean. Run this job on a GPU or a CPU?
    Default is True.

  • data_volume: Optional. String. Data volume to clone data from.
  • queue: Optional. String. Resource queue in which to run the experiment.
  • data_query: Optional. String. Query to execute on the data volume.
  • data_dest: Optional. String. Destination folder and format for cloning data. If provided, must begin with /data.
  • data_iterator: Optional. Boolean. When set to True, data will not be cloned before the experiment and the query will be available for the SDK iterator.
  • recipe: Optional. String. Specifies a non-default recipe file. The recipe file is a YAML file containing flag:value pairs that allows you to specify default values for all parameters for this function. .ml_recipe.yaml is the default recipe file.
  • save_recipe: Optional. String. Saves a recipe for this call to the target file and quits, without scheduling a job. Note that the default values are not encoded into the recipe.
  • env: Optional. A list, whose members are Tuples of two values. The first is the property name (string) and the second is the property value (string). Environment variables to pass for the invocation in key-value format.
  • output_paths: Optional. List of strings. Paths that will be exported to Data Management at the end of the invocation job. The paths will be available to the running code under /path_name.
  • git_identity: Optional. String. [Secure] If provided, the path will be used as git (SSH) identity when pulling code, otherwise your default organization identity will be used.
  • persistent_path: Optional. A list, whose members are Tuples of two strings. The first is the container path and the second is the host path. Maps a path local to the server that is running the job, as a path inside the docker. A persistent path is a mount from the SOURCE path on the server that is hosting the job execution to the TARGET path of the docker container that executes the job.
  • secure_env: Optional. A list, whose members are Tuples of two strings. [Secure] Provides additional environment variables to the job. The format is the same as in env.
  • docker_host: Optional. String. [Secure] If a docker login is needed to pull the image, log in to this host.
  • docker_user: Optional. String. [Secure] If a docker login is needed to pull the image, log in with this user name.
  • docker_password: Optional. String. [Secure] If a docker login is needed to pull the image, log in with this password.
  • requirements_txt: Optional. String. Installs the pip requirements file from this path (relative to the repo). Defaults to requirements.txt.
  • disable_colors: Optional. Boolean. Disable colors in logs? Default is False.
  • attach: Optional. Boolean. Wait before exiting the job submission and print logs of the submitted job in the command line? Default is False.
  • name: Optional. String. Specifies a name for the job. If no name is provided, the name will be auto-generated.
  • name_prefix: Optional. String. Specifies a name prefix for the job. If no argument is provided for name, a random string will be appended to the prefix as the job name.
  • Optional. String.
  • shm_size: Optional. String. Allows manual setting of the size of the shared memory folder /dev/shm in the job container. This flag may be useful if your program uses large amounts of space in that folder. For example, some PyTorch methods might require this.

    The format is <number><unit>.

    • <number> must be greater than 0.
    • <unit> is optional. You can specify b (bytes), k (kilobytes), m (megabytes), or g (gigabytes). If you omit the unit, the system uses bytes.

run.logs

as_api().run.logs(org, job=None, disable_colors=False)
Attach a log to a job after it has been submitted.

Parameters

  • org: String. Organization to use.
  • job: Optional. String. Job Id.
  • disable_colors: Optional. Boolean. Enable colors in the console? Default is False.

run.local

as_api().run.local(link_aws=True, env_aws=True, link_gcp=True, link_azure=True, org=None, cache_path=None, project=None, image=None, git_repo=None, no_source=False, git_tag=None, source_dir=None, command=None, gpu=None, data_volume=None, queue=None, data_query=None, data_dest=None, data_iterator=None, recipe=None, save_recipe=None, env=None, output_paths=None, git_identity=None, persistent_path=None, secure_env=None, docker_host=None, docker_user=None, docker_password=None, requirements_txt=None, disable_colors=None, attach=None, name=None, name_prefix=None)

Run jobs locally.

Parameters

Wherever [Secure] appears in the following descriptions, see Confidential Data for more information.

  • link_aws: Optional. Boolean. Link .aws folder? When configuring the aws-cli an .aws folder is created in your home directory. The folder contains your AWS credentials. When “linked,” the folder will be mounted in your jobs and Resource Manager, allowing them to use the credentials of the host to access AWS services. Default is True.
  • env_aws: Optional. Boolean. Similar to the .aws folder, AWS allows authentication using environment variables. Default is True.
  • link_gcp: Optional. Boolean. Similar to AWS, GCP also has its default directory and this flag allows the folder to be mounted in your jobs and Resource Manager to call GCP APIs. Default is True.
  • link_azure: Optional. Boolean. Similar to AWS. Azure works exactly like the link-aws and link-gcp options, mounting the .azure folder into a container. Default is True.
  • cache_path: Optional. String. Path for pip, conda, and MissingLink caches.

For a description of the remaining parameters, see the run.xp method.