A queue is responsible for executing the jobs that are assigned to it. It also allocates to the job the resources that are defined for the resource groups with which the job is associated.
For a job to run, it must be assigned to a queue. If you do not assign the job to a queue, it will be assigned to the default queue that is created when you configure Resource Management for the first time.
Many organizations have multiple teams running different experiments that require different server types. MissingLink's Resource Queue lets the teams run their experiments in parallel, without interfering with each other.
You can create discrete queues that use distinct server groups that address different technical and business requirements. Then, when submitting a new job, you can target those specific server types.
Once you have created a queue, it is enabled by default. Commands are available for disabling queues, enabling them, and renaming them.
ml run xp to run a resource group in a queue.
You cannot assign jobs to queues that have been disabled.
Let's assume Team One requires strong GPUs for their jobs, while Team Two can suffice with weaker CPUs.
Perform the following steps:
Create the queues:
ml resources queue --create TeamOne
ml resources queue --create TeamTwo
Assign pre-existing resource groups to the new queues:
ml resources group "Strong GPUs" --set queue TeamOne # "Strong GPUs" is displayed in the dashboard for this resource group
ml resources group "Weak CPUs" --set queue TeamTwo # "Weak CPUs" is displayed for this resource group
Now, run jobs using
ml run xp, specifying in which queues to run them:
ml run xp --queue TeamOne
ml run xp --queue TeamTwo
For more information, see the following commands in the CLI reference: