Skip to content

Graceful Shutdown in Generic Frameworks

This topic shows you how to gracefully shutdown an experiment. The topic builds on the script that resulted from steps in Integrating MissingLink with a Generic Network (With Steps).

The following steps are covered:

  • Create a stopped callback function.
  • Set the stopped callback function to MissingLink's callback.
  • Stop the experiment from MissingLink's dashboard.

Preparation

Go through Generic Inegration with Network (With Steps).

Note

Ensure that you can successfully run the full code sample training script that resulted from integration with the MissingLink SDK. In the steps that follow below, the script is further developed to include graceful shutdown.

Write code

  1. Create a stopped callback function.

    Right above the call to experiment, define your stopped callback.

    def start_new_experiment():
        # Write code here that starts a new experiment
        pass
    
    def log_experiment_to_internal_log():
        # Write code here that logs important information
        # to your internal logs
        pass
    
    def stopped_callback():
        start_new_experiment()
        log_experiment_to_internal_log()
    
    with missinglink_project.experiment(
        model,
        metrics={'loss': loss},
        display_name='MNIST multilayer perception',
        description='Two fully connected hidden layers') as experiment:
    
  2. Set the stopped callback to be called.

    Modify the call to experiment and add the stopped_callback parameter:

    # with missinglink_project.experiment(
    #     model,
    #     metrics={'loss': loss},
    #     display_name='MNIST multilayer perception',
    #     description='Two fully connected hidden layers') as experiment:
    
    with missinglink_project.experiment(
        model,
        metrics={'loss': loss},
        display_name='MNIST multilayer perception',
        description='Two fully connected hidden layers',
        stopped_callback=stopped_callback) as experiment:
    

    You can find the full, updated script here .

    You have added the graceful shutdown callback to your experiment. Run the script and stop it using the dashboard.

  3. Stop the experiment through the dashboard while it is still running.

    Go to the MissingLink dashboard, navigate to the experiment, and click Stop.

    You are prompted to confirm the action. Click Stop as shown.

    You should have successfully executed a graceful shutdown of your experiment. Notice that instead of having an exception raised in the environment where you ran the experiment, Experiment stopped from the web appears.

View the addition in the dashboard

You can see the experiment has been marked Stopped in the dashboard.