Skip to content

Graceful Shutdown for PyCaffe

This topic shows you how to gracefully shutdown an experiment. The topic builds on Getting Started for PyCaffe with Solve.

The following steps are covered:

  • Create a stopped callback function.
  • Set the stopped callback function to MissingLink's callback.
  • Stop the experiment from MissingLink's web dashboard.

Preparation

Go through Getting Started for PyCaffe with Solve.

Note

Ensure that you can successfully run the mnist.py training script that resulted from integration with the MissingLink SDK. In the steps that follow below, the script is further developed to include graceful shutdown.

Write code

  1. Create a stopped callback function.

    Right above declaring MissingLink's callback, define your stopped callback:

    def start_new_experiment():
        # Write code here that starts a new experiment
        pass
    
    def log_experiment_to_internal_log():
        # Write code here that logs important information
        # to your internal logs
        pass
    
    def stopped_callback():
        start_new_experiment()
        log_experiment_to_internal_log()
    
    missinglink_callback = missinglink.PyCaffeCallback(
        owner_id=OWNER_ID, project_token=PROJECT_TOKEN)
    
  2. Set the stopped callback to be called.

    In the base script, modify the declaration of MissingLink's callback:

    missinglink_callback = missinglink.PyCaffeCallback(
        owner_id=OWNER_ID, project_token=PROJECT_TOKEN,
        stopped_callback=stopped_callback)
    

    You can find the full updated script here.

    You are done adding the graceful shutdown callback to your experiment. Now run the script and stop it using the web dashboard.

  3. Stop the experiment through the dashboard while it is still running.

    Go to MissingLink.ai's web dashboard, navigate to the experiment and click Stop.

    You will be prompted with a confirmation and you should click on the stop confirmation button as shown.

    You should have successfully executed a graceful shutdown of your experiment. You will notice that instead of having an exception raised in the environment where you ran the experiment, you see Experiment stopped from the web printed.

View the addition in the dashboard

You can see the experiment has been marked as stopped on the dashboard.