What is a Preemptible Instance?
When a Cloud Computing Provider calculates anticipated consumption volume, they set aside spare compute capacity to make sure they can always meet demand. These machines may sit idle as demand fluctuates and are offered by the Cloud Computing Providers at a significant savings as Preemptible Machines or Preemptible Instances. Equally capable in every way to the on-demand versions, the preemptible instances may be taken away as volume changes. Preemptible machines will also be taken out of the job after 24 hours, no matter the percentage of task completion.
Recommended uses for Preemptible Machines
Preemptible machines represent a tremendous cost savings and are perfectly suited for jobs with lower impact to project delivery. Jobs with tasks that will take less than 3 or 4 hours, scout jobs or perhaps "look dev" parts of the workflow.
Understanding the limitations of Preemptible Machines
Relying on preemptible machines is not recommended for processing jobs where restarting and completing lost tasks will cause missed deadlines. At times, 100% of the preemptible machines assigned to a job will complete their tasks. At other times, it may be a small fraction of completion. It all depends on overall compute volume of the provider. Nights and weekends represent the lowest demand times for providers and the lowest risk of preemption, but they also represents a commitment to monitor and restart preempted tasks during those times.
Jobs with preempted tasks will be billed for compute time calculated up until the preemption.
Submitting to Preemptible Machines
As stated above, Preemptible Machines are identical in every way to the Compute instances that you can reserve. In the Client Tools submitter, select the type of machine you need to run your job as you would for an "On-Demand" submission. Select the Preemptible checkbox to simply utilize preemptible instances of the same machine.
How do I know if my job was preempted?
In the event that your job had tasks preempted, they will show up in the Conductor Jobs page as preempted with a purple indicator. As a reminder, compute time used up until the moment of preemption will be charged to the account even though the task is lost.
To resubmit your job, in the jobs page, simply right click over the job and Retry.
This will resubmit the preempted tasks to the exact same configuration to the original submission. (machines, preempted, license, etc.)
If the job needs to be resubmitted with new parameters, including moving them to on-demand machines, the job will need to submitted from the client tool user interface with the preemptible option unchecked.
The Conductor team is working on solving many of the challenges associated with using preemptible machines like automatic restarting preempted tasks. Stay tuned for next steps. In the meantime, enjoy the savings Preemptible Machines offer.