What is a Preemptible Instance?
When a Cloud Computing Provider calculates anticipated consumption volume, they set aside spare compute capacity to make sure they can always meet demand. These machines may sit idle as demand fluctuates and are offered by the Cloud Computing Providers at a significant savings as Preemptible Machines or Preemptible Instances. Equally capable in every way to the on-demand versions, the preemptible instances may be taken away as volume changes. Preemptible machines will also be taken out of the job after 24 hours, no matter the percentage of task completion.
Recommended uses for Preemptible Machines
Preemptible machines represent a tremendous cost savings and are perfectly suited for jobs with lower impact to project delivery. Jobs with tasks that will take less than 3 or 4 hours, scout jobs or perhaps "look dev" parts of the workflow.
Understanding the limitations of Preemptible Machines
Relying on preemptible machines is not recommended for processing jobs where restarting and completing lost tasks will cause missed deadlines. At times, 100% of the preemptible machines assigned to a job will complete their tasks. At other times, it may be a small fraction of completion. It all depends on overall compute volume of the provider. Nights and weekends represent the lowest demand times for providers and the lowest risk of preemption.
Jobs with preempted tasks will be billed for compute time calculated up until the preemption.
Submitting to Preemptible Machines
As stated above, Preemptible Machines are identical in every way to the Compute instances that you can reserve. In the Client Tools submitter, select the type of machine you need to run your job as you would for an "On-Demand" submission. Select the Preemptible checkbox to simply utilize preemptible instances of the same machine. You can also select how many times to automatically retry a task if preempted - this creates a powerful workflow in allowing for cheaper rendering on average by automating the retry process.
Preemptible options in the Conductor Submitter UI
How do I know if my job was preempted?
In the event that your job had tasks preempted, they will show up in the Conductor Jobs page as preempted with a purple indicator. As a reminder, compute time used up until the moment of preemption will be charged to the account even though the task is lost. Fortunately, with the low cost that preemptibles have - retrying the task multiple times can still provide savings compared to standard instances.
If you opt to not do automatic retries and instead prefer to manually retry tasks - simply retry to given job or task by right-clicking over the preempted entry in the dashboard.
This will resubmit the preempted tasks to the exact same configuration to the original submission. (machines, preempted, license, etc.)
If the job needs to be resubmitted with new parameters, including moving them to on-demand machines, the job will need to submitted from the client tool user interface with the preemptible option unchecked.