Creating a Command Line Submission

Follow

The Conductor client tools provide three methods for submitting jobs.

  • Python Submit class
  • Conductor Submit Command
  • Application Plug-in Submitters (Maya/Nuke pyside ui, etc)

Python Submit class

Jobs can be submitted in pure python via the Submit class

Submit Arguments

import logging
from conductor.lib import conductor_submit, package_utils, api_client

# Set up logging (optional)
conductor_submit.set_logging(logging.DEBUG)

# create a dictionary of arguments for the job
args = {}

# specify the project that the Job will be associated with
args["project"] = "default"

# Specify how many cores each instances will have  
args["cores"] = 2

# Specify the what type of instance to use
args["machine_type"] = "standard"

# specify all of the file dependences that the Job requires (accepts files and directories)
args["upload_paths"] = ["/tmp/batman_0008.vrscene",
# vrscene file for frame 8

                 "/tmp/batman_0024.vrscene", # vrscene file for frame 24

                 "/tmp/custom_vray"] # any custom vray scripts/shaders, etc

# Indicate that uploading should occur immediately (don't use daemon)

args["local_upload"] =  True

# Give the job a title
args["job_title"] = "Test Job: Vray Standalone Test"

# Indicate which software packages should be made available for the job
args["software_package_ids"] = ["7208372e306614d514944938c395e0e7"] #  vray-for-maya, etc

# Create a custom environment that will provide access to your own custom/studio plugins

custom_environment = {"VRAY_PLUGINS_x64": "/tmp/custom_vray/plugin",
                     "LD_LIBRARY_PATH": "/tmp/custom_vray/bin"}

# Request the packages that you want to use from conductor.
packages = [package for package in api_client.request_software_packages() if package["package_id"] in args["software_package_ids"]]

# merge the packages' environments with the custom environment
args["environment"] = package_utils.merge_package_environments(packages, base_env=custom_environment)
# Define the directory where the rendered images will be found when the tasks complete
args["output_path"] = "/tmp/output"
# Instantiate a Submit object with the job args
submission = conductor_submit.Submit(args)
  # Run the submission
response, response_code = submission.main()
 
# Inspect the response code
print response_code
# 201
# Inspect the response

print response
# {u'body': u'job submitted.',
#  u'jobid': u'00695',
#  u'status': u'success',
#  u'uri': u'/jobs/00695'}

 

Name Type Default Value Required
chunk_size int 1 no
Description:
The number of frames to render per Task (command execution)
Example:
15

 

Name Type Default Value Required
cores int 8  
Description:
Number of cpu cores that the instance will have (2 minimum for highmem and highcpu instance types)
Example:
64

 

Name Type Default Value Required
enforced_md5s dict of str {} no
Description:
Only relevant when using uploader daemon.  Dictates md5 hash values that specific files must adhere to upon uploading via the uploader daemon. This is a mechanism to ensure that a file has not been modified since the time that the user submitted the job and the time that the uploader daemon has begun uploading it.
Example:
{"/tmp/my_maya_file.ma": "GDVem8GYRUuh4douZwIwFg=="}

 

Name Type Default Value Required
environment dict of str {} no
Description:
Specifies the environment variables to be set when executing the job on Conductor
Example:
{"PYTHONPATH": "/tmp/site-packages:/tmp/pipeline/site-packages"}

 

Name Type Default Value Required
job_title str "Untitled" no
Description:
The title to be displayed for the job in the webui
Example:
"My maya job"

 

Name Type Default Value Required
local_upload bool True no
Description:
Dictates whether the submission will upload the job's upload paths immediately (True) rather than queue the uploading process for an uploader daemon to handle (False)
Example:
False

 

Name Type Default Value Required
location str None no
Description:
A string that indicates which location this job should be associated with. This option is relevant for customers who submits jobs from different locations (e.g. differing geographic locations, or simply from machines that have differing file system views (mounts, lettered drives, etc). Typically each location should have its own conductor downloader/uploader daemon running. The location feature allows each daemon to target only files/jobs that match its corresponding location.
Example:
"mars"

 

Name Type Default Value Required
machine_type str "standard" no
Description:
The instance type to use for the job.  This works together with the cores argument. options: standard, highcpu, highmem
Example:
"highcpu"

 

Name Type Default Value Required
metadata dict None no
Description:
key/value pairs to "tag" the job and it's subsequent data with
Example:
{"shot": "Ez500", "asset": "apple"}

 

Name Type Default Value Required
notify list of str None  
Description:
Which emails addresse(s) to notify when job is complete
Example:
["fred@flintstones.com", "wilma@flintstones.com"]

 

Name Type Default Value Required
output_path str   yes
Description:
the directory path to search (recursively) for output files from the render (exrs, etc). This is also the directory that the output files will be downloaded back to on the client.
Example:
"/tmp/render_output"

 

Name Type Default Value Required
priority int 5 no
Description:
The priority to assign to the job (higher numbers indicate higher urgency)
Example:
800

 

Name Type Default Value Required
project str   yes
Description:
The name of the project to associate the job
Example:
"spiderman"

 

Name Type Default Value Required
scout_frames list of int None no
Description:
The tasks for these frames will be run while the rest of the tasks will be put into a holding state.  Users frame range notation
Example:
"1-100x5"

 

Name Type Default Value Required
slack_notify list of str None no
Description:
Slack recipients to notify when job is complete. Note that a slack webhook must first be associated with the customer account.
Example:
["@fred", "@wilma"]

 

Name Type Default Value Required
software_package_ids list of str   yes
Description:
The ids for the software packages to use for the submitted job, e.g. maya or nuke. Packages are identified by their ID number, which can be queried from Conductor's rest api
Example:
["ae8026b7d9cfe524a77a3ebe94b9130a",  "7208372e306614d514944938c395e0e7"]

 Find Software packages here.

 

Name Type Default Value Required
tasks_data list of dict> None no
Description:
The definition of each task for the job. Each task dictionary should contain the following keys:
  • command: The command that the task will execute
  • frames (optional, informational only.): The frames that this task represents.
Example:
{"command":"vray -display=0 -sceneFile=/tmp/batman_0008.vrscene",
 "frames": "8"}, 
{"command":"vray -display=0 -sceneFile=/tmp/batman_0024.vrscene",
 "frames": "24"}
]

 

Name Type Default Value Required
upload_paths list of str None yes
Description:
A list of paths that will be uploaded and included as part of the job.  A path can be a path to a file or a directory. Directories wil have all of their contents uploaded
Example:
["/tmp/sourceimages", "/tmp/my_maya_file.ma"]