Creating a Command Line Submission

Follow

The Conductor client tools provide three methods for submitting jobs.

  • Python Submit class
  • Conductor Submit Command
  • Application Plug-in Submitters (Maya/Nuke pyside ui, etc)

Python Submit class

Jobs can be submitted in pure python via the Submit class

import logging
from conductor.lib import conductor_submit, package_utils, api_client

# Set up logging (optional)
conductor_submit.set_logging(logging.DEBUG)

# create a dictionary of arguments for the job
args = {}

# specify the project that the Job will be associated with
args["project"] = "default"

# Specify how many cores each instances will have  
args["cores"] = 2

# Specify the what type of instance to use
args["machine_type"] = "standard"

# specify all of the file dependences that the Job requires (accepts files and directories)
args["upload_paths"] = ["/tmp/batman_0008.vrscene", # vrscene file for frame 8

                       "/tmp/batman_0024.vrscene", # vrscene file for frame 24

                       "/tmp/custom_vray"] # any custom vray scripts/shaders, etc

# Indicate that uploading should occur immediately (don't use daemon)
args["local_upload"] =  True

# Give the job a title
args["job_title"] = "Test Job: Vray Standalone Test"

# Indicate which software packages should be made available for the job
args["software_package_ids"] = ["7208372e306614d514944938c395e0e7"] #  vray-for-maya, etc

# Create a custom environment that will provide access to your own custom/studio plugins
custom_environment = {"VRAY_PLUGINS_x64": "/tmp/custom_vray/plugin",
                     "LD_LIBRARY_PATH": "/tmp/custom_vray/bin"}

# Request the packages that you want to use from conductor.
packages = [package for package in api_client.request_software_packages() if package["package_id"] in args["software_package_ids"]]

# merge the packages' environments with the custom environment
args["environment"] = package_utils.merge_package_environments(packages, base_env=custom_environment)

# Define the task data for the job (the command to execute, etc)
args["tasks_data"] = [
                       # task definition 1
                       {"command":"vray -display=0 -sceneFile=/tmp/batman_0008.vrscene -imgFile=/tmp/output/output.exr",
                        "frames": "8"},
                        
                       # task definition 2
                       {"command":"vray -display=0 -sceneFile=/tmp/batman_0024.vrscene -imgFile=/tmp/output/output.exr",
                        "frames": "24"}
                     ]


# Define the directory where the rendered images will be found when the tasks complete
args["output_path"] = "/tmp/output"


# Instantiate a Submit object with the job args
submission = conductor_submit.Submit(args)

# Run the submission
response, response_code = submission.main()

# Inspect the response code
print response_code
# 201

# Inspect the response
print response
# {u'body': u'job submitted.',
#  u'jobid': u'00695',
#  u'status': u'success',
#  u'uri': u'/jobs/00695'}




Name

Type

Default Value

Required

chunk_size

int

1

no

Description


The number of frames to render per Task (command execution)

Example


15



Name

Type

Default Value

Required

cores

int

8

 

Description


Number of cpu cores that the instance will have (2 minimum for highmem and highcpu instance types)

Example


64




Name

Type

Default Value

Required

enforced_md5s

dict of str

{}

no

Description


Only relevant when using uploader daemon.  Dictates md5 hash values that specific files must adhere to upon uploading via the uploader daemon. This is a mechanism to ensure that a file has not been modified since the time that the user submitted the job and the time that the uploader daemon has begun uploading it.

Example


{"/tmp/my_maya_file.ma": "GDVem8GYRUuh4douZwIwFg=="}




Name

Type

Default Value

Required

environment

dict of str

{}

no

Description


Specifies the environment variables to be set when executing the job on Conductor

Example


{"PYTHONPATH": "/tmp/site-packages:/tmp/pipeline/site-packages"}




Name

Type

Default Value

Required

job_title

str

"Untitled"

no

Description


The title to be displayed for the job in the webui

Example


"My maya job"




Name

Type

Default Value

Required

local_upload

bool

True

no

Description


Dictates whether the submission will upload the job's upload paths immediately (True) rather than queue the uploading process for an uploader daemon to handle (False)

Example


False




Name

Type

Default Value

Required

location

str

None

no

Description


A string that indicates which location this job should be associated with. This option is relevant for customers who submits jobs from different locations (e.g. differing geographic locations, or simply from machines that have differing file system views (mounts, lettered drives, etc). Typically each location should have its own conductor downloader/uploader daemon running. The location feature allows each daemon to target only files/jobs that match its corresponding location.

Example


"mars"




Name

Type

Default Value

Required

machine_type

str

"standard"

no

Description


The instance type to use for the job.  This works together with the cores argument.  options: standard, highcpu, highmem

Example


"highcpu"




Name

Type

Default Value

Required

metadata

dict

None

no

Description


key/value pairs to "tag" the job and it's subsequent data with

Example


{"shot": "Ez500", "asset": "apple"}




Name

Type

Default Value

Required

notify

list of str

None

 

Description


Which emails addresse(s) to notify when job is complete

Example


["fred@flintstones.com", "wilma@flintstones.com"]




Name

Type

Default Value

Required

output_path

str

 

yes

Description


the directory path to search (recursively) for output files from the render (exrs, etc). This is also the directory that the output files will be downloaded back to on the client.

Example


"/tmp/render_output"




Name

Type

Default Value

Required

priority

int

5

no

Description


The priority to assign to the job (higher numbers indicate higher urgency)

Example


800




Name

Type

Default Value

Required

project

str

 

yes

Description


The name of the project to associate the job with

Example


"spiderman"




Name

Type

Default Value

Required

scout_frames

list of int

None

no

Description


The tasks for these frames will be run while the rest of the tasks will be put into a holding state.  Users frame range notation

Example


"1-100x5"




Name

Type

Default Value

Required

slack_notify

list of str

None

no

Description


Slack recipients to notify when job is complete. Note that a slack webhook must first be associated with the customer account.

Example


["@fred", "@wilma"]




Name

Type

Default Value

Required

software_package_ids

list of str

 

yes

Description


The ids for the software packages to use for the submitted job, e.g. maya or nuke. Packages are identified by their ID number, which can be queried from Conductor's rest api

Example


["ae8026b7d9cfe524a77a3ebe94b9130a",  "7208372e306614d514944938c395e0e7"]




Name

Type

Default Value

Required

tasks_data

list of dict

None

no

Description


The definition of each task for the job. Each task dictionary should contain the following keys:
command: The command that the task will execute
frames (optional, informational only.): The frames that this task represents.

Example


[
{"command":"vray -display=0 -sceneFile=/tmp/batman_0008.vrscene",
 "frames": "8"},
 
{"command":"vray -display=0 -sceneFile=/tmp/batman_0024.vrscene",
 "frames": "24"}
]




Name

Type

Default Value

Required

upload_paths

list of str

None

yes

Description


A list of paths that will be uploaded and included as part of the job.  A path can be a path to a file or a directory. Directories wil have all of their contents uploaded

Example


["/tmp/sourceimages", "/tmp/my_maya_file.ma"]