Creating A Command Line Submission: Software Packages

Follow

Overview

When you submit a job, you’ll need to specify which software packages that the job requires. In order to indicate which software packages a job should use, specific software package IDs must be specified when submitting that job. A software packages is a specific version (installation) of a specific software product. For example Nuke 9.0v7 or Maya 2016 Extension 1 SP2 or VRay for Katana2.1 3.05.01.26797. Each package has its own unique id, such as 5787e38e6ffcead5c67dbfd7cec921d6. In addition to package ids, a corresponding environment must also be supplied during job submission. Both of these pieces of data can be queried from Conductor.

Available Software Packages

Conductor's software packages can be queried by using the conductor client python library:

>>> from conductor.lib import api_client
>>> packages = api_client.request_software_packages()
270 # 270 unique packages available
>>> katana_packages = [package for package in packages if package["product"] == "katana"]
>>> len(katana_packages)

Package Anatomy

Below is what the structure of a package may look like:

{u'build_id': u'3bc076c1-dd23-4188-91bb-a84ff015f84e',
u'package_id': u'fdc5e6628b276b92c661068b93d24da2',
u'product': u'v-ray-katana',
u'vendor': u'chaosgroup'
u'description': u'',
u'major_version': u'3',
u'minor_version': u'05',
u'release_version': u'01',
u'build_version': u'26771',
u'environment': [{u'name': u'KATANA_RESOURCES',
                  u'value': u'/opt/chaosgroup/v-ray-katana/3/v-ray-katana-katana1_5-3.05.01-26771.',
                  u'merge_policy': u'append'},
                 {u'name': u'VRAY_FOR_KATANA_PLUGINS_x64',
                  u'value': u'/opt/chaosgroup/v-ray-katana/3/v-ray-katana-katana1_5-3.05.01-26771./vrayplugins',
                  u'merge_policy': u'append'},
                 {u'name': u'VRAY_AUTH_CLIENT_FILE_PATH',
                  u'value': u'/etc',
                  u'merge_policy': u'exclusive'}],
u'package': u'vray_adv_30501_katana_1_5_linux_x64_26771.rar',
u'plugin_host_product': u'katana',
u'plugin_host_version': u'1_5',
u'plugin_hosts': [],
u'plugins': [],
u'relative_path': u'chaosgroup/v-ray-katana/3/v-ray-katana-katana1_5-3.05.01-26771.',
u'updated_at': u'Fri, 01 Jul 2016 02:50:42 GMT'}

Of particular importance is the package_id and the environment. Both of these will be necessary when submitting a job.

Package Environment

The package's environment describes what environment variables are required in order for the software to successfully execute. You can see that the supplied environment is comprised of a list of dictionaries, where each dictionary represents a distinct environment variable:

[
 {u'name': u'KATANA_RESOURCES',
  u'value': u'/opt/chaosgroup/v-ray-katana/3/v-ray-katana-katana1_5-3.05.01-26771.',
  u'merge_policy': u'append'},

 {u'name': u'VRAY_FOR_KATANA_PLUGINS_x64',
  u'value': u'/opt/chaosgroup/v-ray-katana/3/v-ray-katana-katana1_5-3.05.01-26771./vrayplugins',
  u'merge_policy': u'append'},

 {u'name': u'VRAY_AUTH_CLIENT_FILE_PATH',
  u'value': u'/etc',
  u'merge_policy': u'exclusive'}
]

The name and value for an environment variable are self explanatory. However, the merge_policy indicates the action to take when combining this environment with another package's environment (or a pipeline's environment). The two merge options are:
append: Indicates that the value should be appended to any previously existing value.
exclusive: Indicates that this package should be the only package that sets this environment variable.

Note that this environment information in each package is not automatically set when a job/task starts. It is the responsibility of the client to query for this information and ultimately construct and specify the full environment that the job will use.

Using Multiple Packages for a Job

Oftentimes multiple packages are required for a job, e.g. "I want to use Maya 2016 SP5 for the primary software application, and Vray for Maya 3.30.01-0 for the renderer plugin". This means that both package ids must be specified, as well as an environment that supports both software packages. The conductor client library provides tools to merge package environments into a format that is suitable for job submission.

Workflow example

The following workflow demonstrates how to:
Query Conductor for available packages
Construct an appropriate environment dictionary for the job that supports those two packages
Find the two packages that I'm interested in using.
Inject my own custom environment to support my custom pipeline scripts, etc
Construct and submit my job with the proper environment and package id arguments.

1. Query Conductor for packages

>>> from conductor.lib import api_client
>>> packages = api_client.request_software_packages()

2. Find the two packages that I'm interested in rendering with. In this case, cherry pick my desired Maya package and Vray package

>>> maya_package = packages[7]
>>> vray_maya_package = packages[210]
>>> job_packages = [maya_package, vray_maya_package]
>>> from pprint import pprint


# print out my packages
>>> pprint(maya_package)
{u'build_id': u'3bc076c1-dd23-4188-91bb-a84ff015f84e',
 u'build_version': u'',
 u'description': u'',
 u'environment': [{u'merge_policy': u'exclusive',
                   u'name': u'MAYA_LOCATION',
                   u'value': u'/opt/autodesk/maya/2016/maya2016.SP5'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'MAYA_DISABLE_CIP',
                   u'value': u'1'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'ADSKFLEX_LICENSE_FILE',
                   u'value': u'@conductor_adlm'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'AUTODESK_ADLM_THINCLIENT_ENV',
                   u'value': u'/tmp/AdlmThinClientCustomEnv.xml'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'MAYA_LICENSE_METHOD',
                   u'value': u'network'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'MAYA_LICENSE',
                   u'value': u'unlimited'},
                  {u'merge_policy': u'append',
                   u'name': u'LD_LIBRARY_PATH',
                   u'value':
 u'/opt/autodesk/maya/2016/maya2016.SP5/lib:/opt/autodesk/maya/2016/maya2016.SP5/plug-ins/xgen/lib:/opt/autodesk/maya/2016/maya2016.SP5/plug-ins/bifrost/lib'}
                 {u'merge_policy': u'append',
                  u'name': u'PATH',
                  u'value': u'/opt/autodesk/maya/2016/maya2016.SP5/bin'}],
 u'major_version': u'2016',
 u'minor_version': u'SP5',
 u'package': u'Autodesk_Maya_2016_SP5_EN_Linux_64bit.tgz',
 u'package_id': u'7c53a84e8001dc40aca7a844ca60ba0b',
 u'plugin_host_product': u'',
 u'plugin_host_version': u'',
 u'plugin_hosts': [],
 u'plugins': [u'0074771ed6622e6975a4770fa0cde004',
              u'112d90408d3871da851f231988ad8145',
              u'117bdb1282d82e741f8bbf9b50a0c547',
              u'304bd6c4b1714b20a2edab81f449b738',
              u'4e96e7df4291d285f30ad73d3871536e',
              u'61c716542e4a8dd4a834390e3b41d4b1',
              u'781b03c5609da53bcf254c5760ef9a25',
              u'79ff37973ef0c543a9dadd9bda8ff774',
              u'82659ed9b17eb70fa5d01b761a0024d6',
              u'88872a43918591b0e215c5a3390a5e71',
              u'92d614d9b3961b34fdeba02ca4345d27',
              u'97d3a45868d9748eee58e49de82849d8',
              u'a5e93d6fd07e290f4ab0660a921f8272',
              u'a6c81da2ab4b322bfb6b77011bd54573',
              u'a6ddf76a25809410b8c822d87da555b8',
              u'ab1f0722997e09c66954326464178f68',
              u'ad9a76c60354805f6d54fa07e20b657a',
              u'ae6b6564ca05e5f9f7c5df0ce72f47c2',
              u'b200eabaa755873677205d785c254d12',
              u'c94e13e3f8defffe2444d509d6e24fe1',
              u'd6ff0e572eb121f5b3c5327f88f3ddf4',
              u'debb2da484387978a7da8da1c728227a'],
  u'product': u'maya',
  u'relative_path': u'autodesk/maya/2016/maya2016.SP5',
  u'release_version': u'',
  u'updated_at': u'Fri, 01 Jul 2016 02:47:45 GMT',
  u'vendor': u'autodesk'}

>>> pprint(vray_maya_package)
{u'build_id': u'3bc076c1-dd23-4188-91bb-a84ff015f84e',
 u'build_version': u'0',
 u'description': u'',
 u'environment': [{u'merge_policy': u'exclusive',
                   u'name': u'VRAY_FOR_MAYA2016_MAIN_x64',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray'},
                  {u'merge_policy': u'append',
                   u'name': u'VRAY_FOR_MAYA2016_PLUGINS_x64',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/vrayplugins'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'VRAY_OSL_PATH_MAYA2016_x64',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/vrayplugins'},
                  {u'merge_policy': u'exclusive',
                   u'name': u'VRAY_AUTH_CLIENT_FILE_PATH',
                   u'value': u'/etc'},
                  {u'merge_policy': u'append',
                   u'name': u'MAYA_PLUG_IN_PATH',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/plug-ins'},
                  {u'merge_policy': u'append',
                   u'name': u'MAYA_RENDER_DESC_PATH',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./bin/rendererDesc'},
                  {u'merge_policy': u'append',
                   u'name': u'MAYA_SCRIPT_PATH',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/scripts'},
                  {u'merge_policy': u'append',
                   u'name': u'PYTHONPATH',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/scripts'},
                  {u'merge_policy': u'append',
                   u'name': u'XBMLANGPATH',
                   u'value': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/icons/%B'},
                  {u'merge_policy': u'append',
                   u'name': u'LD_LIBRARY_PATH',
                   u'value':
 u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_root/lib:/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_root/bin'}],
 u'major_version': u'3',
 u'minor_version': u'30',
 u'package': u'vray_adv_33001_maya2016_linux_x64.zip',
 u'package_id': u'781b03c5609da53bcf254c5760ef9a25',
 u'plugin_host_product': u'maya',
 u'plugin_host_version': u'2016',
 u'plugin_hosts': [u'5787e38e6ffcead5c67dbfd7cec921d6'
                   u'af7ecc0d56c84b2ec2500329af25ca40',
                   u'7c53a84e8001dc40aca7a844ca60ba0b',
                   u'8515d5c5a971e368db7780e54433ff78',
                   u'5d6dd77564e9ed786fe3a38381fc26e7',
                   u'1ad28804f9ee499213b15135a0de276e'],
 u'plugins': [],
 u'product': u'v-ray-maya',
 u'relative_path': u'chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0.',
 u'release_version': u'01',
 u'updated_at': u'Fri, 01 Jul 2016 03:00:00 GMT',
 u'vendor': u'chaosgroup'}

3. Merge the two packages' environments into one, while providing my own custom environment as a base environment to merge into.

>>> import os
>>> from conductor.lib import conductor_submit, package_utils
>>> my_custom_env = {"PYTHONPATH": "/home/scripts/python"}
>>> job_env = package_utils.merge_package_environments(job_packages, base_env=my_custom_env)

>>> pprint(job_env)
{u'ADSKFLEX_LICENSE_FILE': u'@conductor_adlm',
 u'AUTODESK_ADLM_THINCLIENT_ENV': u'/tmp/AdlmThinClientCustomEnv.xml',
 u'LD_LIBRARY_PATH':  u'/opt/autodesk/maya/2016/maya2016.SP5/lib:/opt/autodesk/maya/2016/maya2016.SP5/plug-ins/xgen/lib:/opt/autodesk/maya/2016/maya2016.SP5/plug-ins/bifrost/lib:/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_root/lib:/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_root/bin',
 u'MAYA_DISABLE_CIP': u'1',
 u'MAYA_LICENSE': u'unlimited',
 u'MAYA_LICENSE_METHOD': u'network',
 u'MAYA_LOCATION': u'/opt/autodesk/maya/2016/maya2016.SP5',
 u'MAYA_PLUG_IN_PATH': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/plug-ins',
 u'MAYA_RENDER_DESC_PATH': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./bin/rendererDesc',
 u'MAYA_SCRIPT_PATH': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/scripts',
 u'PATH': u'/opt/autodesk/maya/2016/maya2016.SP5/bin',
'PYTHONPATH': u'/home/scripts/python:/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/scripts',
 u'VRAY_AUTH_CLIENT_FILE_PATH': u'/etc',
 u'VRAY_FOR_MAYA2016_MAIN_x64': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray',
 u'VRAY_FOR_MAYA2016_PLUGINS_x64': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/vrayplugins',
 u'VRAY_OSL_PATH_MAYA2016_x64': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/vrayplugins',
 u'XBMLANGPATH': u'/opt/chaosgroup/v-ray-maya/3/v-ray-maya-maya2016-3.30.01-0./maya_vray/icons/%B'}

4. Construct and submit my job with the proper environment and package id arguments.

# create a list of the package ids I want to use for the job
>>> package_ids = [package["package_id"] for package in job_packages]
>>> print package_ids
[u'7c53a84e8001dc40aca7a844ca60ba0b', u'781b03c5609da53bcf254c5760ef9a25']

# Use the package_ids and environment when constructing the arguments for the Job
>>> job_args = {"environment": job_env, "software_package_ids": package_ids}
>>> job = conductor_submit.Submit(job_args)
>>> job.submit()