pdg package

Submodules

pdg.callback module

Helper clases for implementing logic in Python and interacting with it in C++. Should not be instantiated directly or subclassed from.

class pdg.callback.PythonCallback(cpp_object)

Base class for all object that are created in Python, but owned and called from C++

cppObject(self) → object

Returns the wrapped C++ object. Used by the C++ binding layer, and should not be access directly from Python

pdg.dependency module

Built in dependency types

class pdg.dependency.PyDependency(dependency, key)

Bases: pdg.callback.PythonCallback

Base class for external dependencies implemented in Python

isResolved(self, _pdg.GraphContext, _pdg.PyNode) → boolean

[virtual] Should return True if the dependency was satisfied

All PyDependency subclasses must implement this method

onResolve(self, _pdg.GraphContext, _pdg.PyNode) → boolean

[virtual] Called when the dependency should resolve. Returns True on success and False on failure.

All PyDependency subclasses must implement this method

pdg.depthfirstcook module

class pdg.depthfirstcook.DepthFirstChain(cook_set, name)

Handles one dfc chain

dfc_checkIsRunnable(work_item)

Check if this workitem can be run as far as DFC is concerned, do not change run state.

dfc_onScheduled(work_item)

Workitem is actually being run. Update run state as necessary.

dfc_onWorkItemCompleted(work_item, success)

Called when any workitem completes.

dfc_tick()

Check if we have any pending completions we need to process. This may result in a change of depth

class pdg.depthfirstcook.DepthFirstCookMixin

Implementation of Depth-first cooking scheme. The state of each parallel chain is independent and handled by a DepthFirstChain object.

dfc_checkIsRunnable(work_item)

Check if this workitem can be run as far as DFC is concerned, do not change run state.

dfc_onScheduled(work_item)

Workitem is actually being run. Update run state as necessary.

dfc_onStartCook(cook_set)

Reset our state

dfc_onWorkItemCompleted(name, success)

Called when any workitem completes.

pdg.mapper module

Built in mapper types

class pdg.mapper.PyMapper(node)

Bases: pdg.node.PyNode

Base mapper implementation

pdg.mapper.saveToPython(node, classname='', description='', file_path='', tab_submenu='Custom')

pdg.node module

Built in top-level node types

class pdg.node.PyNode(node)

Bases: pdg.callback.PythonCallback

Base class for nodes implemented in Python.

addDependency(self, str, str) → _pdg.Dependency

Adds a dependency to the node with a given type and name

addFileDependency(self, str) → _pdg.Dependency

Adds a file dependency to the node with a given type and name. Shortcut for addDependency with a type of pdg.dependencyType.File.

addParameter(self, str, str, strm _pdg.dataType, int) → _pdg.Port

Adds a custom parameter to the node with a given name, label, tag, type, and size. The size defaults to 1.

connect(self, str, pdg.PyNode, str) -> (_pdg.Port, _pdg.Port)

Connects two ports on two nodes

disconnect(self, str, pdg.PyNode, str) -> (_pdg.Port, _pdg.Port)

Disconnects two ports on two nodes

isItemFromPort(self, _pdg.WorkItem, str) → boolean

Returns true if the item was passed to the node over the specified port

removeDependency(self, str) → self

Removes an external dependency from the node, using the dependency key

setExpression(self, str, str) → self

Sets the expression of parameter

setExpressions(self, dict, **kwargs) → self

Batch sets node parameter expressions using a dictionary and key word arguments

setParameter(self, str, value, int) → self

Sets a parameter on the node to a value. Must a int, float or string

setParameters(self, dict, **kwargs) → self

Batch sets parameters on the node from a dictionary or keyword args dictionary

setScheduler(self, str) → self

Sets the scheduler for the node

setValue(self, str, value, int) → self

Sets the value of a node parameter

setValues(self, dict, **kwargs) → self

Batch sets the values of parameters

pdg.parms module

pdg.parms.appendPathExtension(path, ext)
pdg.parms.getBaseOutputDir()
pdg.parms.pathHasExtension(path)
pdg.parms.removePathExtension(path)
pdg.parms.resolveOutputDirParm(path, scheduler)
pdg.parms.resolveOutputFilenameParm(path, scheduler)

pdg.partitioner module

Built in partitioner types

class pdg.partitioner.PyPartitioner(node)

Bases: pdg.node.PyNode

Base partitioner implementation

pdg.partitioner.saveToPython(node, classname='', description='', file_path='', tab_submenu='Custom')

pdg.processor module

Built in processor types

class pdg.processor.PyProcessor(node)

Bases: pdg.node.PyNode

Base processor implementation

pdg.processor.saveToPython(node, classname='', description='', file_path='', tab_submenu='Custom')

pdg.scheduler module

class pdg.scheduler.PyScheduler(scheduler, name)

Bases: pdg.callback.PythonCallback

Base class for schedulers implemented in Python

addParameter(self, str, str, strm _pdg.dataType, int) → _pdg.Port

Adds a custom parameter to the node with a given name, label, tag, type, and size. The size defaults to 1.

cleanTempDirectory()

[virtual] Deletes the temporary directory and all its contents.

endSharedServer(sharedserver_name)

[virtual] Returns True on success, else False. Must be implemented by PyScheduler subclasses.

sharedserver_name shared server name

End a shared server - should kill the long-running process

getLogURI(work_item)

[virtual] Returns the farm’s log URI for the given task. Should return a valid URI or empty string. Example: ‘file:///myfarm/tasklogs/jobid20.log’

work_item _pdg.WorkItem

getStatusURI(work_item)

[virtual] Returns the farm’s status URI for the given task. Should return a valid URI or empty string. Example: ‘http://myfarm/status/jobid20’

work_item _pdg.WorkItem

onLocalizePath(path)

[virtual] Returns the localized version of the given path.

path - a file path based on __PDG_SHARED_ROOT__

onSchedule(work_item)

[virtual] Called when the scheduler should schedule a work item. Returns True on success, else False.

Must be implemented by PyScheduler subclasses.

work_item _pdg.WorkItem

onScheduleStatic(dependencies, dependents, ready_items)

[virtual] Called when the scheduler should process a static dependency graph.

dependencies _pdg.WorkItem map of dependencies dependents _pdg.WorkItem map of dependents ready_items _pdg.WorkItem array of work items

onStart()

[virtual] Scheduler start callback. Returns TRUE if started.

Must be implemented by PyScheduler subclasses.

onStartCook(static, cook_set)

[virtual] Cook start callback. Starts a root job for the cook session. Returns TRUE if started.

static TRUE if static cook cook_set set of nodes to cook

Must be implemented by PyScheduler subclasses.

onStop()

[virtual] Scheduler stop callback. Returns TRUE if stopped.

Must be implemented by PyScheduler subclasses.

onStopCook(cancel)

[virtual] Cook stop callback. Called when cooking completes or is cancelled. Returns TRUE if stopped.

cancel TRUE if cook was cancelled

Must be implemented by PyScheduler subclasses.

scriptDir(local)

[virtual] Returns the temp directory for scripts of the Scheduler.

local True means return the path on the local machine, False
means return a network path.
setParameter(self, str, value, boolean) → self

Sets a parameter on the node to a value. Must a int, float or string

setParameters(self, dict, **kwargs) → self

Batch sets parameters on the node from a dictionary or keyword args dictionary

setValue(self, str, value, int) → self

Sets the value of a node parameter

setValues(self, dict, **kwargs) → self

Batch sets the values of parameters

submitAsJob(graph_file, node_path)

[virtual] Called when the scheduler should cook the entire TOP Network as a standalone job. Returns the status URI for the submitted job.

graph_file Path to a .hip file containing the TOP Network, relative to
$PDG_DIR.

node_path Op path to the TOP Network

tempDir(local)

[virtual] Returns the temp directory of the Scheduler.

local True means return the path on the local machine, False
means return a network path.
transferFile(file_path)

[virtual] Called when a file should be moved from the local machine to a remote location, e.g. a shared network drive. Returns True on success, else False.

workItemResultServerAddr()

[virtual] Returns the results server address. This is the TCP port that listens for status being returned by jobs.

It should be a string IP address. Eg: ‘192.168.1.123:83000’

Must be implemented by PyScheduler subclasses.

workingDir(local)

[virtual] Returns the working directory of the Scheduler. This is used to replace __PDG_SHARED_ROOT__ in paths.

local True means return the path on the local machine, False
means return a path resolvable from farm machines.

Must be implemented by PyScheduler subclasses.

pdg.scheduler.saveToPython(node, classname='', description='', file_path='', tab_submenu='Schedulers')

pdg.staticcook module

class pdg.staticcook.StaticCookMixin

Implementation of the static cooking logic

static_completed(work_item_name, index)

Called when a work item finished successfully

static_isWorkItemReady(work_item_name, index)
static_loadDependencies(dependencies, dependents, ready_items)

Called at the beginning of a static cook, in onScheduleStatic

static_onStartCook()

Reset our state

static_schedule(work_item)

Called when the schedule wishes to schedule an item from the static dependency graph

static_tick()

Called by the tick timer

pdg.utils module

class pdg.utils.TickTimer(*args, **kwargs)

Bases: threading._Timer

Utility class for a recurring timer.

run()
pdg.utils.expand_vars(command, job_env)

Expand the given command using env var substitution from the given environment dict. Supports $VAR, ${VAR}

pdg.utils.get_ip_address()

Get the IP address our primary adapter is bound to

pdg.utils.get_result_data_tag(file_path)
pdg.utils.get_result_viewer(file_path)
pdg.utils.natural_sort_key(s, _nsre=<_sre.SRE_Pattern object>)

Natural number-aware sort key for string lists

pdg.utils.parseEnvDefinition(job_env_str, force=False)

Parses a string in houdini .env format into a dict of string : string

Example: # comments MYKEY=My Val MYKEY2 = “path with spaces” # remove a key MYKEY3=

pdg.utils.readEnvParams(parmprefix, work_item, escapeforshell=True)

Parses a scheduler environment dict from a work item’s data. Assumes the parm names follow the standard prefix convention. Returns environement dict and list of empty names as: ( {Name:Value}, [Name] )

pdg.utils.recv_all_data(sock)

Recieve all data from the given socket, returns a bytearray object.

pdg.utils.setMultiParameter(node, multi_name, multi_member_defs, multi_member_values, tag='')

Set a ‘multiparm’ on a PDG node. Overwrites existing parameters.

Multiparms don’t exist in the PDG world, but this class recreates the logic of multiparms and does the right thing as if a multiparm from TOPs was converted to spare parameters in PDG.

Args:

node: The PyNode on which to add the parameters. multi_name: The name of the multi parm. multi_member_defs: A list of tuples of (name, data_type, vec_size) of

the member parms, without the index suffix.
multi_member_values: A parallel list of tuples of values for the names.
Each tuple t should have len(t) == len(multi_member_defs).

Returns: node

pdg.utils.setParmOnItem(item, port)

Copies the value of the given port to the given workitem.

setParmOnItem(self, _pdg.workItem, _pdg.Port) -> NoneType

Does type checking on value an sets the parameter as needed. The value can also be a tuple of a float, str or int, which will automatically expand to multiple setParam calls with different indices

pdg.workitem module

Built in Python work item types

class pdg.workitem.GenericDataProtoWorkItem(item)

Bases: pdg.workitem.ProtobufWorkItem

WorkItem specialized for serializing GenericData workitem data via a protobuf file.

fromProtobuf(proto)
static protobuf_typeclass()
toProtobuf(proto, item)
class pdg.workitem.ProtobufWorkItem(item, protobuf_type)

Bases: pdg.workitem.PyWorkItem

PyWorkItem subclass intended for use with work items that persist to files using the Google protocol buffer format.

clone(self, _pdg.WorkItem) → pdg.PyWorkItem

[virtual] Clones the work item and does a deep copy of the protobuf

deserialize(self, str) → boolean

[virtual] deserializes the str into the backing protobuf instance

fromProtobuf(self, protobuf obj) → boolean

Writes the data from the given protbuf into this object.

serialize(self, _pdg.PyWorkItem, _pdg.PyScheduler) → str

[virtual] serializes the backing protobuf instance to a string

serializeToFile(file_name, item, scheduler)

serializeTofile(self, str, _pdg.PyWorkItem, _pdg.PyScheduler) -> boolean

[virtual] serializes the backing protobuf to a file with a .proto extension.

toProtobuf(self, protobuf obj, _pdg.PyWorkItem) → boolean

Write the contents of this into the given protobuf.

class pdg.workitem.PyWorkItem(item)

Bases: pdg.callback.PythonCallback

Python wrapper class used to define custom work item types.

Custom items should subclass from pdg.PyWorkItem and provide implementations for all of the virtual methods

clone(self, _pdg.WorkItem) → pdg.PyWorkItem

[virtual] Returns a clone of the work item using the provided C++ object

deserialize(self, str) → str

[virtual] Deserializes the work item from a string, used by task graph serialization. Must be implemented by subclasses.

serialize(self, _pdg.PyWorkItem, _pdg.PyScheduler) → str

[virtual] Serialises the work item to a string. Must be implemented by subclasses.

serializeToFile(self, str, _pdg.PyWorkItem, _pdg.Scheduler) → boolean

Serializes the work item to the specified file path

Module contents

Python bindings for the PDG library.

class pdg.SearchType
ImmediateParent = 0
UpstreamStopAtPartitions = 2
UpstreamTraversePartitions = 1
pdg.collectDirectResultData(item, tag, localized=False)

Returns a list of matching result data, if the given item is a list it will return a combined list of all matching results. Does not do any task graph traversal. item item or list of items to search tag tag to search for localized True means localize the results, False to return as-is

pdg.defaultErrorHandler(event)
pdg.envVar(item, var)

Returns an environment variable value for a given work item and env var name. Raises a TypeError if the workitem is None or a KeyError if the var name does not exist item item to search var environment varialbe name

pdg.findData(item, name)

Does a depth first search for item data by name all the way to roots. Returns the matching value. Returns None if not found. item item to search name name to search for

pdg.findDirectData(item, name)

Does a depth first search for item data by name but stops search at partitions. Returns the matching value. Returns None if not found. item item to search name name to search for

pdg.findDirectResultData(item, tag, localized=False, as_list=False)

Does a depth first search for result data by tag but stops search at partitions. Returns the matching resultData. Returns None if not found. item item to search tag tag to search for localized True means localize the results, False to return as-is as_list True means results should be returned as a list

of strings, False means return as one string concatenated with ,
pdg.findResultData(item, tag, localized=False, as_list=False)

Does a depth first search for result data by tag all the way to roots. Returns the matching resultData. Returns None if not found. item item to search tag tag to search for localized True means localize the results, False to return as-is as_list True means results should be returned as a list

of strings, False means return as one string concatenated with ,
pdg.floatData(item, field, index=0)

Returns float data with the given tag and index. Raises ValueError if not found. item item to search field string tag for data index index of data in float array

pdg.floatDataArray(item, field)

Returns float data array for the given tag. Raises ValueError if not found. item item to search field string tag for data

pdg.hasFloatData(item, field, index=0)

Returns True if the item has float data with the given name and the given index. item item to search field string tag for data index index of data in int array

pdg.hasIntData(item, field, index=0)

Returns True if the item has int data with the given name and the given index. item item to search field string tag for data index index of data in int array

pdg.hasResultData(item, tag)

Returns True if there is matching resultData in the given item. item item to search tag tag to search for

pdg.hasStrData(item, field, index=0)

Returns True if the item has string data with the given name and the given index. item item to search field string tag for data index index of data in int array

pdg.intData(item, field, index=0)

Returns int data with the given tag and index. Raises ValueError if not found. item item to search field string tag for data index index of data in int array

pdg.intDataArray(item, field)

Returns int data array for the given tag. Raises ValueError if not found. item item to search field string tag for data

pdg.resultData(item, tag, localized=False, as_list=False)

Returns the matching resultData in the given item. Returns None if not found. item item to search tag tag to search for localized True means localize the results, False to return as-is as_list True means results should be returned as a list

of strings, False means return as one string concatenated with ,
pdg.resultDataIndex(item, tag, index, default='', localized=False)

Returns the matching resultData in the given item, at the given index. Returns None if not found item item to search tag tag to search for index index in the result data array default the default value, if no result is found localized True means localize the results, False to return as-is

pdg.strData(item, field, index=0)

Returns string data with the given name and index. Raises ValueError if not found. item item to search field data field name index index of data in string array

pdg.strDataArray(item, field)

Returns string data array for the given name. Raises ValueError if not found. item item to search field data field name