1. API documentation

1.1. Library

1.1.1. pbs_api_to_cli

1.1.2. pbs_ifl_mock

1.1.3. pbs_testlib

class ptl.lib.pbs_testlib.PtlConfig(conf=None)

Holds configuration options The options can be stored in a file as well as in the OS environment variables.When set, the environment variables will override definitions in the file.By default, on Unix like systems, the file read is /etc/ptl.conf, the environment variable PTL_CONF_FILE can be used to set the path to the file to read.

The format of the file is a series of <key> = <value> properties. A line that starts with a ‘#’ is ignored and can be used for comments

Parameters:conf (str or None) – Path to PTL configuration file
exception ptl.lib.pbs_testlib.PtlException(rv=None, rc=None, msg=None, post=None, *args, **kwargs)

Generic errors raised by PTL operations. Sets a return value, a return code, and a message A post function and associated positional and named arguments are available to perform any necessary cleanup.

Parameters:
  • rv (int or None.) – Return value set for the error occured during PTL operation
  • rc (int or None.) – Return code set for the error occured during PTL operation
  • msg (str or None.) – Message set for the error occured during PTL operation
  • post (callable or None.) – Execute given post callable function if not None.
Raises:

PTL exceptions

exception ptl.lib.pbs_testlib.PtlFailureException(rv=None, rc=None, msg=None, post=None, *args, **kwargs)

Generic failure exception raised by PTL operations. Sets a return value, a return code, and a message A post function and associated positional and named arguments are available to perform any necessary cleanup.

Parameters:
  • rv (int or None.) – Return value set for the failure occured during PTL operation
  • rc (int or None.) – Return code set for the failure occured during PTL operation
  • msg (str or None.) – Message set for the failure occured during PTL operation
  • post (callable or None.) – Execute given post callable function if not None.
Raises:

PTL exceptions

class ptl.lib.pbs_testlib.PbsTypeSize(value=None)

Descriptor class for memory as a numeric entity. Units can be one of b, kb, mb, gb, tb, pt

Parameters:
  • unit (str) – The unit type associated to the memory value
  • value (int or None) – The numeric value of the memory
Raises:

ValueError and TypeError

encode(value=None, valtype='kb', precision=1)

Encode numeric memory input in kilobytes to a string, including unit

Parameters:
  • value (int or None.) – The numeric value of memory to encode
  • valtype (str) – The unit of the input value, defaults to kb
  • precision (int) – Precision of the encoded value, defaults to 1
Returns:

Encoded memory in kb to string

class ptl.lib.pbs_testlib.PbsTypeDuration(val)

Descriptor class for a duration represented as hours, minutes, and seconds,in the form of [HH:][MM:]SS

Parameters:
  • as_seconds (int) – HH:MM:SS represented in seconds
  • as_str (str) – duration represented in HH:MM:SS
class ptl.lib.pbs_testlib.PbsTypeArray(value=None, sep=', ')

Descriptor class for a PBS array list type, e.g. String array

Parameters:
  • value – Array value to be passed
  • sep (str) – Separator for two array elements
Returns:

List

class ptl.lib.pbs_testlib.PbsTypeList(value=None, sep=', ', kvsep='=')

Descriptor class for a generic PBS list that are key/value pairs delimited

Parameters:
  • value – List value to be passed
  • sep (str) – Separator for two key/value pair
  • kvsep (str) – Separator for key and value
Returns:

Dictionary

class ptl.lib.pbs_testlib.PbsTypeLicenseCount(value=None)

Descriptor class for a PBS license_count attribute.

It is a specialized list where key/values are ‘:’ delimited, separated by a ‘ ‘ (space)

Parameters:value – PBS license_count attribute value
Returns:Specialized list
class ptl.lib.pbs_testlib.PbsTypeVariableList(value=None)

Descriptor class for a PBS Variable_List attribute

It is a specialized list where key/values are ‘=’ delimited, separated by a ‘,’ (space)

Parameters:value – PBS Variable_List attribute value
Returns:Specialized list
class ptl.lib.pbs_testlib.PbsTypeSelect(s=None)

Descriptor class for PBS select/schedselect specification. Select is of the form:

<select> ::= <m>":"<chunk> | <select>"+"<select>

<m> ::= <digit> | <digit><m>

<chunk> ::= <resc_name>":"<resc_value> | <chunk>":"<chunk>

<m> is a multiplying factor for each chunk requested

<chunk> are resource key/value pairs

The type populates a list of single chunk of resource key/value pairs, the list can be walked by iterating over the type itself.

Parameters:
  • num_chunks – The total number of chunks in the select
  • resources (Dictionary) – A dictionary of all resource counts in the select
class ptl.lib.pbs_testlib.PbsTypeChunk(vnode=None, resources=None, chunkstr=None)

Descriptor class for a PBS chunk associated to a PbsTypeExecVnode.This type of chunk corresponds to a node solution to a resource request,not to the select specification.

chunk ::= <subchk> | <chunk>"+"<chunk>

subchk ::= <node>":"<resource>

resource ::= <key>":"<val> | <resource>":"<resource>

A chunk expresses a solution to a specific select-chunk request. If multiple chunks are needed to solve a single select-chunk, e.g., on a shared memory system, the chunk will be extended into virtual chunk,vchunk.

Parameters:
  • vnode (str or None) – the vnode name corresponding to the chunk
  • resources (Dictionary or None) – the key value pair of resources in dictionary form
  • vchunk (list) – a list of virtual chunks needed to solve the select-chunk, vchunk is only set if more than one vchunk are required to solve the select-chunk
add(vnode, resources)

Add a chunk specificiation. If a chunk is already defined, add the chunk as a vchunk.

Parameters:
  • vnode (str) – The vnode to add
  • resources (str) – The resources associated to the vnode
Returns:

Added chunk specification

class ptl.lib.pbs_testlib.PbsTypeExecVnode(s=None)

Execvnode representation, expressed as a list of PbsTypeChunk

Parameters:
  • vchunk (List) – List of virtual chunks, only set when more than one vnode is allocated to a host satisfy a chunk requested
  • num_chunks (int) – The number of chunks satisfied by this execvnode
  • vnodes (List) – List of vnode names allocated to the execvnode
  • resource – method to return the amount of a named resource satisfied by this execvnode
resource(name=None)
Parameters:name (str or None) – Name of the resource
class ptl.lib.pbs_testlib.PbsTypeExecHost(s=None)

Descriptor class for exec_host attribute

Parameters:hosts (List) – List of hosts in the exec_host. Each entry is a host info dictionary that maps the number of cpus and its task number
class ptl.lib.pbs_testlib.PbsTypeJobId(value=None)

Descriptor class for a Job identifier

Parameters:
  • id (int) – The numeric portion of a job identifier
  • server_name (str) – The pbs server name
  • server_shortname (str) – The first portion of a FQDN server name
class ptl.lib.pbs_testlib.PbsUser(name, uid=None, groups=None)

The PbsUser type augments a PBS username to associate it to groups to which the user belongs

Parameters:
  • name (str) – The user name referenced
  • uid (int or None) – uid of user
  • groups (List or None) – The list of PbsGroup objects the user belongs to
class ptl.lib.pbs_testlib.PbsGroup(name, gid=None, users=None)

The PbsGroup type augments a PBS groupname to associate it to users to which the group belongs

Parameters:
  • name (str) – The group name referenced
  • gid (int or None) – gid of group
  • users (List or None) – The list of PbsUser objects the group belongs to
class ptl.lib.pbs_testlib.BatchUtils

Utility class to create/convert/display various PBS data structures

list_to_attrl(l)

Convert a list to a PBS attribute list

Parameters:l (List) – List to be converted
Returns:PBS attribute list
list_to_attropl(l, op=0)

Convert a list to a PBS attribute operation list

Parameters:l (List) – List to be converted
Returns:PBS attribute operation list
str_to_attrl(s)

Convert a string to a PBS attribute list

Parameters:s (str) – String to be converted
Returns:PBS attribute list
str_to_attropl(s, op=0)

Convert a string to a PBS attribute operation list

Parameters:s (str) – String to be converted
Returns:PBS attribute operation list
dict_to_attrl(d={})

Convert a dictionary to a PBS attribute list

Parameters:d (Dictionary) – Dictionary to be converted
Returns:PBS attribute list
dict_to_attropl(d={}, op=0)

Convert a dictionary to a PBS attribute operation list

Parameters:d (Dictionary) – Dictionary to be converted
Returns:PBS attribute operation list
convert_to_attrl(attrib)

Generic call to convert Python type to PBS attribute list

Parameters:attrib (List or tuple or dictionary or str) – Attributes to be converted
Returns:PBS attribute list
convert_to_attropl(attrib, cmd=13, op=None)

Generic call to convert Python type to PBS attribute operation list

Parameters:attrib (List or tuple or dictionary or str) – Attributes to be converted
Returns:PBS attribute operation list
command_to_op(cmd=None)

Map command to a SET or UNSET Operation. An unrecognized command will return SET. No command will return None.

Parameters:cmd (str) – Command to be mapped
Returns:SET or UNSET operation for the command
display_attrl(a=None, writer=<open file '<stdout>', mode 'w'>)

Display an attribute list using writer, defaults to sys.stdout

Parameters:a (List) – Attributes
Returns:Displays attribute list
display_attropl(attropl=None, writer=<open file '<stdout>', mode 'w'>)

Display an attribute operation list with writer, defaults to sys.stdout

Parameters:attropl (List) – Attribute operation list
Returns:Displays an attribute operation list
display_dict(d, writer=<open file '<stdout>', mode 'w'>)

Display a dictionary using writer, defaults to sys.stdout

Parameters:d (Dictionary) – Dictionary
Returns:Displays a dictionary
batch_status_to_dictlist(bs=None, attr_names=None, id=None)

Convert a batch status to a list of dictionaries. version 0.1a6 added this conversion as a typemap(out) as part of the swig wrapping itself so there are fewer uses for this function.Returns a list of dictionary representation of batch status

Parameters:
  • bs – Batch status
  • attr_names – Attribute names
Returns:

List of dictionaries

display_batch_status(bs=None, attr_names=None, writer=<open file '<stdout>', mode 'w'>)

Display a batch status using writer, defaults to sys.stdout :param bs: Batch status :param attr_name: Attribute name :type attr_name: str :returns: Displays batch status

display_dictlist(l=[], writer=<open file '<stdout>', mode 'w'>, fmt=None)

Display a list of dictionaries using writer, defaults to sys.stdout

Parameters:
  • l (List) – The list to display
  • writer – The stream on which to write
  • fmt (str or None) – An optional formatting string
Returns:

Displays list of dictionaries

dictlist_to_file(l=[], filename=None, mode='w')

write a dictlist to file

Parameters:
  • l (List) – Dictlist
  • filename (str) – File to which dictlist need to be written
  • mode (str) – Mode of file
Raises:

Exception writing to file

batch_status_as_dictlist_to_file(l=[], writer=<open file '<stdout>', mode 'w'>)

Write a dictlist to file

Parameters:l (List) – Dictlist
Raises:Exception writing to file
file_to_dictlist(file=None, attribs=None, id=None)

Convert a file to a batch dictlist format

Parameters:
  • file (str) – File to be converted
  • attribs – Attributes
Returns:

File converted to a batch dictlist format

file_to_vnodedef(file=None)

Convert a file output of pbsnodes -av to a vnode definition format

Parameters:file – File to be converted
Returns:Vnode definition format
show(l=[], name=None, fmt=None)

Alias to display_dictlist with sys.stdout as writer

Parameters:
  • name (str) – if specified only show the object of that name
  • fmt (str) – Optional formatting string, uses %n for object name, %a for attributes, for example a format of ‘%nE{}nE{}t%aE{}n’ will display objects with their name starting on the first column, a new line, and attributes indented by a tab followed by a new line at the end.
get_objtype(d={})

Get the type of a given object

Parameters:d (Dictionary) – Dictionary
Returns:Type of the object
display_batch_status_as_dictlist(l=[], writer=<open file '<stdout>', mode 'w'>, fmt=None)

Display a batch status as a list of dictionaries using writer, defaults to sys.stdout

Parameters:
  • l (List) – List
  • fmt (str or None) –
    • Optional format string
Returns:

Displays batch status as a list of dictionaries

batch_status_as_dict_to_str(d={}, fmt=None)

Return a string representation of a batch status dictionary

Parameters:
  • d (Dictionary) – Dictionary
  • fmt (str or None) – Optional format string
Returns:

String representation of a batch status dictionary

display_batch_status_as_dict(d={}, writer=<open file '<stdout>', mode 'w'>, fmt=None)

Display a dictionary representation of a batch status using writer, defaults to sys.stdout

Parameters:
  • d (Dictionary) – Dictionary
  • fmt – Optional format string
  • fmt – str
Returns:

Displays dictionary representation of a batch status

decode_dictlist(l=None, json=True)

decode a list of dictionaries

Parameters:
  • l (List) – List of dictionaries
  • json – The target of the decode is meant for JSON formatting
Returns:

Decoded list of dictionaries

convert_to_dictlist(l, attribs=None, mergelines=True, id=None)

Convert a list of records into a dictlist format.

Parameters:
  • l (List) – array of records to convert
  • mergelines – merge qstat broken lines into one
Returns:

Record list converted into dictlist format

convert_to_batch(l, mergelines=True)

Convert a list of records into a batch format.

Parameters:
  • l (List) – array of records to convert
  • mergelines (bool) – qstat breaks long lines over multiple lines, merge them to one by default.
Returns:

A linked list of batch status

file_to_batch(file=None)

Convert a file to batch format

Parameters:file (str or None) – File to be converted
Returns:File converted into batch format
batch_to_file(bs=None, file=None)

Write a batch object to file

Parameters:
  • bs – Batch status
  • file (str) – File to which batch object is to be written
batch_to_vnodedef(bs)
Parameters:bs – Batch status
Returns:The vnode definition string representation of nodes batch_status
dictlist_to_vnodedef(dl=None)
Parameters:dl (List) – Dictionary list
Returns:The vnode definition string representation of a dictlist
objlist_to_dictlist(objlist=None)

Convert a list of PBS/PTL objects (e.g. Server/Job...) into a dictionary list representation of the batch status

Parameters:objlist (List) – List of PBS/PTL objects
Returns:Dictionary list representation of the batch status
obj_to_dict(obj)

Convert a PBS/PTL object (e.g. Server/Job...) into a dictionary format

Parameters:objPBS/PTL object
Returns:Dictionary of PBS/PTL objects
parse_execvnode(s=None)

Parse an execvnode string into chunk objects

Parameters:s (str or None) – Execvnode string
Returns:Chunk objects for parsed execvnode string
anupbs_exechost_numhosts(s=None)
Parameters:s (str or None) – Exechost string
parse_exechost(s=None)

Parse an exechost string into a dictionary representation

Parameters:s (str or None) – String to be parsed
Returns:Dictionary format of the exechost string
parse_select(s=None)

Parse a select/schedselect string into a list of dictionaries.

Parameters:s (str or None) – select/schedselect string
Returns:List of dictonaries
classmethod isfloat(value)

returns true if value is a float or a string representation of a float returns false otherwise

Parameters:value (str or int or float) – value to be checked
Returns:True or False
classmethod decode_value(value)

Decode an attribute/resource value, if a value is made up of digits only then return the numeric value of it, if it is made of alphanumeric values only, return it as a string, if it is of type size, i.e., with a memory unit such as b,kb,mb,gb then return the converted size to kb without the unit

Parameters:value (str or int) – attribute/resource value
Returns:int or float or string
convert_time(val, fmt='%a %b %d %H:%M:%S %Y')

Convert a date time format into number of seconds since epoch

Parameters:
  • val – date time value
  • fmt (str) – date time format
Returns:

seconds

convert_duration(val)

Convert HH:MM:SS into number of seconds If a number is fed in, that number is returned If neither formatted data is fed in, returns 0

Parameters:val (str) – duration value
Raises:Incorrect format error
Returns:seconds
convert_seconds_to_datetime(tm, fmt=None, seconds=True)

Convert time format to number of seconds since epoch

Parameters:
  • tm (str) – the time to convert
  • fmt (str or None) – optional format string. If used, the seconds parameter is ignored.Defaults to %Y%m%d%H%M
  • seconds (bool) – if True, convert time with seconds granularity. Defaults to True.
Returns:

Number of seconds

convert_stime_to_seconds(st)

Convert a time to seconds, if we fail we return the original time

Parameters:st (str) – Time to be converted
Returns:Number of seconds
convert_dedtime(dtime)

Convert dedicated time string of form %m/%d/%Y %H:%M.

Parameters:dtime (str) – A datetime string, as an entry in the dedicated_time file
Returns:A tuple of (from,to) of time since epoch
convert_datetime_to_epoch(mdyhms, fmt='%m/%d/%Y %H:%M:%S')

Convert the date time to epoch

Parameters:
  • mdyhms (str) – date time
  • fmt (str) – Format for date time
Returns:

Epoch time

compare_versions(v1, v2, op=None)

Compare v1 to v2 with respect to operation op

Parameters:
  • v1 – If not a looseversion, it gets converted to it
  • v2 – If not a looseversion, it gets converted to it
  • op (str) – An operation, one of LT, LE, EQ, GE, GT
Returns:

True or False

convert_arglist(attr)

strip the XML attributes from the argument list attribute

Parameters:attr (List) – Argument list attributes
Returns:Stripped XML attributes
convert_to_cli(attrs, op=None, hostname=None, dflt_conf=True, exclude_attrs=None)

Convert attributes into their CLI format counterpart. This method is far from complete, it grows as needs come by and could use a rewrite, especially going along with a rewrite of pbs_api_to_cli

Parameters:
  • attrs (List or str or dictionary) – Attributes to convert
  • op (str or None) – The qualifier of the operation being performed, such as IFL_SUBMIT, IFL_DELETE, IFL_TERMINUTE...
  • hostname (str or None) – The name of the host on which to operate
  • dflt_conf (bool) – Whether we are using the default PBS configuration
  • exclude_attrs (List) – Optional list of attributes to not convert
Returns:

CLI format of attributes

filter_batch_status(bs, attrib)

Filter out elements that don’t have the attributes requested This is needed to adapt to the fact that requesting a resource attribute returns all '<resource-name>.*' attributes so we need to ensure that the specific resource requested is present in the stat’ed object.

This is needed especially when calling expect with an op=NE because we need to filter on objects that have exactly the attributes requested

Parameters:
  • bs – Batch status
  • attrib (str or dictionary) – Requested attributes
Returns:

Filtered batch status

convert_attributes_by_op(attributes, setattrs=False)

Convert attributes by operator, i.e. convert an attribute of the form

<attr_name><op><value> (e.g. resources_available.ncpus>4)

to

<attr_name>: (<op>, <value>) (e.g. resources_available.ncpus: (GT, 4))

Parameters:
  • attributes (List) – the attributes to convert
  • setattrs (bool) – if True, set the attributes with no operator as (SET, ‘’)
Returns:

Converted attributes by operator

operator_in_attribute(attrib)

Returns True if an operator string is present in an attribute name

Parameters:attrib (str) – Attribute name
Returns:True or False
list_resources(objtype=None, objs=[])

Lists the resources

Parameters:
  • objtype (str) – Type of the object
  • objs (List) – Object list
Returns:

List of resources

compare(obj1, obj2, showdiff=False)

Compare two objects.

Parameters:showdiff (bool) – whether to print the specific differences, defaults to False
Returns:0 if objects are identical and non zero otherwise
classmethod random_str(length=1, prefix='')

Generates the random string

Parameters:
  • length (int) – Length of the string
  • prefix (str) – Prefix of the string
Returns:

Random string

update_attributes_list(obj)

Updates the attribute list

Parameters:obj – Objects
Returns:Updated attribute list
parse_fgc_limit(limstr=None)

Parse an FGC limit entry, of the form:

<limtype>[.<resource>]=\[<entity_type>:<entity_name> =<entity_value>\]

Parameters:limstr (str or None) – FGC limit string
Returns:Parsed FGC string in given format
is_job_array(jobid)

If a job array return True, otherwise return False

Parameters:jobid – PBS jobid
Returns:True or False
is_subjob(jobid)

If a subjob of a job array, return the subjob id otherwise return False

Parameters:jobid (str) – PBS job id
Returns:True or False
class ptl.lib.pbs_testlib.PbsTypeFGCLimit(attr, val)

FGC limit entry, of the form:

<limtype>[.<resource>]=\[<entity_type>:<entity_name>= <entity_value>\]

Parameters:
  • attr (str) – FGC limit attribute
  • value (int) – Value of attribute
Returns:

FGC limit entry of given format

class ptl.lib.pbs_testlib.PbsBatchStatus(bs)

Wrapper class for Batch Status object Converts a batch status (as dictlist) into a list of PbsBatchObjects

Parameters:bs (List or dictionary) – Batch status
Returns:List of PBS batch objects
class ptl.lib.pbs_testlib.PbsAttribute(name=None, value=None)

Descriptor class for PBS attribute

Parameters:
  • name (str) – PBS attribute name
  • value (str or int or float) – Value for the attribute
set_name(name)

Set PBS attribute name

Parameters:name (str) – PBS attribute
set_value(value)

Set PBS attribute value

Parameters:value (str or int or float) – Value of PBS attribute
obfuscate_name(a=None)

Obfuscate PBS attribute name

obfuscate_value(v=None)

Obfuscate PBS attribute value

class ptl.lib.pbs_testlib.Entity(etype=None, name=None)

Abstract representation of a PBS consumer that has an external relationship to the PBS system. For example, a user associated to an OS identifier (uid) maps to a PBS user entity.

Entities may be subject to policies, such as limits, consume a certain amount of resource and/or fairshare usage.

Parameters:
  • etype (str or None) – Entity type
  • name (str or None) – Entity name
set_limit(limit=None)
Parameters:limit (str or None) – Limit to be set
set_resource_usage(container=None, resource=None, usage=None)

Set the resource type

Parameters:
  • resource (str or None) – PBS resource
  • usage (str or None) – Resource usage value
set_fairshare_usage(usage=0)

Set fairshare usage

Parameters:usage (int) – Fairshare usage value
class ptl.lib.pbs_testlib.Policy

Abstract PBS policy. Can be one of limits, access control, scheduling policy, etc...this class does not currently support any operations

class ptl.lib.pbs_testlib.Limit(limit_type=None, resource=None, entity_obj=None, value=None, container=None, container_id=None)

Representation of a PBS limit Limits apply to containers, are of a certain type (e.g., max_run_res.ncpus) associated to a given resource (e.g., resource), on a given entity (e.g.,user Bob) and have a certain value.

Parameters:
  • limit_type (str or None) – Type of the limit
  • resource (str or None) – PBS resource
  • entity_obj – Entity object
  • value (int) – Limit value
set_container(container, container_id)

Set the container

Parameters:
  • container (str) – Container which is to be set
  • container_id – Container id
set_limit_type(t)

Set the limit type

Parameters:t (str) – Limit type
set_resource(resource)

Set the resource

Parameters:resource (str) – resource value to set
set_value(value)

Set the resource value

Parameters:value (str) – Resource value
class ptl.lib.pbs_testlib.ExpectActions(action=None, level=20)

List of action handlers to run when Server’s expect function does not get the expected result

Parameters:
  • action (str) – Action to run
  • level – Logging level
add_action(action=None, hostname=None, level=20)

Add an action

Parameters:
  • action – Action to add
  • hostname (str) – Machine hostname
  • level – Logging level
has_action(name)

check whether action exists or not

Parameters:name (str) – Name of action
get_action(name)

Get an action if exists

Parameters:name (str) – Name of action
list_actions(level=20)

List an actions

Parameters:level – Logging level
get_all_actions()

Get all the action

get_actions_by_type(atype=None)

Get an action by type

Parameters:atype (str) – Action type
disable_action(action=None, name=None)

Disable an action

enable_action(action=None, name=None)

Enable an action

disable_all_actions()

Disable all actions

enable_all_actions()

Enable all actions

class ptl.lib.pbs_testlib.ExpectAction(name=None, enabled=True, atype=None, action=None, level=20)

Action function to run when Server’s expect function does not get the expected result

Parameters:atype (str) – Action type
set_name(name, level=20)

Set the actione name

Parameters:name (str) – Action name
class ptl.lib.pbs_testlib.PbsTypeAttribute

Experimental. This is a placeholder object that will be used in the future to map attribute information and circumvent the error-pron dynamic type detection that is currently done using decode_value()

class ptl.lib.pbs_testlib.PBSObject(name, attrs={}, defaults={})

Generic PBS Object encapsulating attributes and defaults

Parameters:
  • name (str) – The name associated to the object
  • attrs (Dictionary) – Dictionary of attributes to set on object
  • defaults (Dictionary) – Dictionary of default attributes. Setting this will override any other object’s default
set_attributes(a={})

set attributes and custom attributes on this object. custom attributes are used when converting attributes to CLI

Parameters:a (Dictionary) – Attribute dictionary
unset_attributes(attrl=[])

Unset attributes from object’s attributes and custom attributes

Parameters:attrl (List) – Attribute list
class ptl.lib.pbs_testlib.PBSService(name=None, attrs={}, defaults={}, pbsconf_file=None, diagmap={}, diag=None)

Generic PBS service object to hold properties of PBS daemons

Parameters:
  • name (str or None) – The name associated to the object
  • attrs (Dictionary) – Dictionary of attributes to set on object
  • defaults (Dictionary) – Dictionary of default attributes. Setting this will override any other object’s default
  • pbsconf_file (str or None) – Optional path to the pbs configuration file
  • diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
  • diag (str or None) – path to PBS diag directory (This will overrides diagmap)
init_logfile_path(conf=None)

Initialize path to log files for this service

Parameters:conf (Dictionary) – PBS conf file parameters
initialise_service()

Purpose of this method is to override and initialise the service

log_lines(logtype, id=None, n=50, tail=True, day=None, starttime=None, endtime=None)

Return the last <n> lines of a PBS log file, which can be one of server, scheduler, MoM, or tracejob

Parameters:
  • logtype (str or object) – The entity requested, an instance of a Scheduler, Server or MoM object, or the string ‘tracejob’ for tracejob
  • id – The id of the object to trace. Only used for tracejob
  • n (str or int) – One of ‘ALL’ of the number of lines to process/display, defaults to 50.
  • tail (bool) – if True, parse log from the end to the start, otherwise parse from the start to the end. Defaults to True.
  • day (int) – Optional day in YYYMMDD format. Defaults to current day
  • starttime – date timestamp to start matching
  • endtime – date timestamp to end matching
Returns:

Last <n> lines of logfile for Server, Scheduler, MoM or tracejob

accounting_match(msg, id=None, n=50, tail=True, allmatch=False, regexp=False, day=None, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)

Match given msg in given n lines of accounting log

Parameters:
  • msg (str) – log message to match, can be regex also when regexp is True
  • id (str) – The id of the object to trace. Only used for tracejob
  • n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
  • tail (bool) – If true (default), starts from the end of the file
  • allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
  • regexp (bool) – If true msg is a Python regular expression. Defaults to False
  • day (str) – Optional day in YYYMMDD format.
  • max_attempts (int) – the number of attempts to make to find a matching entry
  • interval (int) – the interval between attempts
  • starttime (int) – If set ignore matches that occur before specified time
  • endtime (int) – If set ignore matches that occur after specified time
  • level (int) – The logging level, defaults to INFO
  • existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:

(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.

Return type:

tuple

Raises:

PtlLogMatchError – When existence is True and given msg is not found in n line Or When existence is False and given msg found in n line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

tracejob_match(msg, id=None, n=50, tail=True, allmatch=False, regexp=False, day=None, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)

Match given msg in given n lines of tracejob log

Parameters:
  • msg (str) – log message to match, can be regex also when regexp is True
  • id (str) – The id of the object to trace.
  • n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
  • tail (bool) – If true (default), starts from the end of the file
  • allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
  • regexp (bool) – If true msg is a Python regular expression. Defaults to False
  • day (str) – Optional day in YYYMMDD format.
  • max_attempts (int) – the number of attempts to make to find a matching entry
  • interval (int) – the interval between attempts
  • starttime (int) – If set ignore matches that occur before specified time
  • endtime (int) – If set ignore matches that occur after specified time
  • level (int) – The logging level, defaults to INFO
  • existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:

(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.

Return type:

tuple

Raises:

PtlLogMatchError – When existence is True and given msg is not found in n line Or When existence is False and given msg found in n line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

get_tempdir()

platform independent call to get a temporary directory

class ptl.lib.pbs_testlib.Comm(name=None, attrs={}, pbsconf_file=None, diagmap={}, diag=None, server=None, db_access=None)

PBS Comm configuration and control

isUp()

Check for comm up

signal(sig)

Send signal to comm

get_pid()

Get the comm pid

all_instance_pids()

Get all pids of given instance

start(args=None, launcher=None)

Start the comm

Parameters:
  • args (str) – Argument required to start the comm
  • launcher (str or list) – Optional utility to invoke the launch of the service
stop(sig=None)

Stop the comm.

Parameters:sig (str) – Signal to stop the comm
restart()

Restart the comm.

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, day=None, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)

Match given msg in given n lines of Comm log

Parameters:
  • msg (str) – log message to match, can be regex also when regexp is True
  • id (str) – The id of the object to trace. Only used for tracejob
  • n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
  • tail (bool) – If true (default), starts from the end of the file
  • allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
  • regexp (bool) – If true msg is a Python regular expression. Defaults to False
  • day (str) – Optional day in YYYMMDD format.
  • max_attempts (int) – the number of attempts to make to find a matching entry
  • interval (int) – the interval between attempts
  • starttime (int) – If set ignore matches that occur before specified time
  • endtime (int) – If set ignore matches that occur after specified time
  • level (int) – The logging level, defaults to INFO
  • existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:

(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.

Return type:

tuple

Raises:

PtlLogMatchError – When existence is True and given msg is not found in n line Or When existence is False and given msg found in n line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

class ptl.lib.pbs_testlib.Server(name=None, attrs={}, defaults={}, pbsconf_file=None, diagmap={}, diag=None, client=None, client_pbsconf_file=None, db_access=None, stat=True)

PBS server configuration and control

The Server class is a container to PBS server attributes and implements wrappers to the IFL API to perform operations on the server. For example to submit, status, delete, manage, etc... jobs, reservations and configurations.

This class also offers higher-level routines to ease testing, see functions, for example: revert_to_defaults, init_logging, expect, counter.

The ptl_conf dictionary holds general configuration for the framework’s operations, specifically, one can control:

mode: set to PTL_CLI to operate in CLI mode or PTL_API to operate in API mode

expect_max_attempts: the default maximum number of attempts to be usedby expect. Defaults to 60

expect_interval: the default time interval (in seconds) between expectrequests. Defaults to 0.5

update_attributes: the default on whether Object attributes should beupdated using a list of dictionaries. Defaults to True

Parameters:
  • name (str) – The hostname of the server. Defaults to calling pbs_default()
  • attrs (Dictionary) – Dictionary of attributes to set, these will override defaults.
  • defaults (Dictionary) – Dictionary of default attributes. Default: dflt_attributes
  • pbsconf_file (str) – path to config file to parse for PBS_HOME, PBS_EXEC, etc
  • diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
  • diag (str) – path to PBS diag directory (This will overrides diagmap)
  • client (str) – The host to use as client for CLI queries. Defaults to the local hostname.
  • client_pbsconf_file (str) – The path to a custom PBS_CONF_FILE on the client host. Defaults to the same path as pbsconf_file.
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {‘dbname’:...,’user’:...,’port’:...}
  • stat (bool) – if True, stat the server attributes
update_version_info()

Update the version information.

classmethod set_update_attributes(val)

Set update attributes

classmethod set_expect_max_attempts(val)

Set expect max attempts

classmethod set_expect_interval(val)

Set expect interval

set_client(name=None)

Set server client

Parameters:name (str) – Client name
set_connect_timeout(timeout=0)

Set server connection timeout

Parameters:timeout (int) – Timeout value
get_op_mode()

Returns operating mode for calls to the PBS server. Currently, two modes are supported, either the API or the CLI. Default is API

set_op_mode(mode)

set operating mode to one of either PTL_CLI or PTL_API.Returns the mode that was set which can be different from the value requested, for example, if requesting to set PTL_API, in the absence of the appropriate SWIG wrappers, the library will fall back to CLI, or if requesting PTL_CLI and there is no PBS_EXEC on the system, None is returned.

Parameters:mode (str) – Operating mode
add_expect_action(name=None, action=None)

Add an action handler to expect. Expect Actions are custom handlers that are triggered when an unexpected value is encountered

Parameters:
  • name (str or None) – Action name
  • action – Action to add
set_attributes(a={})

set server attributes

Parameters:a (Dictionary) – Attribute dictionary
isUp()

returns True if server is up and False otherwise

signal(sig)

Send signal to server

Parameters:sig (str) – Signal to send
get_pid()

Get the server pid

all_instance_pids()

Get all pids for a given instance

start(args=None, launcher=None)

Start the PBS server

Parameters:
  • args (str) – Argument required to start the server
  • launcher (str or list) – Optional utility to invoke the launch of the service
stop(sig=None)

Stop the PBS server

Parameters:sig (str) – Signal to stop PBS server
restart()

Terminate and start a PBS server.

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, day=None, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)

Match given msg in given n lines of Server log

Parameters:
  • msg (str) – log message to match, can be regex also when regexp is True
  • id (str) – The id of the object to trace. Only used for tracejob
  • n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
  • tail (bool) – If true (default), starts from the end of the file
  • allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
  • regexp (bool) – If true msg is a Python regular expression. Defaults to False
  • day (str) – Optional day in YYYMMDD format.
  • max_attempts (int) – the number of attempts to make to find a matching entry
  • interval (int) – the interval between attempts
  • starttime (int) – If set ignore matches that occur before specified time
  • endtime (int) – If set ignore matches that occur after specified time
  • level (int) – The logging level, defaults to INFO
  • existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:

(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.

Return type:

tuple

Raises:

PtlLogMatchError – When existence is True and given msg is not found in n line Or When existence is False and given msg found in n line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

revert_to_defaults(reverthooks=True, revertqueues=True, revertresources=True, delhooks=True, delqueues=True, delscheds=True, server_stat=None)

reset server attributes back to out of box defaults.

Parameters:
  • reverthooks (bool) – If True disable all hooks. Defaults to True
  • revertqueues (bool) – If True disable all non-default queues. Defaults to True
  • revertresources (bool) – If True, resourcedef file is removed. Defaults to True. Reverting resources causes a server restart to occur.
  • delhooks (bool) – If True, hooks are deleted, if deletion fails, fall back to reverting hooks. Defaults to True.
  • delqueues (bool) – If True, all non-default queues are deleted, will attempt to delete all jobs first, if it fails, revertqueues will be honored, otherwise,revertqueues is ignored. Defaults to True
  • delscheds (bool) – If True all non-default schedulers are deleted The sched_priv and sched_logs directories will be deleted.
Returns:

True upon success and False if an error is encountered.

Raises:

PbsStatusError or PbsManagerError

save_configuration(outfile, mode='a')

Save a server configuration, this includes:

  • server_priv/resourcedef
  • qmgr -c "print server"
  • qmgr -c "print sched"
  • qmgr -c "print hook"
Parameters:
  • outfile (str) – the output file to which onfiguration is saved
  • mode (str) – The mode in which to open outfile to save configuration. The first object being saved should open this file with ‘w’ and subsequent calls from other objects should save with mode ‘a’ or ‘a+’. Defaults to a+
Returns:

True on success, False on error

load_configuration(infile)

load configuration from saved file infile

get_hostname()

return the default server hostname

status_db(obj_type=None, attrib=None, id=None, db_access=None, logerr=True)

Status PBS objects from the SQL database

Parameters:
  • obj_type – The type of object to query, one of the * objects,Default: SERVER
  • attrib (str or list or dictionary) – Attributes to query, can a string, a list, a dictionaryDefault: None. All attributes will be queried
  • id (str) – An optional identifier, the name of the object to status
  • db_access (str or dictionary) – information needed to access the database, can be either a file containing user, port, dbname, password info or a dictionary of key/value entries
status(obj_type=0, attrib=None, id=None, extend=None, level=20, db_access=None, runas=None, resolve_indirectness=False, logerr=True)

Stat any PBS object [queue, server, node, hook, job, resv, sched].If the Server is setup from diag input, see diag or diagmap member, the status calls are routed directly to the data on files from diag.

The server can be queried either through the ‘qstat’ command line tool or through the wrapped PBS IFL api, see set_op_mode.

Return a dictionary representation of a batch status object raises PbsStatsuError on error.

Parameters:
  • obj_type – The type of object to query, one of the * objects.Default: SERVER
  • attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary.Default is to query all attributes.
  • id (str) – An optional id, the name of the object to status
  • extend – Optional extension to the IFL call
  • level (str) – The logging level, defaults to INFO
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
  • runas (str) – run stat as user
  • resolve_indirectness (bool) – If True resolves indirect node resources values
  • logerr (bool) – If True (default) logs run_cmd errors

In addition to standard IFL stat call, this wrapper handles a few cases that aren’t implicitly offered by pbs_stat*, those are for Hooks,Resources, and a formula evaluation.

submit_interactive_job(job, cmd)

submit an interactive job. Returns a job identifier or raises PbsSubmitError on error

Parameters:
  • cmd (str) – The command to run to submit the interactive job
  • job – the job object. The job must have the attribute ‘interactive_job’ populated. That attribute is a list of tuples of the form: (<command>, <expected output>, <...>) for example to send the command hostname and expect ‘myhost.mydomain’ one would set:job.interactive_job = [(‘hostname’, ‘myhost.mydomain’)] If more than one lines are expected they are appended to the tuple.
Raises:

PbsSubmitError

submit(obj, script=None, extend=None, submit_dir=None)

Submit a job or reservation. Returns a job identifier or raises PbsSubmitError on error

Parameters:
  • obj – The Job or Reservation instance to submit
  • script (str or None) – Path to a script to submit. Default: None as an executable/bin/sleep 100 is submitted
  • extend (str or None) – Optional extension to the IFL call. see pbs_ifl.h
  • submit_dir (str or None) – directory from which job is submitted. Defaults to temporary directory
Raises:

PbsSubmitError

deljob(id=None, extend=None, runas=None, wait=False, logerr=True, attr_W=None)

delete a single job or list of jobs specified by id raises PbsDeljobError on error

Parameters:
  • id (str or list) – The identifier(s) of the jobs to delete
  • extend (str or None) – Optional parameters to pass along to PBS
  • runas (str or None) – run as user
  • wait (bool) – Set to True to wait for job(s) to no longer be reported by PBS. False by default
  • logerr (bool) – Whether to log errors. Defaults to True.
  • attr_w (str) – -W args to qdel (Only for cli mode)
Raises:

PbsDeljobError

delresv(id=None, extend=None, runas=None, wait=False, logerr=True)

delete a single job or list of jobs specified by id raises PbsDeljobError on error

Parameters:
  • id (str or list) – The identifier(s) of the jobs to delete
  • extend (str or None) – Optional parameters to pass along to PBS
  • runas (str or None) – run as user
  • wait (bool) – Set to True to wait for job(s) to no longer be reported by PBS. False by default
  • logerr (bool) – Whether to log errors. Defaults to True.
Raises:

PbsDeljobError

delete(id=None, extend=None, runas=None, wait=False, logerr=True)

delete a single job or list of jobs specified by id raises PbsDeleteError on error

Parameters:
  • id (str or list) – The identifier(s) of the jobs/resvs to delete
  • extend (str or none) – Optional parameters to pass along to PBS
  • runas (str) – run as user
  • wait (bool) – Set to True to wait for job(s)/resv(s) to no longer be reported by PBS. False by default
  • logerr (bool) – Whether to log errors. Defaults to True.
Raises:

PbsDeleteError

select(attrib=None, extend=None, runas=None, logerr=True)

Select jobs that match attributes list or all jobs if no attributes raises PbsSelectError on error

Parameters:
  • attrib (str or list or dictionary) – A string, list, or dictionary of attributes
  • extend (str or None) – the extended attributes to pass to select
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Returns:

A list of job identifiers that match the attributes specified

Raises:

PbsSelectError

selstat(select_list, rattrib, runas=None, extend=None)

stat and filter jobs attributes.

Parameters:
  • select_list – The filter criteria
  • rattrib (List) – The attributes to query
  • runas (str or None) – run as user

Note

No CLI counterpart for this call

manager(cmd, obj_type, attrib=None, id=None, extend=None, expect=False, max_attempts=None, level=20, sudo=None, runas=None, logerr=True)

issue a management command to the server, e.g to set an attribute

Returns the return code of qmgr/pbs_manager() on success, if expect is set to True, the return value is that of the call to expect.Raises PbsManagerError on error

Parameters:
  • cmd (str) – The command to issue, MGR_CMD_[SET,UNSET, LIST,...] see pbs_ifl.h
  • obj_type – The type of object to query, one of the * objects
  • attrib (str or list or dictionary) – Attributes to operate on, can be a string, a list,a dictionary
  • id (str or list) – The name or list of names of the object(s) to act upon.
  • extend (str or None) – Optional extension to the IFL call. see pbs_ifl.h
  • expect (bool) – If set to True, query the server expecting the value to beaccurately reflected. Defaults to False
  • max_attempts (int) – Sets a maximum number of attempts to call expect with.
  • level – logging level
  • sudo (bool) – If True, run the manager command as super user. Defaults to None. Some attribute settings should be run with sudo set to True, those are acl_roots, job_sort_formula, hook operations, no_sched_hook_event, in those cases, setting sudo to False is only needed for testing purposes
  • runas (str) – run as user
  • logerr (bool) – If False, CLI commands do not log error, i.e. silent mode
Raises:

PbsManagerError

When expect is False, return the value, 0/!0 returned by pbs_manager When expect is True, return the value, True/False, returned by expect

sigjob(jobid=None, signal=None, extend=None, runas=None, logerr=True)

Send a signal to a job. Raises PbsSignalError on error.

Parameters:
  • jobid (str or list) – identifier of the job or list of jobs to send the signal to
  • signal (str or None) – The signal to send to the job, see pbs_ifl.h
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsSignalError

msgjob(jobid=None, to_file=None, msg=None, extend=None, runas=None, logerr=True)

Send a message to a job. Raises PbsMessageError on error.

Parameters:
  • jobid (str or List) – identifier of the job or list of jobs to send the message to
  • msg (str or None) – The message to send to the job
  • to_file (str or None) – one of MSG_ERR or MSG_OUT or MSG_ERR|MSG_OUT
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsMessageError

alterjob(jobid=None, attrib=None, extend=None, runas=None, logerr=True)

Alter attributes associated to a job. Raises PbsAlterError on error.

Parameters:
  • jobid (str or list) – identifier of the job or list of jobs to operate on
  • attrib (dictionary) – A dictionary of attributes to set
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If False, CLI commands do not log error, i.e. silent mode
Raises:

PbsAlterError

holdjob(jobid=None, holdtype=None, extend=None, runas=None, logerr=True)

Hold a job. Raises PbsHoldError on error.

Parameters:
  • jobid (str or list) – identifier of the job or list of jobs to hold
  • holdtype (str or None) – The type of hold to put on the job
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsHoldError

rlsjob(jobid, holdtype, extend=None, runas=None, logerr=True)

Release a job. Raises PbsReleaseError on error.

Parameters:
  • jobid (str or list) – job or list of jobs to release
  • holdtype (str) – The type of hold to release on the job
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsReleaseError

rerunjob(jobid=None, extend=None, runas=None, logerr=True)

Rerun a job. Raises PbsRerunError on error. :param jobid: job or list of jobs to release :type jobid: str or list :param extend: extend options :param runas: run as user :type runas: str or None :param logerr: If True (default) logs run_cmd errors :type logerr: bool :raises: PbsRerunError

orderjob(jobid1=None, jobid2=None, extend=None, runas=None, logerr=True)

reorder position of jobid1 and jobid2. Raises PbsOrderJob on error.

Parameters:
  • jobid1 (str or None) – first jobid
  • jobid2 (str or None) – second jobid
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsOrderJob

runjob(jobid=None, location=None, async=False, extend=None, runas=None, logerr=False)

Run a job on given nodes. Raises PbsRunError on error.

Parameters:
  • jobid (str or list) – job or list of jobs to run
  • location (str or None) – An execvnode on which to run the job
  • async (bool) – If true the call will return immediately assuming success.
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsRunError

movejob(jobid=None, destination=None, extend=None, runas=None, logerr=True)

Move a job or list of job ids to a given destination queue. Raises PbsMoveError on error.

Parameters:
  • jobid (str or list) – A job or list of job ids to move
  • destination (str or None) – The destination queue@server
  • extend – extend options
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsMoveError

qterm(manner=None, extend=None, server_name=None, runas=None, logerr=True)

Terminate the pbs_server daemon

Parameters:
  • manner – one of (SHUT_IMMEDIATE | SHUT_DELAY | SHUT_QUICK) and can be combined with SHUT_WHO_SCHED, SHUT_WHO_MOM, SHUT_WHO_SECDRY, SHUT_WHO_IDLESECDRY, SHUT_WHO_SECDONLY. :param extend: extend options
  • server_name (str or None) – name of the pbs server
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsQtermError

teminate(manner=None, extend=None, server_name=None, runas=None, logerr=True)

Terminate the pbs_server daemon

Parameters:
  • manner – one of (SHUT_IMMEDIATE | SHUT_DELAY | SHUT_QUICK) and can be combined with SHUT_WHO_SCHED, SHUT_WHO_MOM, SHUT_WHO_SECDRY, SHUT_WHO_IDLESECDRY, SHUT_WHO_SECDONLY. :param extend: extend options
  • server_name (str or None) – name of the pbs server
  • runas (str or None) – run as user
  • logerr (bool) – If True (default) logs run_cmd errors
Raises:

PbsQtermError

geterrmsg()

Get the error message

qdisable(queue=None, runas=None, logerr=True)

Disable queue. CLI mode only

Parameters:
  • queue (str or list) – The name of the queue or list of queue to disable
  • runas (str or None) – Optional name of user to run command as
  • logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises:

PbsQdisableError

qenable(queue=None, runas=None, logerr=True)

Enable queue. CLI mode only

Parameters:
  • queue (str or list) – The name of the queue or list of queue to enable
  • runas (str or None) – Optional name of user to run command as
  • logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises:

PbsQenableError

qstart(queue=None, runas=None, logerr=True)

Start queue. CLI mode only

Parameters:
  • queue (str or list) – The name of the queue or list of queue to start
  • runas (str or None) – Optional name of user to run command as
  • logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises:

PbsQstartError

qstop(queue=None, runas=None, logerr=True)

Stop queue. CLI mode only

Parameters:
  • queue (str or list) – The name of the queue or list of queue to stop
  • runas (str or None) – Optional name of user to run command as
  • logerr (bool) – Set to False ot disable logging command errors. Defaults to True.
Raises:

PbsQstopError

parse_resources()

Parse server resources as defined in the resourcedef file Populates instance variable self.resources

Returns:The resources as a dictionary
remove_resource(name)

Remove an entry from resourcedef

Parameters:
  • name (str) – The name of the resource to remove
  • restart – Whether to restart the server or not. Applicable to update_mode ‘file’ operations only.
  • update_mode – one of ‘file’ or ‘auto’ (the default). If ‘file’, updates the resourcedef file only and will not use the qmgr operations on resources introduced in 12.3. If ‘auto’, will automatically handle the update on resourcedef or using qmgr based on the version of the Server.
add_resource(name, type=None, flag=None)

Define a server resource

Parameters:
  • name (str) – The name of the resource to add to the resourcedef file
  • type – The type of the resource, one of string, long, boolean, float
  • flag (str or None) – The target of the resource, one of n, h, q, or none
  • restart – Whether to restart the server after adding a resource.Applicable to update_mode ‘file’ operations only.
  • update_mode – one of ‘file’ or ‘auto’ (the default). If ‘file’, updates the resourcedef file only and will not use the qmgr operations on resources introduced in 12.3. If ‘auto’, will automatically handle the update on resourcedef or using qmgr based on the version of the Server.
Returns:

True on success False on error

write_resourcedef(resources=None, filename=None, restart=True)

Write into resource def file

Parameters:
  • resources (dictionary) – PBS resources
  • filename (str or None) – resourcedef file name
parse_resourcedef(file=None)

Parse an arbitrary resource definition file passed as input and return a dictionary of resources

Parameters:file (str or None) – resource definition file
Returns:Dictionary of resource
Raises:PbsResourceError
pbs_api_as(cmd=None, obj=None, user=None, **kwargs)

Generic handler to run an API call impersonating a given user.This method is only used for impersonation over the API because CLI impersonation takes place through the generic DshUtils run_cmd mechanism.

Parameters:
  • cmd (str or None) – PBS command
  • user (str or None) – PBS user or current user
Raises:

eval

alterresv(resvid, attrib, extend=None, runas=None, logerr=True)

Alter attributes associated to a reservation. Raises PbsResvAlterError on error.

Parameters:
  • resvid (str.) – identifier of the reservation.
  • attrib (dictionary.) – A dictionary of attributes to set.
  • extend – extend options.
  • runas (str or None.) – run as user.
  • logerr (bool.) – If False, CLI commands do not log error, i.e. silent mode.
Raises:

PbsResvAlterError.

expect(obj_type, attrib=None, id=None, op=4, attrop=0, attempt=0, max_attempts=None, interval=None, count=None, extend=None, offset=0, runas=None, level=20, msg=None)

expect an attribute to match a given value as per an operation.

Parameters:
  • obj_type (str) – The type of object to query, JOB, SERVER, SCHEDULER, QUEUE NODE
  • attrib (str or list or dictionary) – Attributes to query, can be a string, a list, or a dict
  • id – The id of the object to act upon
  • op – An operation to perform on the queried data, e.g., EQ, SET, LT,..
  • attrop – Operation on multiple attributes, either PTL_AND, PTL_OR when an PTL_AND is used, only batch objects having all matches are returned, otherwise an OR is applied
  • attempt (int) – The number of times this function has been called
  • max_attempts (int or None) – The maximum number of attempts to perform.C{param_max_attempts}: 5
  • interval – The interval time btween attempts. C{param_interval}: 1s
  • count (bool) – If True, attrib will be accumulated using function counter
  • extend – passed to the stat call
  • offset (int) – the time to wait before the initial check. Defaults to 0.
  • runas (str or None) – query as a given user. Defaults to current user
  • msg (str or None) – Message from last call of this function, this message will be used while raising PtlExpectError.
Returns:

True if attributes are as expected and False otherwise

is_history_enabled()

Short-hand method to return the value of job_history_enable

cleanup_jobs(extend=None, runas=None)

Helper function to delete all jobs. By default this method will determine whether job_history_enable is on and will cleanup all history jobs. Specifying an extend parameter could override this behavior.

Parameters:runas (str or None) – Clean the job as
cleanup_reservations(extend=None, runas=None)

Helper function to delete all reservations

cleanup_jobs_and_reservations(extend='forcedeletehist')

Helper function to delete all jobs and reservations

Parameters:
  • extend – Optional extend parameter that is passed to delete. It defaults to ‘deletehist’ which is used in qdel and pbs_deljob() to force delete all jobs, including history jobs
  • extend – str
update_attributes(obj_type, bs)

Populate objects from batch status data

counter(obj_type=None, attrib=None, id=None, extend=None, op=None, attrop=None, bslist=None, level=20, idonly=True, grandtotal=False, db_access=None, runas=None, resolve_indirectness=False)

Accumulate properties set on an object. For example, to count number of free nodes: server.counter(VNODE,{'state':'free'})

Parameters:
  • obj_type – The type of object to query, one of the * objects
  • attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary
  • id – The id of the object to act upon
  • extend – The extended parameter to pass to the stat call
  • op (str or None) – The operation used to match attrib to what is queried. SET or None
  • attrop – Operation on multiple attributes, either PTL_AND, PTL_OR
  • bslist – Optional, use a batch status dict list instead of an obj_type
  • idonly (bool) – if true, return the name/id of the matching objects
  • db_access (str or dictionary) – credentials to access db, either a path to file or dictionary
  • runas (str or None) – run as user
filter(obj_type=None, attrib=None, id=None, extend=None, op=None, attrop=None, bslist=None, idonly=True, grandtotal=False, db_access=None, runas=None, resolve_indirectness=False)

Filter objects by properties. For example, to filter all free nodes:server.filter(VNODE,{'state':'free'})

For each attribute queried, if idonly is True, a list of matching object names is returned; if idonly is False, then the value of each attribute queried is returned.

This is unlike Python’s built-in ‘filter’ that returns a subset of objects matching from a pool of objects. The Python filtering mechanism remains very useful in some situations and should be used programmatically to achieve desired filtering goals that can not be met easily with PTL’s filter method.

Parameters:
  • obj_type – The type of object to query, one of the * objects
  • attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary
  • id – The id of the object to act upon
  • extend – The extended parameter to pass to the stat call
  • op (str or None) – The operation used to match attrib to what is queried. SET or None
  • bslist (List or None) – Optional, use a batch status dict list instead of an obj_type
  • idonly (bool) – if true, return the name/id of the matching objects
  • db_access (str or dictionary) – credentials to access db, either path to file or dictionary
  • runas (str or None) – run as user
logit(msg, obj_type, attrib, id, level=20)

Generic logging routine for IFL commands

Parameters:
  • msg (str) – The message to log
  • obj_type – object type, i.e *
  • attrib – attributes to log
  • id (str or list) – name of object to log
  • level – log level, defaults to INFO
equivalence_classes(obj_type=None, attrib={}, bslist=None, op=0, show_zero_resources=True, db_access=None, resolve_indirectness=False)
Parameters:
  • obj_type – PBS Object to query, one of *
  • attrib (dictionary) – attributes to build equivalence classes out of.
  • bslist (List) – Optional, list of dictionary representation of a batch status
  • op – set to RESOURCES_AVAILABLE uses the dynamic amount of resources available, i.e., available - assigned, otherwise uses static amount of resources available
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
show_equivalence_classes(eq=None, obj_type=None, attrib={}, bslist=None, op=0, show_zero_resources=True, db_access=None, resolve_indirectness=False)

helper function to show the equivalence classes

Parameters:
  • eq – equivalence classes as compute by equivalence_classes see equivalence_classes for remaining parameters description
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
whats_available(attrib=None, jobs=None, resvs=None, nodes=None)

Returns what’s available as a list of node equivalence classes listed by availability over time.

Parameters:
  • attrib (List) – attributes to consider
  • jobs – jobs to consider, if None, jobs are queried locally
  • resvs – reservations to consider, if None, they are queried locally
  • nodes – nodes to consider, if None, they are queried locally
show_whats_available(wa=None, attrib=None, jobs=None, resvs=None, nodes=None)

helper function to show availability as computed by whats_available

Parameters:wa (Dictionary) – a dictionary of available attributes. see whats_available for a description of the remaining parameters
utilization(resources=None, nodes=None, jobs=None, entity={})

Return utilization of consumable resources on a set of nodes

Parameters:
  • nodes (List) – A list of dictionary of nodes on which to compute utilization.Defaults to nodes resulting from a stat call to the current server.
  • resources (List) – comma-separated list of resources to compute utilization on. The name of the resource is for example, ncpus or mem
  • entity (Dictionary) – An optional dictionary of entities to compute utilization of, e.g. {'user':u1, 'group':g1, 'project'=p1}

The utilization is returned as a dictionary of percentage utilization for each resource.

Non-consumable resources are silently ignored.

create_vnodes(name=None, attrib=None, num=1, mom=None, additive=False, sharednode=True, restart=True, delall=True, natvnode=None, usenatvnode=False, attrfunc=None, fname=None, vnodes_per_host=1, createnode=True, expect=True)

helper function to create vnodes.

Parameters:
  • name (str or None) – prefix name of the vnode(s) to create
  • attrib – attributes to assign to each node
  • num (int) – the number of vnodes to create. Defaults to 1
  • mom – the MoM object on which the vnode definition is to be inserted
  • additive (bool) – If True, vnodes are added to the existing vnode defs.Defaults to False.
  • sharednode (bool) – If True, all vnodes will share the same host.Defaults to True.
  • restart (bool) – If True the MoM will be restarted.
  • delall (bool) – If True delete all server nodes prior to inserting vnodes
  • natvnode (str or None) – name of the natural vnode.i.e. The node name in qmgr -c “create node <name>”
  • usenatvnode (bool) – count the natural vnode as an allocatable node.
  • attrfunc – an attribute=value function generator, see create_vnode_def
  • fname (str or None) – optional name of the vnode def file
  • vnodes_per_host (int) – number of vnodes per host
  • createnode (bool) – whether to create the node via manage or not. Defaults to True
  • expect (bool) – whether to expect attributes to be set or not. Defaults to True
Returns:

True on success and False otherwise

create_moms(name=None, attrib=None, num=1, delall=True, createnode=True, conf_prefix='pbs.conf_m', home_prefix='pbs_m', momhosts=None, init_port=15011, step_port=2)

Create MoM configurations and optionall add them to the server. Unique pbs.conf files are defined and created on each hosts on which MoMs are to be created.

Parameters:
  • name (str or None) – Optional prefix name of the nodes to create. Defaults to the name of the MoM host.
  • attrib – Optional node attributes to assign to the MoM.
  • num (int) – Number of MoMs to create
  • delall (bool) – Whether to delete all nodes on the server. Defaults to True.
  • createnode (bool) – Whether to create the nodes and add them to the server.Defaults to True.
  • conf_prefix (str) – The prefix of the PBS conf file.Defaults to pbs.conf_m
  • home_prefix (str) – The prefix of the PBS_HOME directory. Defaults to pbs_m
  • momhosts (List) – A list of hosts on which to deploy num MoMs.
  • init_port (int) – The initial port number to start assigning PBS_MOM_SERIVCE_PORT to. Default 15011.
  • step_port (int) – The increments at which ports are allocated. Defaults to 2.

Note

Since PBS requires that PBS_MANAGER_SERVICE_PORT = PBS_MOM_SERVICE_PORT+1 The step number must be greater or equal to 2.

create_hook(name, attrs)

Helper function to create a hook by name.

Parameters:
  • name (str) – The name of the hook to create
  • attrs (str) – The attributes to create the hook with.
Returns:

False if hook already exists

Raises:

PbsManagerError, otherwise return True.

import_hook(name, body)

Helper function to import hook body into hook by name. The hook must have been created prior to calling this function.

Parameters:
  • name (str) – The name of the hook to import body to
  • body (str) – The body of the hook as a string.
Returns:

True on success.

Raises:

PbsManagerError

create_import_hook(name, attrs=None, body=None, overwrite=True)

Helper function to create a hook, import content into it, set the event and enable it.

Parameters:
  • name (str) – The name of the hook to create
  • attrs (str) – The attributes to create the hook with. Event and Enabled are mandatory. No defaults.
  • body (str) – The hook body as a string
  • overwrite – If True, if a hook of the same name already exists, bypass its creation. Defaults to True
Returns:

True on success and False otherwise

evaluate_formula(jobid=None, formula=None, full=True, include_running_jobs=False, exclude_subjobs=True)

Evaluate the job sort formula

Parameters:
  • jobid (str or None) – If set, evaluate the formula for the given jobid, if not set,formula is evaluated for all jobs in state Q
  • formula – If set use the given formula. If not set, the server’s formula, if any, is used
  • full (bool) – If True, returns a dictionary of job identifiers as keys and the evaluated formula as values. Returns None if no formula is used. Each job id formula is returned as a tuple (s,e) where s is the formula expression associated to the job and e is the evaluated numeric value of that expression, for example, if job_sort_formula is ncpus + mem a job requesting 2 cpus and 100kb of memory would return (‘2 + 100’, 102). If False, if a jobid is specified, return the integer value of the evaluated formula.
  • include_running_jobs (bool) – If True, reports formula value of running jobs. Defaults to False.
  • exclude_subjobs (bool) – If True, only report formula of parent job array
parse_server_limits(server=None, db_access=None)

Parse all server limits

Parameters:
  • server (List) – list of dictionary of server data
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
parse_queue_limits(queues=None, id=None, db_access=None)

Parse queue limits

Parameters:
  • queues (List) – list of dictionary of queue data
  • id – The id of the queue to parse limit for. If None, all queue limits are parsed
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
parse_all_limits(server=None, queues=None, db_access=None)

Parse all server and queue limits

Parameters:
  • server (List) – list of dictionary of server data
  • queues (List) – list of dictionary of queue data
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
limits_info(etype=None, ename=None, server=None, queues=None, jobs=None, db_access=None, over=False)

Collect limit information for each entity on which a server/queue limit is applied.

Parameters:
  • etype (str or None) – entity type, one of u, g, p, o
  • ename (str or None) – entity name
  • server (List) – optional list of dictionary representation of server object
  • queues (List) – optional list of dictionary representation of queues object
  • jobs (List) – optional list of dictionary representation of jobs object
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
  • over (bool) – If True, show only entities that are over their limit.Default is False.
Returns:

A list of dictionary similar to that returned by a converted batch_status object, i.e., can be displayed using the Utils.show method

clusterize(conf_file=None, hosts=None, import_jobs=False, db_creds_file=None)

Mimic a pbs_diag snapshot onto a set of hosts running a PBS server,``scheduler``, and MoM.

This method clones the following information from the diag:

Server attributes Server resourcedef Hooks Scheduler configuration Scheduler resource_group Scheduler holiday file Per Queue attributes

Nodes are copied as a vnode definition file inserted into each host’s MoM instance.

Currently no support for cloning the server ‘sched’ object, nor to copy nodes to multi-mom instances.

Jobs are copied over only if import_jobs is True, see below for details

Parameters:
  • asdiag (str) – Path to the pbs_diag snapshot to use
  • conf_file – Configuration file for the MoM instance
  • hosts (List) – List of hosts on which to clone the diag snapshot
  • include_jobs (bool) – [Experimental] if True jobs from the pbs_diag are imported into the host’s database. There are several caveats to this option: The scripts are not imported The users and groups are not created on the local system.There are no actual processes created on the MoM for each job so operations on the job such as signals or delete will fail (delete -W force will still work)
  • db_creds_file (str or None) – Path to file containing credentials to access the DB
class ptl.lib.pbs_testlib.EquivClass(name, attributes={}, entities=[])

Equivalence class holds information on a collection of entities grouped according to a set of attributes :param attributes: Dictionary of attributes :type attributes: Dictionary :param entities: List of entities :type entities: List

add_entity(entity)

Add entities

Parameters:entity (str) – Entity to add
show(showobj=False)

Show the entities

Parameters:showobj (bool) – If true then show the entities
class ptl.lib.pbs_testlib.Resource(name=None, type=None, flag=None)

PBS resource referenced by name, type and flag

Parameters:
  • name (str or None) – Resource name
  • type – Type of resource
set_name(name)

Set the resource name

set_type(type)

Set the resource type

set_flag(flag)

Set the flag

class ptl.lib.pbs_testlib.Holidays

Descriptive calss for Holiday file.

class ptl.lib.pbs_testlib.Scheduler(hostname=None, server=None, pbsconf_file=None, diagmap={}, diag=None, db_access=None, id='default', sched_priv=None)

Container of Scheduler related properties

Parameters:
  • hostname (str or None) – The hostname on which the scheduler instance is operating
  • server – A PBS server instance to which this scheduler is associated
  • pbsconf_file (str or None) – path to a PBS configuration file
  • diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
  • diag (str or None) – path to PBS diag directory (This will overrides diagmap)
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {'dbname':...,'user':...,'port':...}
setup_sched_priv(sched_priv=None)

Initialize Scheduler() member variables on initialization or if sched_priv changes

initialise_service()

initialise the scheduler object

isUp()

Check for PBS scheduler up

signal(sig)

Send a signal to PBS scheduler

get_pid()

Get the PBS scheduler pid

all_instance_pids()

Get the all pids for the instance

start(sched_home=None, args=None, launcher=None)

Start the scheduler :param sched_home: Path to scheduler log and home directory :type sched_home: str :param args: Arguments required to start the scheduler :type args: str :param launcher: Optional utility to invoke the launch of the service :type launcher: str or list

stop(sig=None)

Stop the PBS scheduler

Parameters:sig (str) – Signal to stop the PBS scheduler
restart()

Restart the PBS scheduler

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, day=None, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)

Match given msg in given n lines of Scheduler log

Parameters:
  • msg (str) – log message to match, can be regex also when regexp is True
  • id (str) – The id of the object to trace. Only used for tracejob
  • n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
  • tail (bool) – If true (default), starts from the end of the file
  • allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
  • regexp (bool) – If true msg is a Python regular expression. Defaults to False
  • day (str) – Optional day in YYYMMDD format.
  • max_attempts (int) – the number of attempts to make to find a matching entry
  • interval (int) – the interval between attempts
  • starttime (int) – If set ignore matches that occur before specified time
  • endtime (int) – If set ignore matches that occur after specified time
  • level (int) – The logging level, defaults to INFO
  • existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:

(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.

Return type:

tuple

Raises:

PtlLogMatchError – When existence is True and given msg is not found in n line Or When existence is False and given msg found in n line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

pbs_version()

Get the version of the scheduler instance

parse_sched_config(schd_cnfg=None)

Parse a sceduling configuration file into a dictionary. Special handling of identical keys (e.g., node_sort_key) is done by appending a delimiter, ‘%’, between each value of the key. When printed back to file, each delimited entry gets written on a line of its own. For example, the python dictionary entry:

{'node_sort_key': ["ncpus HIGH unusued" prime", "node_priority HIH" non-prime"]}

will get written as:

node_sort_key: "ncpus HIGH unusued" prime node_sort_key: "node_priority HIGH"  non-prime

Returns sched_config dictionary that gets reinitialized every time this method is called.

check_defaults(config)

Check the values in argument config against default values

apply_config(config=None, validate=True, path=None)

Apply the configuration specified by config

Parameters:
  • config – Configurations to set. Default: self. sched_config
  • validate (bool) – If True (the default) validate that settings did not yield an error. Validation is done by parsing the scheduler log which, in some cases may be slow and therefore undesirable.
  • path (str) – Optional path to file to which configuration is written. If None, the configuration is written to PBS_HOME/sched_priv/sched_config
Returns:

True on success and False otherwise. Success means that upon applying the new configuration the scheduler did not emit an “Error reading line” in its log file.

set_sched_config(confs={}, apply=True, validate=True)

set a sched_config property

Parameters:
  • confs (Dictionary) – dictionary of key value sched_config entries
  • apply (bool) – if True (the default), apply configuration.
  • validate (bool) – if True (the default), validate the configuration settings.
add_server_dyn_res(custom_resource, script_body=None, res_file=None, apply=True, validate=True)

Add a server dynamic resource script or file to the scheduler configuration

Parameters:
  • custom_resource (str) – The name of the custom resource to define
  • script_body – The body of the server dynamic resource
  • res_file (str or None) – Alternatively to passing the script body, use the file instead
  • apply (bool) – if True (the default), apply configuration.
  • validate (bool) – if True (the default), validate the configuration settings.
unset_sched_config(name, apply=True)

Delete a sched_config entry

Parameters:
  • name (str) – the entry to delete from sched_config
  • apply (bool) – if True, apply configuration. Defaults to True
set_dedicated_time_file(file)

Set the path to a dedicated time

revert_to_defaults()

Revert scheduler configuration to defaults.

Returns:True on success, False otherwise
create_scheduler(sched_home=None)

Start scheduler with creating required directories for scheduler :param sched_home: path of scheduler home and log directory :type sched_home: str

save_configuration(outfile, mode='a')

Save scheduler configuration

Parameters:
  • outfile (str) – Path to a file to which configuration is saved
  • mode (str) – mode to use to access outfile. Defaults to append, ‘a’.
Returns:

True on success and False otherwise

load_configuration(infile)

load configuration from saved file infile

get_resources(exclude=[])

returns a list of allocatable resources.

Parameters:exclude (List) – if set, excludes the named resources, if they exist, from the resulting list
add_resource(name, apply=True)

Add a resource to sched_config.

Parameters:
  • name (str) – the resource name to add
  • apply (bool) – if True, apply configuration. Defaults to True
Returns:

True on success and False otherwise. Return True if the resource is already defined.

remove_resource(name, apply=True)

Remove a resource to sched_config.

Parameters:
  • name (str) – the resource name to remove
  • apply (bool) – if True, apply configuration. Defaults to True
Returns:

True on success and False otherwise

holidays_revert_to_default(level=20)

Revert holidays file to default

holidays_parse_file(path=None, obj=None, level=20)

Parse the existing holidays file

Parameters:
  • path (str or None) – optional path to the holidays file to parse
  • obj – optional holidays object to be used instead of internal
Returns:

The content of holidays file as a list of lines

holidays_set_day(day_id, prime='', nonprime='', apply=True, obj=None, level=20)

Set prime time values for a day

Parameters:
  • day_id (str) – the day to be set (string)
  • prime – the prime time value
  • nonprime – the non-prime time value
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
Returns:

The position (0-7) of the set day

holidays_get_day(day_id, obj=None, level=20)
Parameters:
  • obj – optional holidays object to be used instead of internal
  • day_id (str) – either a day’s name or “all”
Returns:

A copy of info about a day/all set days

holidays_reposition_day(day_id, new_pos, apply=True, obj=None, level=20)

Change position of a day (0-7) as it appears in the holidays file

Parameters:
  • day_id (str) – name of the day
  • new_pos – new position
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
Returns:

The new position of the day

holidays_unset_day(day_id, apply=True, obj=None, level=20)

Unset prime time values for a day

Parameters:
  • day_id (str) – day to unset (string)
  • apply – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal

Note

we do not unset the ‘valid’ field here so the entry will still be displayed but without any values

holidays_invalidate_day(day_id, apply=True, obj=None, level=20)

Remove a day’s entry from the holidays file

Parameters:
  • day_id (str) – the day to remove (string)
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
holidays_validate_day(day_id, apply=True, obj=None, level=20)

Make valid a previously set day’s entry

Parameters:
  • day_id (str) – the day to validate (string)
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal

Note

The day will retain its previous position in the file

holidays_delete_entry(entry_type, idx=None, apply=True, obj=None, level=20)

Delete one/all entries from holidays file

Parameters:
  • entry_type (str) – ‘y’:year, ‘d’:day, ‘h’:holiday or ‘a’: all
  • idx (str or None) – either a day of week (monday, tuesday etc.) or Julian date of a holiday
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
Returns:

False if entry_type is invalid, otherwise True

Note

The day cannot be validated and will lose it’s position in the file

holidays_set_year(new_year='', apply=True, obj=None, level=20)

Set the year value

Parameters:
  • newyear (str) – year value to set
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
holidays_unset_year(apply=True, obj=None, level=20)

Unset the year value

Parameters:
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
holidays_get_year(obj=None, level=20)
Parameters:obj – optional holidays object to be used instead of internal
Returns:The year entry of holidays file
holidays_add_holiday(date=None, apply=True, obj=None, level=20)

Add a calendar holiday to the holidays file

Parameters:
  • date – Date value for the holiday
  • apply (bool) – to reflect the changes to file
  • obj – optional holidays object to be used instead of internal
holidays_get_holidays(obj=None, level=20)
Parameters:obj – optional holidays object to be used instead of internal
Returns:The list of holidays in holidays file
holidays_write_file(content=None, out_path=None, hup=True, obj=None, level=20)

Write to the holidays file with content given/generated

Parameters:
  • hup (bool) – SIGHUP the scheduler after writing the holidays file
  • obj – optional holidays object to be used instead of internal
parse_dedicated_time(file=None)

Parse the dedicated_time file and populate dedicated times as both a string dedicated_time array of dictionaries defined as [{'from': datetime, 'to': datetime}, ...] as well as a dedicated_time_as_str array with a string representation of each entry

Parameters:file (str or None) – optional file to parse. Defaults to the one under PBS_HOME/sched_priv
Returns:The dedicated_time list of dictionaries or None on error.Return an empty array if dedicated time file is empty.
clear_dedicated_time(hup=True)

Clear the dedicated time file

add_dedicated_time(as_str=None, start=None, end=None, hup=True)

Append a dedicated time entry. The function can be called in one of two ways, either by passing in start and end as time values, or by passing as_str, a string that gets appended to the dedicated time entries and formatted as follows, note that no check on validity of the format will be made the function uses strftime to parse the datetime and will fail if the strftime can not convert the string. MM/DD/YYYY HH:MM MM/DD/YYYY HH:MM

Returns:True on success and False otherwise
valgrind()

run scheduler instance through valgrind

alloc_to_execvnode(chunks)

convert a resource allocation to an execvnode string representation

cycles(start=None, end=None, firstN=None, lastN=None)

Analyze scheduler log and return cycle information

Parameters:
  • start – Optional setting of the start time to consider
  • end – Optional setting of the end time to consider
  • firstN – Optional setting to consider the given first N cycles
  • lastN – Optional setting to consider only the given last N cycles
query_fairshare(name=None, id=None)

Parse fairshare data using pbsfs and populates fairshare_tree.If name or id are specified, return the data associated to that id.Otherwise return the entire fairshare tree

set_fairshare_usage(name=None, usage=None)

Set the fairshare usage associated to a given entity.

Parameters:
  • name (str or PbsUser or None) – The entity to set the fairshare usage of
  • usage – The usage value to set
decay_fairshare_tree()

Decay the fairshare tree through pbsfs

cmp_fairshare_entities(name1=None, name2=None)

Compare two fairshare entities. Wrapper of pbsfs -c e1 e2

Parameters:
  • name1 (str or PbsUser or None) – name of first entity to compare
  • name2 (str or PbsUser or None) – name of second entity to compare
Returns:

the name of the entity of higher priority or None on error

parse_resource_group(hostname=None, resource_group=None)

Parse the Scheduler’s resource_group file

Parameters:
  • hostname (str or None) – The name of the host from which to parse resource_group
  • resource_group (str or None) – The path to a resource_group file
Returns:

A fairshare tree

add_to_resource_group(name, id, parent, nshares)

Add an entry to the resource group file

Parameters:
  • name (str or PbsUser) – The name of the entity to add
  • id (int) – The numeric identifier of the entity to add
  • parent (str) – The name of the parent group
  • nshares (int) – The number of shares associated to the entity
job_formula(jobid=None, starttime=None, max_attempts=5)

Extract formula value out of scheduler log

Parameters:
  • jobid (str or int) – Optional, the job identifier for which to get the formula.
  • starttime – The time at which to start parsing the scheduler log
  • max_attempts (int) – The number of attempts to search for formula in the logs
Returns:

If jobid is specified, return the formula value associated to that job if no jobid is specified, returns a dictionary mapping job ids to formula

class ptl.lib.pbs_testlib.FairshareTree(hostname=None, resource_group=None)

Object representation of the Scheduler’s resource_group file and pbsfs data

Parameters:hostname (str) – Hostname of the machine
add_node(node, apply=True)

add node to the fairshare tree

create_node(name, id, parent_name, nshares)

Add an entry to the resource_group file

Parameters:
  • name (str) – The name of the entity to add
  • id (int) – The uniqe numeric identifier of the entity
  • parent (str) – The name of the parent/group of the entity
  • nshares (int) – The number of shares assigned to this entity
Returns:

True on success, False otherwise

get_node(name=None, id=None)

Return a node of the fairshare tree identified by either name or id.

Parameters:
  • name (str or None) – The name of the entity to query
  • id – The id of the entity to query
Returns:

The fairshare information of the entity when found, if not, returns None

Note

The name takes precedence over the id.

class ptl.lib.pbs_testlib.FairshareNode(name=None, id=None, parent_name=None, parent_id=None, nshares=None, usage='unknown', perc=None)

Object representation of the fairshare data as queryable through the command pbsfs.

Parameters:
  • name (str or None) – Name of fairshare node
  • nshares (int or None) – Number of shares
  • usage – Fairshare usage
  • perc – Percentage the entity has of the tree
class ptl.lib.pbs_testlib.MoM(name=None, attrs={}, pbsconf_file=None, diagmap={}, diag=None, server=None, db_access=None)

Container for MoM properties. Provides various MoM operations, such as creation, insertion, deletion of vnodes.

Parameters:
  • name (str or None) – The hostname of the server. Defaults to calling pbs_default()
  • attrs (Dictionary) – Dictionary of attributes to set, these will override defaults.
  • pbsconf_file (str or None) – path to config file to parse for PBS_HOME, PBS_EXEC, etc
  • diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
  • diag (str or None) – path to PBS diag directory (This will overrides diagmap)
  • server – A PBS server instance to which this mom is associated
  • db_acccess – set to either file containing credentials to DB access or dictionary containing {‘dbname’:...,’user’:...,’port’:...}
isUp()

Check for PBS mom up

signal(sig)

Send signal to PBS mom

get_pid()

Get the PBS mom pid

all_instance_pids()

Get all pids of a instance

start(args=None, launcher=None)

Start the PBS mom

Parameters:
  • args (str or None) – Arguments to start the mom
  • launcher (str or list or None) – Optional utility to invoke the launch of the service
stop(sig=None)

Stop the PBS mom

Parameters:sig (str) – Signal to stop the PBS mom
restart()

Restart the PBS mom

log_match(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, day=None, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)

Match given msg in given n lines of MoM log

Parameters:
  • msg (str) – log message to match, can be regex also when regexp is True
  • id (str) – The id of the object to trace. Only used for tracejob
  • n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
  • tail (bool) – If true (default), starts from the end of the file
  • allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
  • regexp (bool) – If true msg is a Python regular expression. Defaults to False
  • day (str) – Optional day in YYYMMDD format.
  • max_attempts (int) – the number of attempts to make to find a matching entry
  • interval (int) – the interval between attempts
  • starttime (int) – If set ignore matches that occur before specified time
  • endtime (int) – If set ignore matches that occur after specified time
  • level (int) – The logging level, defaults to INFO
  • existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns:

(x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.

Return type:

tuple

Raises:

PtlLogMatchError – When existence is True and given msg is not found in n line Or When existence is False and given msg found in n line.

Note

The matching line number is relative to the record number, not the absolute line number in the file.

pbs_version()

Get the PBS version

revert_to_defaults(delvnodedefs=True)
  1. Revert MoM configuration to defaults.
  2. Remove epilogue and prologue

3. Delete all vnode definitions HUP MoM

Parameters:delvnodedefs (bool) – if True (the default) delete all vnode definitions and restart the MoM
Returns:True on success and False otherwise
save_configuration(outfile, mode='a')

Save a MoM mom_priv/config

Parameters:
  • outfile (str) – the output file to which onfiguration is saved
  • mode (str) – the mode in which to open outfile to save configuration.
Returns:

True on success, False on error

Note

first object being saved should open this file with ‘w’ and subsequent calls from other objects should save with mode ‘a’ or ‘a+’. Defaults to a+

load_configuration(infile)

load configuration from saved file infile

is_cray()

Returns True if the version of PBS used was built for Cray platforms

is_cpuset_mom()

Check for cpuset mom

create_vnode_def(name, attrs={}, numnodes=1, sharednode=True, pre='[', post=']', usenatvnode=False, attrfunc=None, vnodes_per_host=1)

Create a vnode definition string representation

Parameters:
  • name (str) – The prefix for name of vnode to create, name of vnode will be prefix + pre + <num> + post
  • attrs (Dictionary) – Dictionary of attributes to set on each vnode
  • numnodes (int) – The number of vnodes to create
  • sharednode (bool) – If true vnodes are shared on a host
  • pre (str) – The symbol preceding the numeric value of that vnode.
  • post (str) – The symbol following the numeric value of that vnode.
  • usenatvnode (bool) – use the natural vnode as the first vnode to allocate this only makes sense starting with PBS 11.3 when natural vnodes are reported as a allocatable
  • attrfunc – function to customize the attributes, signature is (name, numnodes, curnodenum, attrs), must return a dict that contains new or modified attrs that will be added to the vnode def. The function is called once per vnode being created, it does not modify attrs itself across calls.
  • vnodes_per_host (int) – number of vnodes per host
Returns:

A string representation of the vnode definition file

parse_config()

Parse mom config file into a dictionary of configuration options.

Returns:A dictionary of configuration options on success, and None otherwise
add_config(conf={}, hup=True)

Add config options to mom_priv_config.

Parameters:
  • conf (Dictionary) – The configurations to add to mom_priv/config
  • hup (bool) – If True (default) HUP the MoM
Returns:

True on success and False otherwise

unset_mom_config(name, hup=True)

Delete a mom_config entry

Parameters:
  • name (String) – The entry to remove from mom_priv/config
  • hup (bool) – if True (default) HUP the MoM
Returns:

True on success and False otherwise

apply_config(conf={}, hup=True, restart=False)

Apply configuration options to MoM.

Parameters:
  • conf (Dictionary) – A dictionary of configuration options to apply to MoM
  • hup (bool) – If True (default) , HUP the MoM to apply the configuration
Returns:

True on success and False otherwise.

get_vnode_def(vnodefile=None)
Returns:A vnode def file as a single string
insert_vnode_def(vdef, fname=None, additive=False, restart=True)

Insert and enable a vnode definition. Root privilege is required

Parameters:
  • vdef (str) – The vnode definition string as created by create_vnode_def
  • fname (str or None) – The filename to write the vnode def string to
  • additive (bool) – If True, keep all other vnode def files under config.d Default is False
  • delete (bool) – If True, delete all nodes known to the server. Default is True
  • restart (bool) – If True, restart the MoM. Default is True
has_vnode_defs()

Check for vnode definition(s)

delete_vnode_defs(vdefname=None)

delete vnode definition(s) on this MoM

Parameters:vdefname (str) – name of a vnode definition file to delete, if None all vnode definitions are deleted
Returns:True if delete succeed otherwise False
has_pelog(filename=None)

Check for prologue and epilogue

has_prologue()

Check for prologue

has_epilogue()

Check for epilogue

delete_pelog()

Delete any prologue and epilogue files that may have been defined on this MoM

create_pelog(body=None, src=None, filename=None)

create prologue and epilogue files, functionality accepts either a body of the script or a source file.

Returns:True on success and False on error
prologue(body=None, src=None)

create prologue

epilogue(body=None, src=None)

Create epilogue

action(act, script)

Define action script. Not currently implemented

class ptl.lib.pbs_testlib.Hook(name=None, attrs={}, server=None)

PBS hook objects. Holds attributes information and pointer to server

Parameters:
  • name (str or None) – Hook name
  • attrs (Dictionary) – Hook attributes
  • server – Pointer to server
class ptl.lib.pbs_testlib.ResourceResv(name, attrs={}, defaults={})

Generic PBS resource reservation, i.e., job or advance/standing reservation

execvnode(attr='exec_vnode')

PBS type execution vnode

exechost()

PBS type execution host

resvnodes()

nodes assigned to a reservation

classmethod get_hosts(exechost=None)
Returns:The hosts portion of the exec_host
get_vnodes(execvnode=None)
Returns:The unique vnode names of an execvnode as a list
class ptl.lib.pbs_testlib.Job(username=None, attrs={}, jobname=None)

PBS Job. Attributes and Resources

Parameters:
  • username (str or None) – Job username
  • attrs (Dictionary) – Job attributes
  • jobname (str or None) – Name of the PBS job
add_cray_vntype(select=None)

Cray specific function to add vntype as cray_compute to each select chunk

Parameters:select (str or None) – PBS select statement
set_attributes(a={})

set attributes and custom attributes on this job. custom attributes are used when converting attributes to CLI. In case of Cray platform if ‘Resource_List.vntype’ is set already then remove it and add vntype value to each chunk of a select statement.

Parameters:a (Dictionary) – Attribute dictionary
set_variable_list(user=None, workdir=None)

Customize the Variable_List job attribute to <user>

set_sleep_time(duration)

Set the sleep duration for this job.

Parameters:duration (int) – The duration, in seconds, to sleep
set_execargs(executable, arguments=None)

Set the executable and arguments to use for this job

Parameters:
  • executable (str) – path to an executable. No checks are made.
  • arguments (str or list or int) – arguments to executable.
create_script(body=None, asuser=None, hostname=None)

Create a job script from a given body of text into a temporary location

Parameters:
  • body (str or None) – the body of the script
  • asuser (str or None) – Optionally the user to own this script, defaults ot current user
  • hostname (str or None) – The host on which the job script is to be created
create_subjob_id(job_array_id, subjob_index)

insert subjob index into the square brackets of job array id

Parameters:
  • job_array_id (str) – PBS parent array job id
  • subjob_index (int) – index of subjob
Returns:

subjob id string

class ptl.lib.pbs_testlib.Reservation(username=None, attrs={})

PBS Reservation. Attributes and Resources

Parameters:attrs (Dictionary) – Reservation attributes
class ptl.lib.pbs_testlib.InteractiveJob(job, cmd, host)

An Interactive Job thread

Interactive Jobs are submitted as a thread that sets the jobid as soon as it is returned by qsub -I, such that the caller can get back to monitoring the state of PBS while the interactive session goes on in the thread.

The commands to be run within an interactive session are specified in the job’s interactive_script attribute as a list of tuples, where the first item in each tuple is the command to run, and the subsequent items are the expected returned data.

Implementation details:

Support for interactive jobs is currently done through the pexpect module which must be installed separately from PTL. Interactive jobs are submitted through CLI only, there is no API support for this operation yet.

The submission of an interactive job requires passing in job attributes,the command to execute (i.e. path to qsub -I) and the hostname

when not impersonating:

pexpect spawns the qsub -I command and expects a prompt back, for each tuple in the interactive_script, it sends the command and expects to match the return value.

when impersonating:

pexpect spawns sudo -u <user> qsub -I. The rest is as described in non- impersonating mode.

run()

Run the interactive job

class ptl.lib.pbs_testlib.Queue(name=None, attrs={}, server=None)

PBS Queue container, holds attributes of the queue and pointer to server

Parameters:
  • name (str or None) – Queue name
  • attrs (Dictionary) – Queue attributes
revert_to_defaults()

reset queue attributes to defaults

class ptl.lib.pbs_testlib.PBSInitServices(hostname=None, conf=None)

PBS initialization services

Parameters:
  • hostname (str or None) – Machine hostname
  • conf (str or None) – PBS configuaration file
initd(hostname=None, op='status', conf_file=None, init_script=None, daemon='all')

Run the init script for a given operation

Parameters:
  • hostname (str or None) – hostname on which to execute the init script
  • op (str) – one of status, start, stop, restart
  • conf_file (str or None) – optional path to a configuration file
  • init_script (str or None) – optional path to a PBS init script
  • daemon (str) – name of daemon to operate on. one of server, mom, sched, comm or all
restart(hostname=None, init_script=None)

Run the init script for a restart operation

Parameters:
  • hostname (str or None) – hostname on which to execute the init script
  • init_script (str or None) – optional path to a PBS init script
restart_server(hostname=None, init_script=None)

Run the init script for a restart server

Parameters:
  • hostname (str or None) – hostname on which to restart server
  • init_script (str or None) – optional path to a PBS init script
restart_mom(hostname=None, init_script=None)

Run the init script for a restart mom

Parameters:
  • hostname (str or None) – hostname on which to restart mom
  • init_script (str or None) – optional path to a PBS init script
restart_sched(hostname=None, init_script=None)

Run the init script for a restart sched

Parameters:
  • hostname (str or None) – hostname on which to restart sched
  • init_script (str or None) – optional path to a PBS init script
restart_comm(hostname=None, init_script=None)

Run the init script for a restart comm

Parameters:
  • hostname (str or None) – hostname on which to restart comm
  • init_script (str or None) – optional path to a PBS init script
start(hostname=None, init_script=None)

Run the init script for a start operation

Parameters:
  • hostname (str or None) – hostname on which to execute the init script
  • init_script (str or None) – optional path to a PBS init script
start_server(hostname=None, init_script=None)

Run the init script for a start server

Parameters:
  • hostname (str or None) – hostname on which to start server
  • init_script (str or None) – optional path to a PBS init script
start_mom(hostname=None, init_script=None)

Run the init script for a start mom

Parameters:
  • hostname (str or None) – hostname on which to start mom
  • init_script (str or None) – optional path to a PBS init script
start_sched(hostname=None, init_script=None)

Run the init script for a start sched

Parameters:
  • hostname (str or None) – hostname on which to start sched
  • init_script (str or None) – optional path to a PBS init script
start_comm(hostname=None, init_script=None)

Run the init script for a start comm

Parameters:
  • hostname (str or None) – hostname on which to start comm
  • init_script (str or None) – optional path to a PBS init script
stop(hostname=None, init_script=None)

Run the init script for a stop operation

Parameters:
  • hostname (str or None) – hostname on which to execute the init script
  • init_script (str or None) – optional path to a PBS init script
stop_server(hostname=None, init_script=None)

Run the init script for a stop server

Parameters:
  • hostname (str or None) – hostname on which to stop server
  • init_script (str or None) – optional path to a PBS init script
stop_mom(hostname=None, init_script=None)

Run the init script for a stop mom

Parameters:
  • hostname (str or None) – hostname on which to stop mom
  • init_script (str or None) – optional path to a PBS init script
stop_sched(hostname=None, init_script=None)

Run the init script for a stop sched

Parameters:
  • hostname (str or None) – hostname on which to stop sched
  • init_script (str or None) – optional path to a PBS init script
stop_comm(hostname=None, init_script=None)

Run the init script for a stop comm

Parameters:
  • hostname (str or None) – hostname on which to stop comm
  • init_script (str or None) – optional path to a PBS init script
status(hostname=None, init_script=None)

Run the init script for a status operation

Parameters:
  • hostname (str or None) – hostname on which to execute the init script
  • init_script (str or None) – optional path to a PBS init script
status_server(hostname=None, init_script=None)

Run the init script for a status server

Parameters:
  • hostname (str or None) – hostname on which to status server
  • init_script (str or None) – optional path to a PBS init script
status_mom(hostname=None, init_script=None)

Run the init script for a status mom

Parameters:
  • hostname (str or None) – hostname on which to status mom
  • init_script (str or None) – optional path to a PBS init script
status_sched(hostname=None, init_script=None)

Run the init script for a status sched

Parameters:
  • hostname (str or None) – hostname on which to status sched
  • init_script (str or None) – optional path to a PBS init script
status_comm(hostname=None, init_script=None)

Run the init script for a status comm

Parameters:
  • hostname (str or None) – hostname on which to status comm
  • init_script (str or None) – optional path to a PBS init script
switch_version(hostname=None, version=None)

Switch to another version of PBS installed on the system

Parameters:
  • hostname (str or None) – The hostname to operate on
  • version – version to switch

1.2. Utilities

1.2.1. pbs_cliutils

class ptl.utils.pbs_cliutils.CliUtils

Command line interface utility

classmethod get_logging_level(level)

Get the logging levels

Parameters:level (str) – Name of the logging level
static check_bin(bin_name)

Check for command exist

Parameters:bin_name (str) – Command to be checked
Returns:True if command exist else return False
static expand_abs_path(path)

Expand the path to absolute path

static priv_ports_info(hostname=None)

Return a list of privileged ports in use on a given host

Parameters:hostname (str or None) – The host on which to query privilege ports usage. Defaults to the local host

1.2.2. pbs_covutils

class ptl.utils.pbs_covutils.LcovUtils(cov_bin=None, html_bin=None, cov_out=None, data_dir=None, html_nosrc=False, html_baseurl=None)

Coverage Utils

Parameters:
  • cov_bin – Coverage binary
  • html_bin – Coverage html binary
  • cov_out (str or None) – Coverage output directory
  • data_dir (str or None) – Coverage data directory
  • html_nosrc – HTML reports without PBS source
  • html_baseurl (str or None) – HTML base url
set_html_baseurl(baseurl)

Set HTML base url

set_html_nosource(nosource=False)

Set HTML nosource parameter to False.

set_coverage_bin(cov_bin=None)

Set the coverage binary

Parameters:cov_binary – Coverage bin to set
set_genhtml_bin(html_bin=None)

Set HTML generation utility.

Parameters:html_bin – HTML bin to set
set_coverage_out(cov_out=None)

Set the coverage output directory

Parameters:cov_out – Coverage output directory path.
set_coverage_data_dir(data=None)

Set the coverage data directory

Parameters:data – Data directory path
Returns:True if file name ends with .gcno else return False
add_trace(trace)

Add coverage trace

Parameters:trace – Coverage trace
create_coverage_data_files(path)

Create .gcda counterpart files for every .gcno file and give it read/write permissions

initialize_coverage(out=None, name=None)

Initialize coverage

Parameters:
  • out (str or None) – Output path
  • name (str or None) – name of the command
capture_coverage(out=None, name=None)

Capture the coverage parameters

zero_coverage()

Zero the data counters. Note that a process would need to be restarted in order to collect data again, running --initialize will not get populate the data counters

merge_coverage_traces(out=None, name=None, exclude=None)

Merge the coverage traces

generate_html(out=None, html_out=None, html_nosrc=False)

Generate the HTML report

change_baseurl(html_out=None, html_baseurl=None)

Change the HTML base url

summarize_coverage(out=None)

Summarize the coverage output

1.2.3. pbs_dshutils

exception ptl.utils.pbs_dshutils.PbsConfigError(message=None, rv=None, rc=None, msg=None)

Initialize PBS configuration error

exception ptl.utils.pbs_dshutils.PtlUtilError(message=None, rv=None, rc=None, msg=None)

Initialize PTL Util error

class ptl.utils.pbs_dshutils.DshUtils

PBS shell utilities

A set of tools to run commands, copy files, get process information and parse a PBS configuration on an arbitrary host

get_platform(hostname=None, pyexec=None)

Get a local or remote platform info, essentially the value of Python’s sys.platform, in case of Cray it will return a string as “cray” for actual Cray cluster and “craysim” for Cray ALPS simulator

Parameters:
  • hostname (str or None) – The hostname to query for platform info
  • pyexec (str or None) – A path to a Python interpreter to use to query a remote host for platform info

For efficiency the value is cached and retrieved from the cache upon subsequent request

get_os_info(hostname=None, pyexec=None)

Get a local or remote OS info

Parameters:
  • hostname (str or None) – The hostname to query for platform info
  • pyexec (str or None) – A path to a Python interpreter to use to query a remote host for platform info
Returns:

a ‘str’ object containing os info

get_pbs_conf_file(hostname=None)

Get the path of the pbs conf file. Defaults back to /etc/pbs.conf if unsuccessful

Parameters:hostname (str or None) – Hostname of the machine
Returns:Path to pbs conf file
parse_pbs_config(hostname=None, file=None)

initialize pbs_conf dictionary by parsing pbs config file

Parameters:file (str or None) – PBS conf file
set_pbs_config(hostname=None, fin=None, fout=None, append=True, confs=None)

Set environment/configuration variables in a pbs.conf file

Parameters:
  • hostname (str or None) – the name of the host on which to operate
  • fin (str or None) – the input pbs.conf file
  • fout (str or None) – the name of the output pbs.conf file, defaults to /etc/pbs.conf
  • append (boolean) – whether to append to fout or not, defaults to True
  • confs (Dictionary or None) – The key/value pairs to create
unset_pbs_config(hostname=None, fin=None, fout=None, confs=None)

Unset environment/configuration variables in a pbs.conf file

Parameters:
  • hostname (str or None) – the name of the host on which to operate
  • fin (str or None) – the input pbs.conf file
  • fout (str or None) – the name of the output pbs.conf file, defaults to /etc/pbs.conf
  • confs (List or str or dict or None) – The configuration keys to unset
get_pbs_server_name(pbs_conf=None)

Return the name of the server which may be different than PBS_SERVER,in order, this method looks at PBS_PRIMARY, PBS_SERVER_HOST_NAME, and PBS_LEAF_NAME, and PBS_SERVER

parse_pbs_environment(hostname=None, file='/var/spool/pbs/pbs_environment')

Initialize pbs_conf dictionary by parsing pbs config file

set_pbs_environment(hostname=None, fin='/var/spool/pbs/pbs_environment', fout=None, append=True, environ=None)

Set the PBS environment

Parameters:
  • environ (dict or None) – variables to set
  • hostname (str or None) – Hostname of the machine
  • fin (str) – pbs_environment input file
  • fout (str or None) – pbs_environment output file
  • append (bool) – whether to append to fout or not, defaults defaults to true
unset_pbs_environment(hostname=None, fin='/var/spool/pbs/pbs_environment', fout=None, environ=None)

Unset environment variables in a pbs_environment file

Parameters:
  • hostname (str or None) – the name of the host on which to operate
  • fin (str) – the input pbs_environment file
  • fout (str or None) – the name of the output pbs.conf file, defaults to /var/spool/pbs/pbs_environment
  • environ (List or str or dict or None) – The environment keys to unset
parse_rhosts(hostname=None, user=None)

Parse remote host

Parameters:
  • hostname (str or None) – Hostname of the machine
  • user (str or None) – User name
set_rhosts(hostname=None, user=None, entry={}, append=True)

Set the remote host attributes

Parameters:
  • entry (Dictionary) – remote hostname user dictionary
  • append (boolean) – If true append key value else not
map_pbs_conf_to_cmd(cmd_map={}, pconf={})

Map PBS configuration parameter to command

Parameters:
  • cmd_map (Dictionary) – command mapping
  • pconf (Dictionary) – PBS conf parameter dictionary
get_current_user()

helper function to return the name of the current user

check_user_exists(username=None, hostname=None)

Check if user exist or not

Parameters:
  • username (str or None) – Username to check
  • hostname (str or None) – Machine hostname
Returns:

True if exist else return False

check_group_membership(username=None, uid=None, grpname=None, gid=None)

Checks whether a user, passed in as username or uid, is a member of a group, passed in as group name or group id.

Parameters:
  • username (str or None) – The username to inquire about
  • uid – The uid of the user to inquire about (alternative to username)
  • grpname (str or None) – The groupname to check for user membership
  • gid – The group id to check for user membership (alternative to grpname)
group_memberships(group_list=[])

Returns all group memberships as a dictionary of group names and associated memberships

get_id_info(user)

Return user info in dic format obtained by "id -a <user>" command for given user

Parameters:user (str) – The username to inquire about
Returns:dic format:

{

“uid”: <uid of given user>,

“gid”: <gid of given user’s primary group>,

“name”: <name of given user>,

“pgroup”: <name of primary group of given user>,

“groups”: <list of names of groups of given user>

}

get_tempdir(hostname=None)
Returns:The temporary directory on the given host Default host is localhost.
run_cmd(hosts=None, cmd=None, sudo=False, stdin=None, stdout=-1, stderr=-1, input=None, cwd=None, env=None, runas=None, logerr=True, as_script=False, wait_on_script=True, level=18)

Run a command on a host or list of hosts.

Parameters:
  • hosts (str or None) – the name of hosts on which to run the command, can be a comma-separated string or a list. Defaults to localhost
  • cmd (str or None) – the command to run
  • sudo (boolean) – whether to run the command as root or not. Defaults to False.
  • stdin – custom stdin. Defaults to PIPE
  • stdout – custom stdout. Defaults to PIPE
  • stderr – custom stderr. Defaults to PIPE
  • input – input to pass to the pipe on target host, e.g. PBS answer file
  • cwd – working directory on local host from which command is run
  • env – environment variables to set on local host
  • runas – run command as given user. Defaults to calling user
  • logerr (boolean) – whether to log error messages or not. Defaults to True
  • as_script (boolean) – if True, run the command in a script created as a temporary file that gets deleted after being run. This is used mainly to circumvent some implementations of sudo that prevent passing environment variables through sudo.
  • wait_on_script (boolean) – If True (default) waits on process launched as script to return.
Returns:

error, output, return code as a dictionary: {'out':...,'err':...,'rc':...}

run_copy(hosts=None, src=None, dest=None, sudo=False, uid=None, gid=None, mode=None, env=None, logerr=True, recursive=False, runas=None, preserve_permission=True, level=18)

copy a file or directory to specified target hosts.

Parameters:
  • hosts (str or None) – the host(s) to which to copy the data. Can be a comma-separated string or a list
  • src (str or None) – the path to the file or directory to copy. If src is remote,it must be prefixed by the hostname. e.g. remote1:/path,remote2:/path
  • dest (str or None) – the destination path.
  • sudo (boolean) – whether to copy as root or not. Defaults to False
  • uid – optionally change ownership of dest to the specified user id,referenced by uid number or username
  • gid – optionally change ownership of dest to the specified group name/id
  • mode – optinoally set mode bits of dest
  • env – environment variables to set on the calling host
  • logerr – whether to log error messages or not. Defaults to True.
  • recursive (boolean) – whether to copy a directory (when true) or a file.Defaults to False.
  • runas (str or None) – run command as user
  • preserve_permission – Preserve file permission while copying file (cp cmd with -p flag) Defaults to True

:type preserve_permission:boolean :param level: logging level, defaults to DEBUG :type level: int :returns: {‘out’:<outdata>, ‘err’: <errdata>, ‘rc’:<retcode>}

upon and None if no source file specified
run_ptl_cmd(hostname, cmd, sudo=False, stdin=None, stdout=-1, stderr=-1, input=None, cwd=None, env=None, runas=None, logerr=True, as_script=False, wait_on_script=True, level=18)

Wrapper method of run_cmd to run PTL command

classmethod set_sudo_cmd(cmd)

set the sudo command

classmethod set_copy_cmd(cmd)

set the copy command

classmethod set_rsh_cmd(cmd)

set the remote shell command

is_localhost(host=None)
Parameters:host (str or None) – Hostname of machine
Returns:true if specified host (by name) is the localhost all aliases matching the hostname are searched
isdir(hostname=None, path=None, sudo=False, runas=None, level=18)
Parameters:
  • hostname (str or None) – The name of the host on which to check for directory
  • path (str or None) – The path to the directory to check
  • sudo (boolean) – Whether to run the command as a privileged user
  • runas (str or None) – run command as user
  • level – Logging level
Returns:

True if directory pointed to by path exists and False otherwise

isfile(hostname=None, path=None, sudo=False, runas=None, level=18)
Parameters:
  • hostname (str or None) – The name of the host on which to check for file
  • path (str or None) – The path to the file to check
  • sudo (boolean) – Whether to run the command as a privileged user
  • runas (str or None) – run command as user
  • level – Logging level
Returns:

True if file pointed to by path exists, and False otherwise

getmtime(hostname=None, path=None, sudo=False, runas=None, level=18)
Parameters:
  • hostname (str or None) – The name of the host on which file exists
  • path (str or None) – The path to the file to get mtime
  • sudo (boolean) – Whether to run the command as a privileged user
  • runas (str or None) – run command as user
  • level – Logging level
Returns:

Modified time of given file

listdir(hostname=None, path=None, sudo=False, runas=None, fullpath=True, level=18)
Parameters:
  • hostname (str or None) – The name of the host on which to list for directory
  • path (str or None) – The path to directory to list
  • sudo (bool) – Whether to chmod as root or not. Defaults to False
  • runas (str or None) – run command as user
  • fullpath (bool) – Return full paths?
  • level (int) – Logging level.
Returns:

A list containing the names of the entries in the directory

chmod(hostname=None, path=None, mode=None, sudo=False, runas=None, recursive=False, logerr=True, level=18)

Generic function of chmod with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • path (str or None) – the path to the file or directory to chmod
  • mode – mode to apply as octal number like 0777, 0666 etc.
  • sudo (boolean) – whether to chmod as root or not. Defaults to False
  • runas (str or None) – run command as user
  • recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
  • level – logging level, defaults to INFOCLI2
Returns:

True on success otherwise False

chown(hostname=None, path=None, uid=None, gid=None, sudo=False, recursive=False, runas=None, logerr=True, level=18)

Generic function of chown with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • path (str or None) – the path to the file or directory to chown
  • uid – uid to apply (must be either user name or uid or -1)
  • gid – gid to apply (must be either group name or gid or -1)
  • sudo (boolean) – whether to chown as root or not. Defaults to False
  • recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
  • runas (str or None) – run command as user
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
  • level – logging level, defaults to INFOCLI2
Returns:

True on success otherwise False

chgrp(hostname=None, path=None, gid=None, sudo=False, recursive=False, runas=None, logerr=True, level=18)

Generic function of chgrp with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • path (str or None) – the path to the file or directory to chown
  • gid – gid to apply (must be either group name or gid or -1)
  • sudo (boolean) – whether to chgrp as root or not. Defaults to False
  • recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
  • runas (str or None) – run command as user
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
  • level – logging level, defaults to INFOCLI2
Returns:

True on success otherwise False

which(hostname=None, exe=None, level=18)

Generic function of which with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • exe (str or None) – executable to locate (can be full path also) (if exe is full path then only basename will be used to locate)
  • level – logging level, defaults to INFOCLI2
rm(hostname=None, path=None, sudo=False, runas=None, recursive=False, force=False, cwd=None, logerr=True, as_script=False, level=18)

Generic function of rm with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • path (str or None) – the path to the files or directories to remove for more than one files or directories pass as list
  • sudo (boolean) – whether to remove files or directories as root or not.Defaults to False
  • runas – remove files or directories as given user Defaults to calling user
  • recursive (boolean) – remove files or directories and their contents recursively
  • force (boolean) – force remove files or directories
  • cwd – working directory on local host from which command is run
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
  • as_script (boolean) – if True, run the rm in a script created as a temporary file that gets deleted after being run. This is used mainly to handle wildcard in path list. Defaults to False.
  • level – logging level, defaults to INFOCLI2
Returns:

True on success otherwise False

mkdir(hostname=None, path=None, mode=None, sudo=False, runas=None, parents=True, cwd=None, logerr=True, as_script=False, level=18)

Generic function of mkdir with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • path (str or None) – the path to the directories to create for more than one directories pass as list
  • mode – mode to use while creating directories (must be octal like 0777)
  • sudo (boolean) – whether to create directories as root or not. Defaults to False
  • runas – create directories as given user. Defaults to calling user
  • parents (boolean) – create parent directories as needed. Defaults to True
  • cwd (str or None) – working directory on local host from which command is run
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
  • as_script (boolean) – if True, run the command in a script created as a temporary file that gets deleted after being run. This is used mainly to handle wildcard in path list. Defaults to False.
  • level – logging level, defaults to INFOCLI2
Returns:

True on success otherwise False

cat(hostname=None, filename=None, sudo=False, runas=None, logerr=True, level=18)

Generic function of cat with remote host support

Parameters:
  • hostname (str or None) – hostname (default current host)
  • filename (str or None) – the path to the filename to cat
  • sudo (boolean) – whether to create directories as root or not. Defaults to False
  • runas (str or None) – create directories as given user. Defaults to calling user
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
Returns:

output of run_cmd

cmp(hostname=None, fileA=None, fileB=None, sudo=False, runas=None, logerr=True)

Compare two files and return 0 if they are identical or non-zero if not

Parameters:
  • hostname (str or None) – the name of the host to operate on
  • fileA (str or None) – the first file to compare
  • fileB (str or None) – the file to compare fileA to
  • sudo (boolean) – run the command as a privileged user
  • runas (str or None) – run the cmp command as given user
  • logerr (boolean) – whether to log error messages or not. Defaults to True.
useradd(name, uid=None, gid=None, shell='/bin/bash', create_home_dir=True, home_dir=None, groups=None, logerr=True, level=18)

Add the user

Parameters:
  • name (str) – User name
  • shell – shell to use
  • create_home_dir (boolean) – If true then create home directory
  • home_dir (str or None) – path to home directory
  • groups – User groups
userdel(name, del_home=True, force=True, logerr=True, level=18)

Delete the user

Parameters:
  • del_home (boolean) – If true then delete user home
  • force (boolean) – If true then delete forcefully
groupadd(name, gid=None, logerr=True, level=18)

Add a group

create_temp_file(hostname=None, suffix='', prefix='PtlPbs', dirname=None, text=False, asuser=None, body=None, level=18)

Create a temp file by calling tempfile.mkstemp

Parameters:
  • hostname (str or None) – the hostname on which to query tempdir from
  • suffix (str) – the file name will end with this suffix
  • prefix (str) – the file name will begin with this prefix
  • dirname (str or None) – the file will be created in this directory
  • text (boolean) – the file is opened in text mode is this is true else in binary mode
  • asuser (str or None) – Optional username or uid of temp file owner
  • body (str or None) – Optional content to write to the temporary file
  • level (int) – logging level, defaults to INFOCLI2
mkdtemp(hostname=None, suffix='', prefix='PtlPbs', dir=None, uid=None, gid=None, mode=None, level=18)

Create a temp dir by calling tempfile.mkdtemp

Parameters:
  • hostname (str or None) – the hostname on which to query tempdir from
  • suffix (str) – the directory name will end with this suffix
  • prefix (str) – the directory name will begin with this prefix
  • dir (str or None) – the directory will be created in this directory
  • uid – Optional username or uid of temp directory owner
  • gid – Optional group name or gid of temp directory group owner
  • mode – Optional mode bits to assign to the temporary directory
  • level – logging level, defaults to INFOCLI2
parse_strace(lines)

strace parsing. Just the regular expressions for now

1.2.4. pbs_fileutils

class ptl.utils.pbs_fileutils.FileUtils(f, mode='head')

Utility to walk a file from 'head' or 'tail' on the local filesystem

Parameters:
  • f (str) – File to process
  • mode – One of FILE_HEAD or FILE_TAIL, which respectively set the file for processing from head or tail. Defaults to head.
get_file_descriptor()

Return the file descriptor associated to the file being processed

set_buf_size(bs=1024)

Set the buffer size to read blocks of file into

set_mode(m=None)
Parameters:mFILE_TAIL if file to be tailed, and FILE_HEAD to read from head
tell()
Returns:The current file 'cursor'
get_size()
Returns:The size of the file
get_num_lines()
Returns:No of lines for the file
next(n=1)

Get the next n lines of the file

Parameters:n (int) – the numer of lines to retrieve
get_line(n=1)
Returns:The nth line from file
get_block(from_n=1, to_n=1)
Returns:A block of lines between from_n and to_n
next_head(n=1)

Next line(s) from head

next_tail(n=1)

Next line(s) from tail

head(n=1)
Returns:n lines of head
tail(n=1)
Returns:n lines of tail

1.2.5. pbs_logutils

class ptl.utils.pbs_logutils.PBSLogUtils

Miscellaneous utilities to process log files

classmethod convert_date_time(datetime=None, fmt='%m/%d/%Y %H:%M:%S')

convert a date time string of the form given by fmt into number of seconds since epoch

Parameters:
  • datetime (str or None) – the datetime string to convert
  • fmt (str) – Format to which datetime is to be converted
Returns:

None if conversion fails

get_num_lines(log, hostname=None, sudo=False)

Get the number of lines of particular log

Parameters:log (str) – the log file name
open_log(log, hostname=None, sudo=False)
Parameters:
  • log (str) – the log file name to read from
  • hostname (str or None) – the hostname from which to read the file
  • sudo (boolean) – Whether to access log file as a privileged user.
Returns:

A file instance

get_timestamps(logfile=None, hostname=None, num=None)

Helper function to parse logfile

Returns:Each timestamp in a list as number of seconds since epoch
match_msg(lines, msg, allmatch=False, regexp=False, starttime=None, endtime=None)

Returns (x,y) where x is the matching line y, or None if nothing is found.

Parameters:
  • allmatch (boolean) – If True (False by default), return a list of matching tuples.
  • regexp (bool) – If True, msg is a Python regular expression. Defaults to False.
  • starttime – If set ignore matches that occur before specified time
  • endtime – If set ignore matches that occur after specified time
static convert_resv_date_time(date_time)

Convert reservation datetime to seconds

static convert_hhmmss_time(tm)

Convert datetime in hhmmss format to seconds

get_rate(l=[])
Returns:The frequency of occurrences of array l The array is expected to be sorted
in_range(tm, start=None, end=None)
Parameters:
  • tm – time to check within a provided range
  • start – Lower limit for the time range
  • end – Higer limit for the time range
Returns:

True if time is in the range else return False

static get_day(tm=None)
Parameters:tm – Time for which to get a day
static percentile(N, percent)

Find the percentile of a list of values.

Parameters:
  • N (List) – A list of values. Note N MUST BE already sorted.
  • percent (Float) – A float value from 0.0 to 1.0.
Returns:

The percentile of the values

static process_intervals(intervals, groups, frequency=60)

Process the intervals

get_log_files(hostname, path, start, end, sudo=False)
Parameters:
  • hostname (str) – Hostname of the machine
  • path (str) – Path for the log file
  • start – Start time for the log file
  • end – End time for the log file
Returns:

list of log file(s) found or an empty list

class ptl.utils.pbs_logutils.PBSLogAnalyzer(schedlog=None, serverlog=None, momlog=None, acctlog=None, genericlog=None, hostname=None, show_progress=False)

Utility to analyze the PBS logs

set_custom_match(pattern, frequency=None)

Set the custome matching

Parameters:
  • pattern (str) – Matching pattern
  • frequency (int) – Frequency of match
set_conditional_match(conditions)

Set the conditional match

Parameters:conditions – Conditions for macthing
analyze_scheduler_log(filename=None, start=None, end=None, hostname=None, summarize=True)

Analyze the scheduler log

Parameters:
  • filename (str or None) – Scheduler log file name
  • start – Time from which log to be analyzed
  • end – Time till which log to be analyzed
  • hostname (str or None) – Hostname of the machine
  • summarize (bool) – Summarize data parsed if True else not
analyze_server_log(filename=None, start=None, end=None, hostname=None, summarize=True)

Analyze the server log

analyze_accounting_log(filename=None, start=None, end=None, hostname=None, summarize=True)

Analyze the accounting log

analyze_mom_log(filename=None, start=None, end=None, hostname=None, summarize=True)

Analyze the mom log

parse_conditional(rec, start, end)

Match a sequence of regular expressions against multiple consecutive lines in a generic log. Calculate the number of conditional matching lines.

Example usage: to find the number of times the scheduler stat’ing the server causes the scheduler to miss jobs ending, which could possibly indicate a race condition between the view of resources assigned to nodes and the actual jobs running, one would call this function by setting re_conditional to ['Type 20 request received from Scheduler', 'Exit_status'] Which can be read as counting the number of times that the Type 20 message is preceded by an Exit_status message

analyze(path=None, start=None, end=None, hostname=None, summarize=True, sudo=False)

Parse any log file. This method is not context-specific to each log file type.

Parameters:
  • path (str or None) – name of file/dir to parse
  • start – optional record time at which to start analyzing
  • end – optional record time after which to stop analyzing
  • hostname (str or None) – name of host on which to operate. Defaults to localhost
  • summarize (bool) – if True, summarize data parsed. Defaults to True.
  • sudo (bool) – If True, access log file(s) as privileged user.
analyze_logs(schedlog=None, serverlog=None, momlog=None, acctlog=None, genericlog=None, start=None, end=None, hostname=None, showjob=False)

Analyze logs

class ptl.utils.pbs_logutils.PBSServerLog(filename=None, hostname=None, show_progress=False)
Parameters:
  • filename (str or None) – Server log filename
  • hostname (str or None) – Hostname of the machine
parse_runjob(line)

Parse server log for run job records. For each record keep track of the job id, and time in a dedicated array

parse_endjob(line)

Parse server log for run job records. For each record keep track of the job id, and time in a dedicated array

parse_nodeup(line)

Parse server log for nodes that are up

parse_enquejob(line)

Parse server log for enqued jobs

class ptl.utils.pbs_logutils.JobEstimatedStartTimeInfo(jobid)

Information regarding Job estimated start time

add_estimate(tm)

Add a job’s new estimated start time If the new estimate is now later than any preivous one, we add that difference to the drift time. If the new drift time is pulled earlier it is not added to the drift time.

drift time is a measure of "negative perception" that comes along a job being estimated to run at a later date than earlier "advertised".

class ptl.utils.pbs_logutils.PBSMoMLog(filename=None, hostname=None, show_progress=False)

Container and Parser of a PBS MoM log

summary()

Mom log summary

class ptl.utils.pbs_logutils.PBSAccountingLog(filename=None, hostname=None, show_progress=False)

Container and Parser of a PBS accounting log

enable_running_jobs_parsing()

Enable parsing for running jobs

enable_utilization_parsing(hostname=None, nodesfile=None, jobsfile=None)

Enable utilization parsing

Parameters:
  • hostname (str or None) – Hostname of the machine
  • nodesfile (str or None) – optional file containing output of pbsnodes -av
  • jobsfile (str or None) – optional file containing output of qstat -f
enable_job_info_parsing()

Enable job information parsing

enable_accounting_workload_parsing()

Enable accounting workload parsing

process_nodes_data(hostname=None, nodesfile=None, jobsfile=None)

Get job and node information by stat’ing and parsing node data from the server. Compute the number of nodes and populate a list of running job ids on those nodes.

Parameters:
  • hostname (str or None) – The host to query
  • nodesfile (str or None) – optional file containing output of pbsnodes -av
  • jobsfile (str or None) – optional file containing output of qstat -f

The node data is needed to compute counts of nodes and cpus The job data is needed to compute the amount of resources requested

accounting_parsing(rec, start, end)

Parsing accounting log

job_info(rec)

PBS Job information

finished_jobs_nodes(last=None, tm_range=None)
Parameters:
  • tm_range – a tuple of time where the first item is the start time
  • last – If tm_range is None and last is specified, a time range from now till ‘last’ seconds from now is used as a range of time to consider and the second item is the end time to consider
Returns:

A dictionary of jobs that ended in the time range as keys,and nodes (hostnames) on which those jobs were running as values.

summary()

Accounting log summary

1.2.6. pbs_procutils

class ptl.utils.pbs_procutils.ProcUtils

Utilities to query process information

get_ps_cmd(hostname=None)

Get the ps command

Parameters:hostname (str or None) – hostname of the machine
get_proc_info(hostname=None, name=None, pid=None, regexp=False)

Return process information from a process name, or pid, on a given host

Parameters:
  • hostname (str or none) – The hostname on which to query the process info. On Windows,only localhost is queried.
  • name (str or None) – The name of the process to query.
  • pid (int or None) – The pid of the process to query
  • regexp (bool) – Match processes by regular expression. Defaults to True. Does not apply to matching by PID.
Returns:

A list of ProcInfo objects, one for each matching process.

Note

If both, name and pid, are specified, name is used.

get_proc_state(hostname=None, pid=None)
Returns:PID’s process state on host hostname

On error the empty string is returned.

get_proc_children(hostname=None, ppid=None)
Returns:A list of children PIDs associated to PPID on host hostname.

On error, an empty list is returned.

class ptl.utils.pbs_procutils.ProcInfo(name=None, pid=None)

Process information reports PID, RSS, VSZ, Command and Time at which process information is collected

class ptl.utils.pbs_procutils.ProcMonitor(name=None, regexp=False, frequency=60)

A background process monitoring tool

set_frequency(value=60)

Set the frequency

Parameters:value (int) – Frequency value
run()

Run the process monitoring

stop()

Stop the process monitoring

1.2.7. pbs_testsuite

ptl.utils.pbs_testsuite.skip(reason='Skipped test execution')

Unconditionally skip a test.

Parameters:reason (str or None) – Reason for the skip
ptl.utils.pbs_testsuite.timeout(val)

Decorator to set timeout value of test case

ptl.utils.pbs_testsuite.checkModule(modname)

Decorator to check if named module is available on the system and if not skip the test

ptl.utils.pbs_testsuite.skipOnCray(function)

Decorator to skip a test on a Cray system

ptl.utils.pbs_testsuite.skipOnCpuSet(function)

Decorator to skip a test on a CpuSet system

class ptl.utils.pbs_testsuite.PBSServiceInstanceWrapper(*args, **kwargs)

A wrapper class to handle multiple service (i.e., mom, server, scheduler)``instances as passed along through the test harness ``(pbs_benchpress).Returns an ordered dictionary of PBS service instances (i.e., mom/server/ scheduler)

Users may invoke PTL using pointers to multiple services per host, for example:

pbs_benchpress -p moms=hostA@/etc/pbs.conf,hostB,hostA@/etc/pbs.conf3

In such cases, the moms instance variable must be able to distinguish the self.moms['hostA'] instances, each instance will be mapped to a unique configuration file

class ptl.utils.pbs_testsuite.PBSTestSuite(methodName='runTest')

Generic setup, teardown, and logging functions to be used as parent class for most tests. Class instantiates:

server object connected to localhost

scheduler objected connected to localhost

mom object connected to localhost

Custom parameters:

Parameters:
  • server – The hostname on which the PBS server/scheduler are running
  • mom – The hostname on which the PBS MoM is running
  • servers – Colon-separated list of hostnames hosting a PBS server. Servers are then accessible as a dictionary in the instance variable servers.
  • client – For CLI mode only, name of the host on which the PBS client commands are to be run from. Format is <host>@<path-to-config-file>
  • moms – Colon-separated list of hostnames hosting a PBS MoM. MoMs are made accessible as a dictionary in the instance variable moms.
  • comms – Colon-separated list of hostnames hosting a PBS Comm. Comms are made accessible as a dictionary in the instance variable comms.
  • nomom=<host1>:<host2>... – expect no MoM on given set of hosts
  • mode – Sets mode of operation to PBS server. Can be either 'cli' or 'api'.Defaults to API behavior.
  • conn_timeout – set a timeout in seconds after which a pbs_connect IFL call is refreshed (i.e., disconnected)
  • skip-setup – Bypasses setUp of PBSTestSuite (not custom ones)
  • skip-teardown – Bypasses tearDown of PBSTestSuite (not custom ones)
  • procinfo – Enables process monitoring thread, logged into ptl_proc_info test metrics. The value can be set to _all_ to monitor all PBS processes,including pbs_server, pbs_sched, pbs_mom, or a process defined by name.
  • revert-to-defaults=<True|False> – if False, will not revert to defaults.True by default.
  • revert-hooks=<True|False> – if False, do not revert hooks to defaults.Defaults to True. revert-to-defaults set to False overrides this setting.
  • del-hooks=<True|False> – If False, do not delete hooks. Defaults to False.``revert-to-defaults`` set to False overrides this setting.
  • revert-queues=<True|False> – If False, do not revert queues to defaults.Defaults to True. revert-to-defaults set to False overrides this setting.
  • revert-resources=<True|False> – If False, do not revert resources to defaults. Defaults to True. revert-to-defaults set to False overrides this setting.
  • del-queues=<True|False> – If False, do not delete queues. Defaults to False.``revert-to-defaults`` set to Falseoverrides this setting.
  • del-vnodes=<True|False> – If False, do not delete vnodes on MoM instances.Defaults to True.
  • server-revert-to-defaults=<True|False> – if False, don’t revert Server to defaults
  • comm-revert-to-defaults=<True|False> – if False, don’t revert Comm to defaults
  • mom-revert-to-defaults=<True|False> – if False, don’t revert MoM to defaults
  • sched-revert-to-defaults=<True|False> – if False, don’t revert Scheduler to defaults
  • procmon – Enables process monitoring. Multiple values must be colon separated. For example to monitor server, sched, and mom use procmon=pbs_server:pbs_sched:pbs_mom
  • procmon-freq – Sets a polling frequency for the process monitoring tool.Defaults to 10 seconds.
  • test-users – colon-separated list of users to use as test users. The users specified override the default users in the order in which they appear in the PBS_USERS list.
  • default-testcase-timeout – Default test case timeout value.
  • data-users – colon-separated list of data users.
  • oper-users – colon-separated list of operator users.
  • mgr-users – colon-separated list of manager users.
  • root-users – colon-separated list of root users.
  • build-users – colon-separated list of build users.
  • clienthost – the hostnames to set in the MoM config file
classmethod check_users_exist()

Check whether the user is exist or not

classmethod kicksched_action(server, obj_type, *args, **kwargs)

custom scheduler action to kick a scheduling cycle when expectig a job state change

classmethod parse_param()

get test configuration parameters as a comma-separated list of attributes.

Attributes may be '=' separated key value pairs or standalone entries.

Multi-property attributes are colon-delimited.

classmethod is_server_licensed(server)

Check if server is licensed or not

classmethod init_from_conf(conf, single=None, multiple=None, skip=None, func=None)

Helper method to parse test parameters for`` mom/server/scheduler`` instances.

The supported format of each service request is:

hostname@configuration/path

For example:

pbs_benchpress -p server=remote@/etc/pbs.conf.12.0

initializes a remote server instance that is configured according to the remote file /etc/pbs.conf.12.0

classmethod init_servers(init_server_func=None, skip=None)

Initialize servers

classmethod init_comms(init_comm_func=None, skip=None)

Initialize comms

classmethod init_schedulers(init_sched_func=None, skip=None)

Initialize schedulers

classmethod init_moms(init_mom_func=None, skip='nomom')

Initialize moms

classmethod init_server(hostname, pbsconf_file=None)

Initialize a server instance

Define custom expect action to trigger a scheduling cycle when job is not in running state

Returns:The server instance on success and None on failure
classmethod init_comm(hostname, pbsconf_file=None, server=None)

Initialize a Comm instance associated to the given hostname.

This method must be called after init_server

Parameters:
  • hostname (str) – The host on which the Comm is running
  • pbsconf_file (str or None) – Optional path to an alternate pbs config file
  • server (str) – The server name associated to the Comm
Returns:

The instantiated Comm upon success and None on failure.

Return the instantiated Comm upon success and None on failure.

classmethod init_scheduler(hostname, pbsconf_file=None, server=None)

Initialize a Scheduler instance associated to the given server. This method must be called after init_server

Parameters:
  • server (str) – The server name associated to the scheduler
  • pbsconf_file (str or None) – Optional path to an alternate config file
  • hostname (str) – The host on which Sched is running
Returns:

The instantiated scheduler upon success and None on failure

classmethod init_mom(hostname, pbsconf_file=None, server=None)

Initialize a MoM instance associated to the given hostname.

This method must be called after init_server

Parameters:
  • hostname (str) – The host on which the MoM is running
  • pbsconf_file (str or None) – Optional path to an alternate pbs config file
Returns:

The instantiated MoM upon success and None on failure.

init_proc_mon()

Initialize process monitoring when requested

revert_servers(force=False)

Revert the values set for servers

revert_comms(force=False)

Revert the values set for comms

revert_schedulers(force=False)

Revert the values set for schedulers

revert_moms(force=False)

Revert the values set for moms

revert_server(server, force=False)

Revert the values set for server

revert_comm(comm, force=False)

Revert the values set for comm

revert_scheduler(scheduler, force=False)

Revert the values set for scheduler

revert_mom(mom, force=False)

Revert the values set for mom

Parameters:
  • mom (MoM object) – the MoM object whose values are to be reverted
  • force (bool) – Option to reverse forcibly
analyze_logs()

analyze accounting and scheduler logs from time test was started until it finished

start_proc_monitor(name=None, regexp=False, frequency=60)

Start the process monitoring

Parameters:
  • name (str or None) – Process name
  • regexp (bool) – Regular expression to match
  • frequency (int) – Frequency of monitoring
stop_proc_monitor()

Stop the process monitoring

skipTest(reason=None)

Skip Test

Parameters:reason (str or None) – message to indicate why test is skipped
skip_test(reason=None)

Skip Test

Parameters:reason (str or None) – message to indicate why test is skipped
tearDown()

verify that server and scheduler are up clean up jobs and reservations

1.2.8. Plugins

1.2.8.1. ptl_test_data

class ptl.utils.plugins.ptl_test_data.PTLTestData

Save post analysis data on test cases failure or error

options(parser, env)

Register command line options

configure(options, config)

Configure the plugin and system, based on selected options

1.2.8.2. ptl_test_db

exception ptl.utils.plugins.ptl_test_db.PTLDbError(rv=None, rc=None, msg=None, post=None, *args, **kwargs)

PTL database error class

Parameters:
  • rv (str or None) – Return value for the database error
  • rc (str or None) – Return code for the database error
  • msg (str or None) – Error message
class ptl.utils.plugins.ptl_test_db.DBType(dbtype, dbpath, dbaccess)

Base class for each database type Any type of database must inherit from me

Parameters:
  • dbtype (str) – Database type
  • dbpath (str) – Path to database
  • dbaccess (str) – Path to a file that defines db options
write(data, logfile=None)
Parameters:
  • data – Data to write
  • logfile (str or None) – Can be one of server, scheduler, mom, accounting or procs
close()

Close the database

class ptl.utils.plugins.ptl_test_db.PostgreSQLDb(dbtype, dbpath, dbaccess)

PostgreSQL type database

class ptl.utils.plugins.ptl_test_db.SQLiteDb(dbtype, dbpath, dbaccess)

SQLite type database

class ptl.utils.plugins.ptl_test_db.FileDb(dbtype, dbpath, dbaccess)

File type database

class ptl.utils.plugins.ptl_test_db.HTMLDb(dbtype, dbpath, dbaccess)

HTML type database

class ptl.utils.plugins.ptl_test_db.PTLTestDb

PTL Test Database Plugin

options(parser, env)

Register command line options

set_data(dbtype, dbpath, dbaccess)

Set the data

configure(options, config)

Configure the plugin and system, based on selected options

Parameters:options – Configuration options for plugin and system
process_output(info={}, dbout=None, dbtype=None, dbaccess=None, name=None, logtype=None, summary=False)

Send analyzed log information to either the screen or to a database file.

Parameters:
  • info (Dictionary) – A dictionary of log analysis metrics.
  • dbout (str or None) – The name of the database file to send output to
  • dbtype – Type of database
  • dbaccess – Path to a file that defines db options (PostreSQL only)
  • name (str or None) – The name of the log file being analyzed
  • logtype – The log type, one of accounting, schedsummary, scheduler, server, or mom
  • summary – If True output summary only

1.2.8.3. ptl_test_info

class ptl.utils.plugins.ptl_test_info.PTLTestInfo

Load test cases from given parameter

options(parser, env)

Register command line options

set_data(suites, list_test, showinfo, verbose, gen_ts_tree)

Set the data required for running the tests

Parameters:
  • suites – Test suites to run
  • list_test – List of test to run
  • gen_ts_tree – Generate test suite tree
configure(options, config)

Configure the plugin and system, based on selected options

Parameters:options – Options to configure plugin and system
wantClass(cls)

Is the class wanted?

wantFunction(function)

Is the function wanted?

wantMethod(method)

Is the method wanted?

1.2.8.4. ptl_test_loader

class ptl.utils.plugins.ptl_test_loader.PTLTestLoader

Load test cases from given parameter

options(parser, env)

Register command line options

set_data(testgroup, suites, excludes, follow)

Set the data required for loading test data

Parameters:
  • testgroup – Test group
  • suites – Test suites to load
  • excludes – Tests to exclude while running
configure(options, config)

Configure the plugin and system, based on selected options

check_unknown()

Check for unknown test suite and test case

prepareTestLoader(loader)

Prepare test loader

is_already_allowed(cls, method=None)
Parameters:method – Method to check
Returns:True if method is already allowed else False
wantClass(cls)

Is the class wanted?

wantFunction(function)

Is the function wanted?

wantMethod(method)

Is the method wanted?

1.2.8.5. ptl_test_runner

exception ptl.utils.plugins.ptl_test_runner.TimeOut

Raise this exception to mark a test as timed out.

exception ptl.utils.plugins.ptl_test_runner.TCThresholdReached

Raise this exception to tell that tc-failure-threshold reached

class ptl.utils.plugins.ptl_test_runner.TestLogCaptureHandler

Log handler for capturing logs which test case print using logging module

class ptl.utils.plugins.ptl_test_runner.PtlTestRunner(stream=<open file '<stdout>', mode 'w'>, descriptions=True, verbosity=3, config=None)

Test runner that uses PtlTestResult to enable errorClasses, as well as providing hooks for plugins to override or replace the test output stream, results, and the test case itself.

run(test)

Overrides to provide plugin hooks and defer all output to the test result class.

class ptl.utils.plugins.ptl_test_runner.PTLTestRunner

PTL Test Runner Plugin

options(parser, env)

Register command line options

configure(options, config)

Configure the plugin and system, based on selected options

prepareTestRunner(runner)

Prepare test runner

prepareTestResult(result)

Prepare test result

startTest(test)

Start the test

stopTest(test)

Stop the test

addError(test, err)

Add error

addFailure(test, err)

Add failure

addSuccess(test)

Add success

1.2.8.6. ptl_test_tags

ptl.utils.plugins.ptl_test_tags.tags(*args, **kwargs)

Decorator that adds tags to classes or functions or methods

ptl.utils.plugins.ptl_test_tags.get_tag_value(method, cls, tag_name, default=False)

Look up an tag on a method/function. If the tag isn’t found there, looking it up in the method’s class, if any.

class ptl.utils.plugins.ptl_test_tags.EvalHelper(method, cls)

Object that can act as context dictionary for eval and looks up names as attributes on a method/function and its class.

class ptl.utils.plugins.ptl_test_tags.PTLTestTags

Load test cases from given parameter

options(parser, env)

Register command line options

configure(options, config)

Configure the plugin and system, based on selected options.

attr and eval_attr may each be lists.

self.attribs will be a list of lists of tuples. In that list, each list is a group of attributes, all of which must match for the rule to match.

is_tags_matching(method, cls=None)

Verify whether a method has the required tags The method is considered a match if it matches all tags for any tag group.

prepareTestRunner(runner)

Prepare test runner

wantClass(cls)

Accept the class if its subclass of TestCase and has at-least one test case

wantFunction(function)

Accept the function if its tags match.

wantMethod(method)

Accept the method if its tags match.