1. API documentation¶
1.1. Library¶
1.1.1. pbs_api_to_cli¶
1.1.2. pbs_ifl_mock¶
1.1.3. pbs_testlib¶
-
class
ptl.lib.pbs_testlib.
PtlConfig
(conf=None)¶ Holds configuration options The options can be stored in a file as well as in the OS environment variables.When set, the environment variables will override definitions in the file.By default, on Unix like systems, the file read is
/etc/ptl.conf
, the environment variablePTL_CONF_FILE
can be used to set the path to the file to read.The format of the file is a series of
<key> = <value>
properties. A line that starts with a ‘#’ is ignored and can be used for commentsParameters: conf (str or None) – Path to PTL configuration file
-
exception
ptl.lib.pbs_testlib.
PtlException
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶ Generic errors raised by PTL operations. Sets a
return value
, areturn code
, and amessage
A post function and associated positional and named arguments are available to perform any necessary cleanup.Parameters: - rv (int or None.) – Return value set for the error occured during PTL operation
- rc (int or None.) – Return code set for the error occured during PTL operation
- msg (str or None.) – Message set for the error occured during PTL operation
- post (callable or None.) – Execute given post callable function if not None.
Raises: PTL exceptions
-
exception
ptl.lib.pbs_testlib.
PtlFailureException
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶ Generic failure exception raised by PTL operations. Sets a
return value
, areturn code
, and amessage
A post function and associated positional and named arguments are available to perform any necessary cleanup.Parameters: - rv (int or None.) – Return value set for the failure occured during PTL operation
- rc (int or None.) – Return code set for the failure occured during PTL operation
- msg (str or None.) – Message set for the failure occured during PTL operation
- post (callable or None.) – Execute given post callable function if not None.
Raises: PTL exceptions
-
exception
ptl.lib.pbs_testlib.
PbsServiceError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsConnectError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsStatusError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsSubmitError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsManagerError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsDeljobError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsDelresvError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsDeleteError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsRunError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsSignalError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsMessageError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsHoldError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsReleaseError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsOrderError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsRerunError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsMoveError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsAlterError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsResourceError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsSelectError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsSchedConfigError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsMomConfigError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsQdisableError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsQenableError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsQstartError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsQstopError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PtlExpectError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsInitServicesError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsQtermError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PtlLogMatchError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
exception
ptl.lib.pbs_testlib.
PbsResvAlterError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶
-
class
ptl.lib.pbs_testlib.
PbsTypeSize
(value=None)¶ Descriptor class for memory as a numeric entity. Units can be one of
b
,kb
,mb
,gb
,tb
,pt
Parameters: - unit (str) – The unit type associated to the memory value
- value (int or None) – The numeric value of the memory
Raises: ValueError and TypeError
-
encode
(value=None, valtype='kb', precision=1)¶ Encode numeric memory input in kilobytes to a string, including unit
Parameters: - value (int or None.) – The numeric value of memory to encode
- valtype (str) – The unit of the input value, defaults to kb
- precision (int) – Precision of the encoded value, defaults to 1
Returns: Encoded memory in kb to string
-
class
ptl.lib.pbs_testlib.
PbsTypeDuration
(val)¶ Descriptor class for a duration represented as
hours
,minutes
, andseconds
,in the form of[HH:][MM:]SS
Parameters: - as_seconds (int) – HH:MM:SS represented in seconds
- as_str (str) – duration represented in HH:MM:SS
-
class
ptl.lib.pbs_testlib.
PbsTypeArray
(value=None, sep=', ')¶ Descriptor class for a PBS array list type, e.g. String array
Parameters: - value – Array value to be passed
- sep (str) – Separator for two array elements
Returns: List
-
class
ptl.lib.pbs_testlib.
PbsTypeList
(value=None, sep=', ', kvsep='=')¶ Descriptor class for a generic PBS list that are key/value pairs delimited
Parameters: - value – List value to be passed
- sep (str) – Separator for two key/value pair
- kvsep (str) – Separator for key and value
Returns: Dictionary
-
class
ptl.lib.pbs_testlib.
PbsTypeLicenseCount
(value=None)¶ Descriptor class for a PBS license_count attribute.
It is a specialized list where key/values are ‘:’ delimited, separated by a ‘ ‘ (space)
Parameters: value – PBS license_count attribute value Returns: Specialized list
-
class
ptl.lib.pbs_testlib.
PbsTypeVariableList
(value=None)¶ Descriptor class for a PBS Variable_List attribute
It is a specialized list where key/values are ‘=’ delimited, separated by a ‘,’ (space)
Parameters: value – PBS Variable_List attribute value Returns: Specialized list
-
class
ptl.lib.pbs_testlib.
PbsTypeSelect
(s=None)¶ Descriptor class for PBS select/schedselect specification. Select is of the form:
<select> ::= <m>":"<chunk> | <select>"+"<select>
<m> ::= <digit> | <digit><m>
<chunk> ::= <resc_name>":"<resc_value> | <chunk>":"<chunk>
<m>
is a multiplying factor for each chunk requested<chunk>
are resource key/value pairsThe type populates a list of single chunk of resource
key/value
pairs, the list can be walked by iterating over the type itself.Parameters: - num_chunks – The total number of chunks in the select
- resources (Dictionary) – A dictionary of all resource counts in the select
-
class
ptl.lib.pbs_testlib.
PbsTypeChunk
(vnode=None, resources=None, chunkstr=None)¶ Descriptor class for a PBS chunk associated to a
PbsTypeExecVnode
.This type of chunk corresponds to a node solution to a resource request,not to the select specification.chunk ::= <subchk> | <chunk>"+"<chunk>
subchk ::= <node>":"<resource>
resource ::= <key>":"<val> | <resource>":"<resource>
A chunk expresses a solution to a specific select-chunk request. If multiple chunks are needed to solve a single select-chunk, e.g., on a shared memory system, the chunk will be extended into virtual chunk,vchunk.
Parameters: - vnode (str or None) – the vnode name corresponding to the chunk
- resources (Dictionary or None) – the key value pair of resources in dictionary form
- vchunk (list) – a list of virtual chunks needed to solve the select-chunk, vchunk is only set if more than one vchunk are required to solve the select-chunk
-
add
(vnode, resources)¶ Add a chunk specificiation. If a chunk is already defined, add the chunk as a vchunk.
Parameters: - vnode (str) – The vnode to add
- resources (str) – The resources associated to the vnode
Returns: Added chunk specification
-
class
ptl.lib.pbs_testlib.
PbsTypeExecVnode
(s=None)¶ Execvnode representation, expressed as a list of PbsTypeChunk
Parameters: - vchunk (List) – List of virtual chunks, only set when more than one vnode is allocated to a host satisfy a chunk requested
- num_chunks (int) – The number of chunks satisfied by this execvnode
- vnodes (List) – List of vnode names allocated to the execvnode
- resource – method to return the amount of a named resource satisfied by this execvnode
-
resource
(name=None)¶ Parameters: name (str or None) – Name of the resource
-
class
ptl.lib.pbs_testlib.
PbsTypeExecHost
(s=None)¶ Descriptor class for exec_host attribute
Parameters: hosts (List) – List of hosts in the exec_host. Each entry is a host info dictionary that maps the number of cpus and its task number
-
class
ptl.lib.pbs_testlib.
PbsTypeJobId
(value=None)¶ Descriptor class for a Job identifier
Parameters: - id (int) – The numeric portion of a job identifier
- server_name (str) – The pbs server name
- server_shortname (str) – The first portion of a FQDN server name
-
class
ptl.lib.pbs_testlib.
PbsUser
(name, uid=None, groups=None)¶ The PbsUser type augments a PBS username to associate it to groups to which the user belongs
Parameters: - name (str) – The user name referenced
- uid (int or None) – uid of user
- groups (List or None) – The list of PbsGroup objects the user belongs to
-
class
ptl.lib.pbs_testlib.
PbsGroup
(name, gid=None, users=None)¶ The PbsGroup type augments a PBS groupname to associate it to users to which the group belongs
Parameters: - name (str) – The group name referenced
- gid (int or None) – gid of group
- users (List or None) – The list of PbsUser objects the group belongs to
-
class
ptl.lib.pbs_testlib.
BatchUtils
¶ Utility class to create/convert/display various PBS data structures
-
list_to_attrl
(l)¶ Convert a list to a PBS attribute list
Parameters: l (List) – List to be converted Returns: PBS attribute list
-
list_to_attropl
(l, op=0)¶ Convert a list to a PBS attribute operation list
Parameters: l (List) – List to be converted Returns: PBS attribute operation list
-
str_to_attrl
(s)¶ Convert a string to a PBS attribute list
Parameters: s (str) – String to be converted Returns: PBS attribute list
-
str_to_attropl
(s, op=0)¶ Convert a string to a PBS attribute operation list
Parameters: s (str) – String to be converted Returns: PBS attribute operation list
-
dict_to_attrl
(d={})¶ Convert a dictionary to a PBS attribute list
Parameters: d (Dictionary) – Dictionary to be converted Returns: PBS attribute list
-
dict_to_attropl
(d={}, op=0)¶ Convert a dictionary to a PBS attribute operation list
Parameters: d (Dictionary) – Dictionary to be converted Returns: PBS attribute operation list
-
convert_to_attrl
(attrib)¶ Generic call to convert Python type to PBS attribute list
Parameters: attrib (List or tuple or dictionary or str) – Attributes to be converted Returns: PBS attribute list
-
convert_to_attropl
(attrib, cmd=13, op=None)¶ Generic call to convert Python type to PBS attribute operation list
Parameters: attrib (List or tuple or dictionary or str) – Attributes to be converted Returns: PBS attribute operation list
-
command_to_op
(cmd=None)¶ Map command to a
SET
orUNSET
Operation. An unrecognized command will return SET. No command will return None.Parameters: cmd (str) – Command to be mapped Returns: SET
orUNSET
operation for the command
-
display_attrl
(a=None, writer=<open file '<stdout>', mode 'w'>)¶ Display an attribute list using writer, defaults to sys.stdout
Parameters: a (List) – Attributes Returns: Displays attribute list
-
display_attropl
(attropl=None, writer=<open file '<stdout>', mode 'w'>)¶ Display an attribute operation list with writer, defaults to sys.stdout
Parameters: attropl (List) – Attribute operation list Returns: Displays an attribute operation list
-
display_dict
(d, writer=<open file '<stdout>', mode 'w'>)¶ Display a dictionary using writer, defaults to sys.stdout
Parameters: d (Dictionary) – Dictionary Returns: Displays a dictionary
-
batch_status_to_dictlist
(bs=None, attr_names=None, id=None)¶ Convert a batch status to a list of dictionaries. version 0.1a6 added this conversion as a typemap(out) as part of the swig wrapping itself so there are fewer uses for this function.Returns a list of dictionary representation of batch status
Parameters: - bs – Batch status
- attr_names – Attribute names
Returns: List of dictionaries
-
display_batch_status
(bs=None, attr_names=None, writer=<open file '<stdout>', mode 'w'>)¶ Display a batch status using writer, defaults to sys.stdout :param bs: Batch status :param attr_name: Attribute name :type attr_name: str :returns: Displays batch status
-
display_dictlist
(l=[], writer=<open file '<stdout>', mode 'w'>, fmt=None)¶ Display a list of dictionaries using writer, defaults to sys.stdout
Parameters: - l (List) – The list to display
- writer – The stream on which to write
- fmt (str or None) – An optional formatting string
Returns: Displays list of dictionaries
-
dictlist_to_file
(l=[], filename=None, mode='w')¶ write a dictlist to file
Parameters: - l (List) – Dictlist
- filename (str) – File to which dictlist need to be written
- mode (str) – Mode of file
Raises: Exception writing to file
-
batch_status_as_dictlist_to_file
(l=[], writer=<open file '<stdout>', mode 'w'>)¶ Write a dictlist to file
Parameters: l (List) – Dictlist Raises: Exception writing to file
-
file_to_dictlist
(file=None, attribs=None, id=None)¶ Convert a file to a batch dictlist format
Parameters: - file (str) – File to be converted
- attribs – Attributes
Returns: File converted to a batch dictlist format
-
file_to_vnodedef
(file=None)¶ Convert a file output of pbsnodes -av to a vnode definition format
Parameters: file – File to be converted Returns: Vnode definition format
-
show
(l=[], name=None, fmt=None)¶ Alias to display_dictlist with sys.stdout as writer
Parameters: - name (str) – if specified only show the object of that name
- fmt (str) – Optional formatting string, uses %n for object name, %a for attributes, for example a format of ‘%nE{}nE{}t%aE{}n’ will display objects with their name starting on the first column, a new line, and attributes indented by a tab followed by a new line at the end.
-
get_objtype
(d={})¶ Get the type of a given object
Parameters: d (Dictionary) – Dictionary Returns: Type of the object
-
display_batch_status_as_dictlist
(l=[], writer=<open file '<stdout>', mode 'w'>, fmt=None)¶ Display a batch status as a list of dictionaries using writer, defaults to sys.stdout
Parameters: - l (List) – List
- fmt (str or None) –
- Optional format string
Returns: Displays batch status as a list of dictionaries
-
batch_status_as_dict_to_str
(d={}, fmt=None)¶ Return a string representation of a batch status dictionary
Parameters: - d (Dictionary) – Dictionary
- fmt (str or None) – Optional format string
Returns: String representation of a batch status dictionary
-
display_batch_status_as_dict
(d={}, writer=<open file '<stdout>', mode 'w'>, fmt=None)¶ Display a dictionary representation of a batch status using writer, defaults to sys.stdout
Parameters: - d (Dictionary) – Dictionary
- fmt – Optional format string
- fmt – str
Returns: Displays dictionary representation of a batch status
-
decode_dictlist
(l=None, json=True)¶ decode a list of dictionaries
Parameters: - l (List) – List of dictionaries
- json – The target of the decode is meant for
JSON
formatting
Returns: Decoded list of dictionaries
-
convert_to_dictlist
(l, attribs=None, mergelines=True, id=None)¶ Convert a list of records into a dictlist format.
Parameters: - l (List) – array of records to convert
- mergelines – merge qstat broken lines into one
Returns: Record list converted into dictlist format
-
convert_to_batch
(l, mergelines=True)¶ Convert a list of records into a batch format.
Parameters: - l (List) – array of records to convert
- mergelines (bool) – qstat breaks long lines over multiple lines, merge them to one by default.
Returns: A linked list of batch status
-
file_to_batch
(file=None)¶ Convert a file to batch format
Parameters: file (str or None) – File to be converted Returns: File converted into batch format
-
batch_to_file
(bs=None, file=None)¶ Write a batch object to file
Parameters: - bs – Batch status
- file (str) – File to which batch object is to be written
-
batch_to_vnodedef
(bs)¶ Parameters: bs – Batch status Returns: The vnode definition string representation of nodes batch_status
-
dictlist_to_vnodedef
(dl=None)¶ Parameters: dl (List) – Dictionary list Returns: The vnode definition string representation of a dictlist
-
objlist_to_dictlist
(objlist=None)¶ Convert a list of PBS/PTL objects
(e.g. Server/Job...)
into a dictionary list representation of the batch statusParameters: objlist (List) – List of PBS/PTL
objectsReturns: Dictionary list representation of the batch status
-
obj_to_dict
(obj)¶ Convert a PBS/PTL object (e.g. Server/Job…) into a dictionary format
Parameters: obj – PBS/PTL
objectReturns: Dictionary of PBS/PTL
objects
-
parse_execvnode
(s=None)¶ Parse an execvnode string into chunk objects
Parameters: s (str or None) – Execvnode string Returns: Chunk objects for parsed execvnode string
-
anupbs_exechost_numhosts
(s=None)¶ Parameters: s (str or None) – Exechost string
-
parse_exechost
(s=None)¶ Parse an exechost string into a dictionary representation
Parameters: s (str or None) – String to be parsed Returns: Dictionary format of the exechost string
-
parse_select
(s=None)¶ Parse a
select/schedselect
string into a list of dictionaries.Parameters: s (str or None) – select/schedselect string Returns: List of dictonaries
-
classmethod
isfloat
(value)¶ returns true if value is a float or a string representation of a float returns false otherwise
Parameters: value (str or int or float) – value to be checked Returns: True or False
-
classmethod
decode_value
(value)¶ Decode an attribute/resource value, if a value is made up of digits only then return the numeric value of it, if it is made of alphanumeric values only, return it as a string, if it is of type size, i.e., with a memory unit such as b,kb,mb,gb then return the converted size to kb without the unit
Parameters: value (str or int) – attribute/resource value Returns: int or float or string
-
convert_time
(val, fmt='%a %b %d %H:%M:%S %Y')¶ Convert a date time format into number of seconds since epoch
Parameters: - val – date time value
- fmt (str) – date time format
Returns: seconds
-
convert_duration
(val)¶ Convert HH:MM:SS into number of seconds If a number is fed in, that number is returned If neither formatted data is fed in, returns 0
Parameters: val (str) – duration value Raises: Incorrect format error Returns: seconds
-
convert_seconds_to_datetime
(tm, fmt=None, seconds=True)¶ Convert time format to number of seconds since epoch
Parameters: - tm (str) – the time to convert
- fmt (str or None) – optional format string. If used, the seconds
parameter is ignored.Defaults to
%Y%m%d%H%M
- seconds (bool) – if True, convert time with seconds granularity. Defaults to True.
Returns: Number of seconds
-
convert_stime_to_seconds
(st)¶ Convert a time to seconds, if we fail we return the original time
Parameters: st (str) – Time to be converted Returns: Number of seconds
-
convert_dedtime
(dtime)¶ Convert dedicated time string of form %m/%d/%Y %H:%M.
Parameters: dtime (str) – A datetime string, as an entry in the dedicated_time file Returns: A tuple of (from,to) of time since epoch
-
convert_datetime_to_epoch
(mdyhms, fmt='%m/%d/%Y %H:%M:%S')¶ Convert the date time to epoch
Parameters: - mdyhms (str) – date time
- fmt (str) – Format for date time
Returns: Epoch time
-
compare_versions
(v1, v2, op=None)¶ Compare v1 to v2 with respect to operation op
Parameters: - v1 – If not a looseversion, it gets converted to it
- v2 – If not a looseversion, it gets converted to it
- op (str) – An operation, one of
LT
,LE
,EQ
,GE
,GT
Returns: True or False
-
convert_arglist
(attr)¶ strip the XML attributes from the argument list attribute
Parameters: attr (List) – Argument list attributes Returns: Stripped XML attributes
-
convert_to_cli
(attrs, op=None, hostname=None, dflt_conf=True, exclude_attrs=None)¶ Convert attributes into their CLI format counterpart. This method is far from complete, it grows as needs come by and could use a rewrite, especially going along with a rewrite of pbs_api_to_cli
Parameters: - attrs (List or str or dictionary) – Attributes to convert
- op (str or None) – The qualifier of the operation being performed,
such as
IFL_SUBMIT
,IFL_DELETE
,IFL_TERMINUTE
… - hostname (str or None) – The name of the host on which to operate
- dflt_conf (bool) – Whether we are using the default PBS configuration
- exclude_attrs (List) – Optional list of attributes to not convert
Returns: CLI format of attributes
-
filter_batch_status
(bs, attrib)¶ Filter out elements that don’t have the attributes requested This is needed to adapt to the fact that requesting a resource attribute returns all
'<resource-name>.*'
attributes so we need to ensure that the specific resource requested is present in the stat’ed object.This is needed especially when calling expect with an op=NE because we need to filter on objects that have exactly the attributes requested
Parameters: - bs – Batch status
- attrib (str or dictionary) – Requested attributes
Returns: Filtered batch status
-
convert_attributes_by_op
(attributes, setattrs=False)¶ Convert attributes by operator, i.e. convert an attribute of the form
<attr_name><op><value>
(e.g. resources_available.ncpus>4)to
<attr_name>: (<op>, <value>)
(e.g. resources_available.ncpus: (GT, 4))Parameters: - attributes (List) – the attributes to convert
- setattrs (bool) – if True, set the attributes with no operator as (SET, ‘’)
Returns: Converted attributes by operator
-
operator_in_attribute
(attrib)¶ Returns True if an operator string is present in an attribute name
Parameters: attrib (str) – Attribute name Returns: True or False
-
list_resources
(objtype=None, objs=[])¶ Lists the resources
Parameters: - objtype (str) – Type of the object
- objs (List) – Object list
Returns: List of resources
-
compare
(obj1, obj2, showdiff=False)¶ Compare two objects.
Parameters: showdiff (bool) – whether to print the specific differences, defaults to False Returns: 0 if objects are identical and non zero otherwise
-
classmethod
random_str
(length=1, prefix='')¶ Generates the random string
Parameters: - length (int) – Length of the string
- prefix (str) – Prefix of the string
Returns: Random string
-
update_attributes_list
(obj)¶ Updates the attribute list
Parameters: obj – Objects Returns: Updated attribute list
-
parse_fgc_limit
(limstr=None)¶ Parse an
FGC
limit entry, of the form:<limtype>[.<resource>]=\[<entity_type>:<entity_name> =<entity_value>\]
Parameters: limstr (str or None) – FGC limit string Returns: Parsed FGC string in given format
-
is_job_array
(jobid)¶ If a job array return True, otherwise return False
Parameters: jobid – PBS jobid Returns: True or False
-
is_subjob
(jobid)¶ If a subjob of a job array, return the subjob id otherwise return False
Parameters: jobid (str) – PBS job id Returns: True or False
-
-
class
ptl.lib.pbs_testlib.
PbsTypeFGCLimit
(attr, val)¶ FGC limit entry, of the form:
<limtype>[.<resource>]=\[<entity_type>:<entity_name>= <entity_value>\]
Parameters: - attr (str) – FGC limit attribute
- value (int) – Value of attribute
Returns: FGC limit entry of given format
-
class
ptl.lib.pbs_testlib.
PbsBatchStatus
(bs)¶ Wrapper class for Batch Status object Converts a batch status (as dictlist) into a list of PbsBatchObjects
Parameters: bs (List or dictionary) – Batch status Returns: List of PBS batch objects
-
class
ptl.lib.pbs_testlib.
PbsBatchObject
(bs)¶ -
set_batch_status
(bs)¶ Sets the batch status
Parameters: bs – Batch status
-
-
class
ptl.lib.pbs_testlib.
PbsAttribute
(name=None, value=None)¶ Descriptor class for PBS attribute
Parameters: - name (str) – PBS attribute name
- value (str or int or float) – Value for the attribute
-
set_name
(name)¶ Set PBS attribute name
Parameters: name (str) – PBS attribute
-
set_value
(value)¶ Set PBS attribute value
Parameters: value (str or int or float) – Value of PBS attribute
-
obfuscate_name
(a=None)¶ Obfuscate PBS attribute name
-
obfuscate_value
(v=None)¶ Obfuscate PBS attribute value
-
class
ptl.lib.pbs_testlib.
Entity
(etype=None, name=None)¶ Abstract representation of a PBS consumer that has an external relationship to the PBS system. For example, a user associated to an OS identifier (uid) maps to a PBS user entity.
Entities may be subject to policies, such as limits, consume a certain amount of resource and/or fairshare usage.
Parameters: - etype (str or None) – Entity type
- name (str or None) – Entity name
-
set_limit
(limit=None)¶ Parameters: limit (str or None) – Limit to be set
-
set_resource_usage
(container=None, resource=None, usage=None)¶ Set the resource type
Parameters: - resource (str or None) – PBS resource
- usage (str or None) – Resource usage value
Set fairshare usage
Parameters: usage (int) – Fairshare usage value
-
class
ptl.lib.pbs_testlib.
Policy
¶ Abstract PBS policy. Can be one of
limits
,access control
,scheduling policy
, etc…this class does not currently support any operations
-
class
ptl.lib.pbs_testlib.
Limit
(limit_type=None, resource=None, entity_obj=None, value=None, container=None, container_id=None)¶ Representation of a PBS limit Limits apply to containers, are of a certain type (e.g., max_run_res.ncpus) associated to a given resource (e.g., resource), on a given entity (e.g.,user Bob) and have a certain value.
Parameters: - limit_type (str or None) – Type of the limit
- resource (str or None) – PBS resource
- entity_obj – Entity object
- value (int) – Limit value
-
set_container
(container, container_id)¶ Set the container
Parameters: - container (str) – Container which is to be set
- container_id – Container id
-
set_limit_type
(t)¶ Set the limit type
Parameters: t (str) – Limit type
-
set_resource
(resource)¶ Set the resource
Parameters: resource (str) – resource value to set
-
set_value
(value)¶ Set the resource value
Parameters: value (str) – Resource value
-
class
ptl.lib.pbs_testlib.
ExpectActions
(action=None, level=20)¶ List of action handlers to run when Server’s expect function does not get the expected result
Parameters: - action (str) – Action to run
- level – Logging level
-
add_action
(action=None, hostname=None, level=20)¶ Add an action
Parameters: - action – Action to add
- hostname (str) – Machine hostname
- level – Logging level
-
has_action
(name)¶ check whether action exists or not
Parameters: name (str) – Name of action
-
get_action
(name)¶ Get an action if exists
Parameters: name (str) – Name of action
-
list_actions
(level=20)¶ List an actions
Parameters: level – Logging level
-
get_all_actions
()¶ Get all the action
-
get_actions_by_type
(atype=None)¶ Get an action by type
Parameters: atype (str) – Action type
-
disable_action
(action=None, name=None)¶ Disable an action
-
enable_action
(action=None, name=None)¶ Enable an action
-
disable_all_actions
()¶ Disable all actions
-
enable_all_actions
()¶ Enable all actions
-
class
ptl.lib.pbs_testlib.
ExpectAction
(name=None, enabled=True, atype=None, action=None, level=20)¶ Action function to run when Server’s expect function does not get the expected result
Parameters: atype (str) – Action type -
set_name
(name, level=20)¶ Set the actione name
Parameters: name (str) – Action name
-
-
class
ptl.lib.pbs_testlib.
PbsTypeAttribute
¶ Experimental. This is a placeholder object that will be used in the future to map attribute information and circumvent the error-pron dynamic type detection that is currently done using
decode_value()
-
class
ptl.lib.pbs_testlib.
PBSObject
(name, attrs={}, defaults={})¶ Generic PBS Object encapsulating attributes and defaults
Parameters: - name (str) – The name associated to the object
- attrs (Dictionary) – Dictionary of attributes to set on object
- defaults (Dictionary) – Dictionary of default attributes. Setting this will override any other object’s default
-
set_attributes
(a={})¶ set attributes and custom attributes on this object. custom attributes are used when converting attributes to CLI
Parameters: a (Dictionary) – Attribute dictionary
-
unset_attributes
(attrl=[])¶ Unset attributes from object’s attributes and custom attributes
Parameters: attrl (List) – Attribute list
-
class
ptl.lib.pbs_testlib.
PBSService
(name=None, attrs={}, defaults={}, pbsconf_file=None, diagmap={}, diag=None)¶ Generic PBS service object to hold properties of PBS daemons
Parameters: - name (str or None) – The name associated to the object
- attrs (Dictionary) – Dictionary of attributes to set on object
- defaults (Dictionary) – Dictionary of default attributes. Setting this will override any other object’s default
- pbsconf_file (str or None) – Optional path to the pbs configuration file
- diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
- diag (str or None) – path to PBS diag directory (This will overrides diagmap)
-
init_logfile_path
(conf=None)¶ Initialize path to log files for this service
Parameters: conf (Dictionary) – PBS conf file parameters
-
initialise_service
()¶ Purpose of this method is to override and initialise the service
-
log_lines
(logtype, id=None, n=50, tail=True, starttime=None, endtime=None)¶ Return the last
<n>
lines of a PBS log file, which can be one ofserver
,scheduler
,MoM
, ortracejob
Parameters: - logtype (str or object) – The entity requested, an instance of a Scheduler, Server or MoM object, or the string ‘tracejob’ for tracejob
- id – The id of the object to trace. Only used for tracejob
- n (str or int) – One of ‘ALL’ of the number of lines to process/display, defaults to 50.
- tail (bool) – if True, parse log from the end to the start, otherwise parse from the start to the end. Defaults to True.
- day (int) – Optional day in
YYYMMDD
format. Defaults to current day - starttime – date timestamp to start matching
- endtime – date timestamp to end matching
Returns: Last
<n>
lines of logfile forServer
,Scheduler
,MoM or tracejob
-
accounting_match
(msg, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶ Match given
msg
in givenn
lines of accounting logParameters: - msg (str) – log message to match, can be regex also when
regexp
is True - id (str) – The id of the object to trace. Only used for tracejob
- n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
- tail (bool) – If true (default), starts from the end of the file
- allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
- regexp (bool) – If true msg is a Python regular expression. Defaults to False
- max_attempts (int) – the number of attempts to make to find a matching entry
- interval (int) – the interval between attempts
- starttime (int) – If set ignore matches that occur before specified time
- endtime (int) – If set ignore matches that occur after specified time
- level (int) – The logging level, defaults to INFO
- existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns: (x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type: tuple
Raises: PtlLogMatchError – When
existence
is True and givenmsg
is not found inn
line Or Whenexistence
is False and givenmsg
found inn
line.Note
The matching line number is relative to the record number, not the absolute line number in the file.
- msg (str) – log message to match, can be regex also when
-
tracejob_match
(msg, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶ Match given
msg
in givenn
lines of tracejob logParameters: - msg (str) – log message to match, can be regex also when
regexp
is True - id (str) – The id of the object to trace.
- n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
- tail (bool) – If true (default), starts from the end of the file
- allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
- regexp (bool) – If true msg is a Python regular expression. Defaults to False
- max_attempts (int) – the number of attempts to make to find a matching entry
- interval (int) – the interval between attempts
- starttime (int) – If set ignore matches that occur before specified time
- endtime (int) – If set ignore matches that occur after specified time
- level (int) – The logging level, defaults to INFO
- existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns: (x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type: tuple
Raises: PtlLogMatchError – When
existence
is True and givenmsg
is not found inn
line Or Whenexistence
is False and givenmsg
found inn
line.Note
The matching line number is relative to the record number, not the absolute line number in the file.
- msg (str) – log message to match, can be regex also when
-
get_tempdir
()¶ platform independent call to get a temporary directory
-
cleanup_files
()¶ This function removes any dynamic resource files created by server/mom objects
-
class
ptl.lib.pbs_testlib.
Comm
(name=None, attrs={}, pbsconf_file=None, diagmap={}, diag=None, server=None, db_access=None)¶ PBS
Comm
configuration and control-
isUp
()¶ Check for comm up
-
signal
(sig)¶ Send signal to comm
-
get_pid
()¶ Get the comm pid
-
all_instance_pids
()¶ Get all pids of given instance
-
start
(args=None, launcher=None)¶ Start the comm
Parameters: - args (str) – Argument required to start the comm
- launcher (str or list) – Optional utility to invoke the launch of the service
-
stop
(sig=None)¶ Stop the comm.
Parameters: sig (str) – Signal to stop the comm
-
restart
()¶ Restart the comm.
-
log_match
(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶ Match given
msg
in givenn
lines of Comm logParameters: - msg (str) – log message to match, can be regex also when
regexp
is True - id (str) – The id of the object to trace. Only used for tracejob
- n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
- tail (bool) – If true (default), starts from the end of the file
- allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
- regexp (bool) – If true msg is a Python regular expression. Defaults to False
- max_attempts (int) – the number of attempts to make to find a matching entry
- interval (int) – the interval between attempts
- starttime (int) – If set ignore matches that occur before specified time
- endtime (int) – If set ignore matches that occur after specified time
- level (int) – The logging level, defaults to INFO
- existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns: (x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type: tuple
Raises: PtlLogMatchError – When
existence
is True and givenmsg
is not found inn
line Or Whenexistence
is False and givenmsg
found inn
line.Note
The matching line number is relative to the record number, not the absolute line number in the file.
- msg (str) – log message to match, can be regex also when
-
-
class
ptl.lib.pbs_testlib.
Server
(name=None, attrs={}, defaults={}, pbsconf_file=None, diagmap={}, diag=None, client=None, client_pbsconf_file=None, db_access=None, stat=True)¶ PBS server
configuration
andcontrol
The Server class is a container to PBS server attributes and implements wrappers to the
IFL API
to perform operations on the server. For example to submit, status, delete, manage, etc… jobs, reservations and configurations.This class also offers higher-level routines to ease testing, see functions, for
example: revert_to_defaults, init_logging, expect, counter.
The ptl_conf dictionary holds general configuration for the framework’s operations, specifically, one can control:
mode: set to
PTL_CLI
to operate inCLI
mode orPTL_API
to operate inAPI
modeexpect_max_attempts: the default maximum number of attempts to be usedby expect. Defaults to 60
expect_interval: the default time interval (in seconds) between expectrequests. Defaults to 0.5
update_attributes: the default on whether Object attributes should beupdated using a list of dictionaries. Defaults to True
Parameters: - name (str) – The hostname of the server. Defaults to calling pbs_default()
- attrs (Dictionary) – Dictionary of attributes to set, these will override defaults.
- defaults (Dictionary) – Dictionary of default attributes. Default: dflt_attributes
- pbsconf_file (str) – path to config file to parse for PBS_HOME, PBS_EXEC, etc
- diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
- diag (str) – path to PBS diag directory (This will overrides diagmap)
- client (str) – The host to use as client for CLI queries. Defaults to the local hostname.
- client_pbsconf_file (str) – The path to a custom PBS_CONF_FILE on the client host. Defaults to the same path as pbsconf_file.
- db_acccess – set to either file containing credentials to DB access or dictionary containing {‘dbname’:…,’user’:…,’port’:…}
- stat (bool) – if True, stat the server attributes
-
update_version_info
()¶ Update the version information.
-
classmethod
set_update_attributes
(val)¶ Set update attributes
-
classmethod
set_expect_max_attempts
(val)¶ Set expect max attempts
-
classmethod
set_expect_interval
(val)¶ Set expect interval
-
set_client
(name=None)¶ Set server client
Parameters: name (str) – Client name
-
set_connect_timeout
(timeout=0)¶ Set server connection timeout
Parameters: timeout (int) – Timeout value
-
get_op_mode
()¶ Returns operating mode for calls to the PBS server. Currently, two modes are supported, either the
API
or theCLI
. Default isAPI
-
set_op_mode
(mode)¶ set operating mode to one of either
PTL_CLI
orPTL_API
.Returns the mode that was set which can be different from the value requested, for example, if requesting to setPTL_API
, in the absence of the appropriate SWIG wrappers, the library will fall back toCLI
, or if requestingPTL_CLI
and there is noPBS_EXEC
on the system, None is returned.Parameters: mode (str) – Operating mode
-
add_expect_action
(name=None, action=None)¶ Add an action handler to expect. Expect Actions are custom handlers that are triggered when an unexpected value is encountered
Parameters: - name (str or None) – Action name
- action – Action to add
-
set_attributes
(a={})¶ set server attributes
Parameters: a (Dictionary) – Attribute dictionary
-
isUp
()¶ returns
True
if server is up andFalse
otherwise
-
signal
(sig)¶ Send signal to server
Parameters: sig (str) – Signal to send
-
get_pid
()¶ Get the server pid
-
all_instance_pids
()¶ Get all pids for a given instance
-
start
(args=None, launcher=None)¶ Start the PBS server
Parameters: - args (str) – Argument required to start the server
- launcher (str or list) – Optional utility to invoke the launch of the service
-
stop
(sig=None)¶ Stop the PBS server
Parameters: sig (str) – Signal to stop PBS server
-
restart
()¶ Terminate and start a PBS server.
-
log_match
(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶ Match given
msg
in givenn
lines of Server logParameters: - msg (str) – log message to match, can be regex also when
regexp
is True - id (str) – The id of the object to trace. Only used for tracejob
- n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
- tail (bool) – If true (default), starts from the end of the file
- allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
- regexp (bool) – If true msg is a Python regular expression. Defaults to False
- max_attempts (int) – the number of attempts to make to find a matching entry
- interval (int) – the interval between attempts
- starttime (int) – If set ignore matches that occur before specified time
- endtime (int) – If set ignore matches that occur after specified time
- level (int) – The logging level, defaults to INFO
- existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns: (x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type: tuple
Raises: PtlLogMatchError – When
existence
is True and givenmsg
is not found inn
line Or Whenexistence
is False and givenmsg
found inn
line.Note
The matching line number is relative to the record number, not the absolute line number in the file.
- msg (str) – log message to match, can be regex also when
-
revert_to_defaults
(reverthooks=True, revertqueues=True, revertresources=True, delhooks=True, delqueues=True, delscheds=True, delnodes=True, server_stat=None)¶ reset server attributes back to out of box defaults.
Parameters: - reverthooks (bool) – If True disable all hooks. Defaults to True
- revertqueues (bool) – If True disable all non-default queues. Defaults to True
- revertresources (bool) – If True, resourcedef file is removed. Defaults to True. Reverting resources causes a server restart to occur.
- delhooks (bool) – If True, hooks are deleted, if deletion fails, fall back to reverting hooks. Defaults to True.
- delqueues (bool) – If True, all non-default queues are deleted, will attempt to delete all jobs first, if it fails, revertqueues will be honored, otherwise,revertqueues is ignored. Defaults to True
- delscheds (bool) – If True all non-default schedulers are deleted The sched_priv and sched_logs directories will be deleted.
- delnodes (bool) – If True all vnodes are deleted
Returns: True upon success and False if an error is encountered.
Raises: PbsStatusError or PbsManagerError
-
save_configuration
(outfile, mode='a')¶ Save a server configuration, this includes:
server_priv/resourcedef
qmgr -c "print server"
qmgr -c "print sched"
qmgr -c "print hook"
Parameters: - outfile (str) – the output file to which onfiguration is saved
- mode (str) – The mode in which to open outfile to save configuration. The first object being saved should open this file with ‘w’ and subsequent calls from other objects should save with mode ‘a’ or ‘a+’. Defaults to a+
Returns: True on success, False on error
-
load_configuration
(infile)¶ load configuration from saved file
infile
-
get_hostname
()¶ return the default server hostname
-
status_db
(obj_type=None, attrib=None, id=None, db_access=None, logerr=True)¶ Status PBS objects from the SQL database
Parameters: - obj_type – The type of object to query, one of the * objects,Default: SERVER
- attrib (str or list or dictionary) – Attributes to query, can a string, a list, a dictionaryDefault: None. All attributes will be queried
- id (str) – An optional identifier, the name of the object to status
- db_access (str or dictionary) – information needed to access the database, can be either a file containing user, port, dbname, password info or a dictionary of key/value entries
-
status
(obj_type=0, attrib=None, id=None, extend=None, level=20, db_access=None, runas=None, resolve_indirectness=False, logerr=True)¶ Stat any PBS object
[queue, server, node, hook, job, resv, sched]
.If the Server is setup from diag input, see diag or diagmap member, the status calls are routed directly to the data on files from diag.The server can be queried either through the ‘qstat’ command line tool or through the wrapped PBS IFL api, see set_op_mode.
Return a dictionary representation of a batch status object raises
PbsStatsuError on error
.Parameters: - obj_type – The type of object to query, one of the * objects.Default: SERVER
- attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary.Default is to query all attributes.
- id (str) – An optional id, the name of the object to status
- extend – Optional extension to the IFL call
- level (str) – The logging level, defaults to INFO
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
- runas (str) – run stat as user
- resolve_indirectness (bool) – If True resolves indirect node resources values
- logerr (bool) – If True (default) logs run_cmd errors
In addition to standard IFL stat call, this wrapper handles a few cases that aren’t implicitly offered by pbs_stat*, those are for Hooks,Resources, and a formula evaluation.
-
submit_interactive_job
(job, cmd)¶ submit an
interactive
job. Returns a job identifier or raises PbsSubmitError on errorParameters: - cmd (str) – The command to run to submit the interactive job
- job – the job object. The job must have the attribute ‘interactive_job’ populated. That attribute is a list of tuples of the form: (<command>, <expected output>, <…>) for example to send the command hostname and expect ‘myhost.mydomain’ one would set:job.interactive_job = [(‘hostname’, ‘myhost.mydomain’)] If more than one lines are expected they are appended to the tuple.
Raises: PbsSubmitError
-
submit
(obj, script=None, extend=None, submit_dir=None)¶ Submit a job or reservation. Returns a job identifier or raises PbsSubmitError on error
Parameters: - obj – The Job or Reservation instance to submit
- script (str or None) – Path to a script to submit. Default: None as an executable/bin/sleep 100 is submitted
- extend (str or None) – Optional extension to the IFL call. see pbs_ifl.h
- submit_dir (str or None) – directory from which job is submitted. Defaults to temporary directory
Raises: PbsSubmitError
-
deljob
(id=None, extend=None, runas=None, wait=False, logerr=True, attr_W=None)¶ delete a single job or list of jobs specified by id raises
PbsDeljobError
on errorParameters: - id (str or list) – The identifier(s) of the jobs to delete
- extend (str or None) – Optional parameters to pass along to PBS
- runas (str or None) – run as user
- wait (bool) – Set to True to wait for job(s) to no longer be reported by PBS. False by default
- logerr (bool) – Whether to log errors. Defaults to True.
- attr_w (str) – -W args to qdel (Only for cli mode)
Raises: PbsDeljobError
-
delresv
(id=None, extend=None, runas=None, wait=False, logerr=True)¶ delete a single job or list of jobs specified by id raises
PbsDeljobError
on errorParameters: - id (str or list) – The identifier(s) of the jobs to delete
- extend (str or None) – Optional parameters to pass along to PBS
- runas (str or None) – run as user
- wait (bool) – Set to True to wait for job(s) to no longer be reported by PBS. False by default
- logerr (bool) – Whether to log errors. Defaults to True.
Raises: PbsDeljobError
-
delete
(id=None, extend=None, runas=None, wait=False, logerr=True)¶ delete a single job or list of jobs specified by id raises
PbsDeleteError
on errorParameters: - id (str or list) – The identifier(s) of the jobs/resvs to delete
- extend (str or none) – Optional parameters to pass along to PBS
- runas (str) – run as user
- wait (bool) – Set to True to wait for job(s)/resv(s) to no longer be reported by PBS. False by default
- logerr (bool) – Whether to log errors. Defaults to True.
Raises: PbsDeleteError
-
select
(attrib=None, extend=None, runas=None, logerr=True)¶ Select jobs that match attributes list or all jobs if no attributes raises
PbsSelectError
on errorParameters: - attrib (str or list or dictionary) – A string, list, or dictionary of attributes
- extend (str or None) – the extended attributes to pass to select
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Returns: A list of job identifiers that match the attributes specified
Raises: PbsSelectError
-
selstat
(select_list, rattrib, runas=None, extend=None)¶ stat and filter jobs attributes.
Parameters: - select_list – The filter criteria
- rattrib (List) – The attributes to query
- runas (str or None) – run as user
Note
No
CLI
counterpart for this call
-
manager
(cmd, obj_type, attrib=None, id=None, extend=None, level=20, sudo=None, runas=None, logerr=True)¶ issue a management command to the server, e.g to set an attribute
Returns 0 for Success and non 0 number for Failure
Parameters: - cmd (str) – The command to issue,
MGR_CMD_[SET,UNSET, LIST,...]
see pbs_ifl.h - obj_type – The type of object to query, one of the * objects
- attrib (str or list or dictionary) – Attributes to operate on, can be a string, a list,a dictionary
- id (str or list) – The name or list of names of the object(s) to act upon.
- extend (str or None) – Optional extension to the IFL call. see pbs_ifl.h
- level – logging level
- sudo (bool) – If True, run the manager command as super user. Defaults to None. Some attribute settings should be run with sudo set to True, those are acl_roots, job_sort_formula, hook operations, no_sched_hook_event, in those cases, setting sudo to False is only needed for testing purposes
- runas (str) – run as user
- logerr (bool) – If False, CLI commands do not log error, i.e. silent mode
Raises: PbsManagerError
- cmd (str) – The command to issue,
-
sigjob
(jobid=None, signal=None, extend=None, runas=None, logerr=True)¶ Send a signal to a job. Raises
PbsSignalError
on error.Parameters: - jobid (str or list) – identifier of the job or list of jobs to send the signal to
- signal (str or None) – The signal to send to the job, see pbs_ifl.h
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsSignalError
-
msgjob
(jobid=None, to_file=None, msg=None, extend=None, runas=None, logerr=True)¶ Send a message to a job. Raises
PbsMessageError
on error.Parameters: - jobid (str or List) – identifier of the job or list of jobs to send the message to
- msg (str or None) – The message to send to the job
- to_file (str or None) – one of
MSG_ERR
orMSG_OUT
orMSG_ERR|MSG_OUT
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsMessageError
-
alterjob
(jobid=None, attrib=None, extend=None, runas=None, logerr=True)¶ Alter attributes associated to a job. Raises
PbsAlterError
on error.Parameters: - jobid (str or list) – identifier of the job or list of jobs to operate on
- attrib (dictionary) – A dictionary of attributes to set
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If False, CLI commands do not log error, i.e. silent mode
Raises: PbsAlterError
-
holdjob
(jobid=None, holdtype=None, extend=None, runas=None, logerr=True)¶ Hold a job. Raises
PbsHoldError
on error.Parameters: - jobid (str or list) – identifier of the job or list of jobs to hold
- holdtype (str or None) – The type of hold to put on the job
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsHoldError
-
rlsjob
(jobid, holdtype, extend=None, runas=None, logerr=True)¶ Release a job. Raises
PbsReleaseError
on error.Parameters: - jobid (str or list) – job or list of jobs to release
- holdtype (str) – The type of hold to release on the job
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsReleaseError
-
rerunjob
(jobid=None, extend=None, runas=None, logerr=True)¶ Rerun a job. Raises
PbsRerunError
on error. :param jobid: job or list of jobs to release :type jobid: str or list :param extend: extend options :param runas: run as user :type runas: str or None :param logerr: If True (default) logs run_cmd errors :type logerr: bool :raises: PbsRerunError
-
orderjob
(jobid1=None, jobid2=None, extend=None, runas=None, logerr=True)¶ reorder position of
jobid1
andjobid2
. RaisesPbsOrderJob
on error.Parameters: - jobid1 (str or None) – first jobid
- jobid2 (str or None) – second jobid
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsOrderJob
-
runjob
(jobid=None, location=None, async=False, extend=None, runas=None, logerr=False)¶ Run a job on given nodes. Raises
PbsRunError
on error.Parameters: - jobid (str or list) – job or list of jobs to run
- location (str or None) – An execvnode on which to run the job
- async (bool) – If true the call will return immediately assuming success.
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsRunError
-
movejob
(jobid=None, destination=None, extend=None, runas=None, logerr=True)¶ Move a job or list of job ids to a given destination queue. Raises
PbsMoveError
on error.Parameters: - jobid (str or list) – A job or list of job ids to move
- destination (str or None) – The destination queue@server
- extend – extend options
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsMoveError
-
qterm
(manner=None, extend=None, server_name=None, runas=None, logerr=True)¶ Terminate the
pbs_server
daemonParameters: - manner – one of
(SHUT_IMMEDIATE | SHUT_DELAY | SHUT_QUICK)
and can be combined with SHUT_WHO_SCHED, SHUT_WHO_MOM, SHUT_WHO_SECDRY, SHUT_WHO_IDLESECDRY, SHUT_WHO_SECDONLY. :param extend: extend options - server_name (str or None) – name of the pbs server
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsQtermError
- manner – one of
-
teminate
(manner=None, extend=None, server_name=None, runas=None, logerr=True)¶ Terminate the
pbs_server
daemonParameters: - manner – one of
(SHUT_IMMEDIATE | SHUT_DELAY | SHUT_QUICK)
and can be combined with SHUT_WHO_SCHED, SHUT_WHO_MOM, SHUT_WHO_SECDRY, SHUT_WHO_IDLESECDRY, SHUT_WHO_SECDONLY. :param extend: extend options - server_name (str or None) – name of the pbs server
- runas (str or None) – run as user
- logerr (bool) – If True (default) logs run_cmd errors
Raises: PbsQtermError
- manner – one of
-
geterrmsg
()¶ Get the error message
-
qdisable
(queue=None, runas=None, logerr=True)¶ Disable queue.
CLI
mode onlyParameters: - queue (str or list) – The name of the queue or list of queue to disable
- runas (str or None) – Optional name of user to run command as
- logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises: PbsQdisableError
-
qenable
(queue=None, runas=None, logerr=True)¶ Enable queue.
CLI
mode onlyParameters: - queue (str or list) – The name of the queue or list of queue to enable
- runas (str or None) – Optional name of user to run command as
- logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises: PbsQenableError
-
qstart
(queue=None, runas=None, logerr=True)¶ Start queue.
CLI
mode onlyParameters: - queue (str or list) – The name of the queue or list of queue to start
- runas (str or None) – Optional name of user to run command as
- logerr (bool) – Set to False ot disable logging command errors.Defaults to True.
Raises: PbsQstartError
-
qstop
(queue=None, runas=None, logerr=True)¶ Stop queue.
CLI
mode onlyParameters: - queue (str or list) – The name of the queue or list of queue to stop
- runas (str or None) – Optional name of user to run command as
- logerr (bool) – Set to False ot disable logging command errors. Defaults to True.
Raises: PbsQstopError
-
parse_resources
()¶ Parse server resources as defined in the resourcedef file Populates instance variable self.resources
Returns: The resources as a dictionary
-
remove_resource
(name)¶ Remove an entry from resourcedef
Parameters: - name (str) – The name of the resource to remove
- restart – Whether to restart the server or not. Applicable to update_mode ‘file’ operations only.
- update_mode – one of ‘file’ or ‘auto’ (the default). If ‘file’, updates the resourcedef file only and will not use the qmgr operations on resources introduced in 12.3. If ‘auto’, will automatically handle the update on resourcedef or using qmgr based on the version of the Server.
-
add_resource
(name, type=None, flag=None)¶ Define a server resource
Parameters: - name (str) – The name of the resource to add to the resourcedef file
- type – The type of the resource, one of string, long, boolean, float
- flag (str or None) – The target of the resource, one of n, h, q, or none
- restart – Whether to restart the server after adding a resource.Applicable to update_mode ‘file’ operations only.
- update_mode – one of ‘file’ or ‘auto’ (the default). If ‘file’, updates the resourcedef file only and will not use the qmgr operations on resources introduced in 12.3. If ‘auto’, will automatically handle the update on resourcedef or using qmgr based on the version of the Server.
Returns: True on success False on error
-
write_resourcedef
(resources=None, filename=None, restart=True)¶ Write into resource def file
Parameters: - resources (dictionary) – PBS resources
- filename (str or None) – resourcedef file name
-
parse_resourcedef
(file=None)¶ Parse an arbitrary resource definition file passed as input and return a dictionary of resources
Parameters: file (str or None) – resource definition file Returns: Dictionary of resource Raises: PbsResourceError
-
pbs_api_as
(cmd=None, obj=None, user=None, **kwargs)¶ Generic handler to run an
API
call impersonating a given user.This method is only used for impersonation over theAPI
becauseCLI
impersonation takes place through the genericDshUtils
run_cmd mechanism.Parameters: - cmd (str or None) – PBS command
- user (str or None) – PBS user or current user
Raises: eval
-
alterresv
(resvid, attrib, extend=None, runas=None, logerr=True)¶ Alter attributes associated to a reservation. Raises
PbsResvAlterError
on error.Parameters: - resvid (str.) – identifier of the reservation.
- attrib (dictionary.) – A dictionary of attributes to set.
- extend – extend options.
- runas (str or None.) – run as user.
- logerr (bool.) – If False, CLI commands do not log error, i.e. silent mode.
Raises: PbsResvAlterError.
-
expect
(obj_type, attrib=None, id=None, op=4, attrop=1, attempt=0, max_attempts=None, interval=None, count=None, extend=None, offset=0, runas=None, level=20, msg=None)¶ expect an attribute to match a given value as per an operation.
Parameters: - obj_type (str) – The type of object to query, JOB, SERVER, SCHEDULER, QUEUE NODE
- attrib (str or list or dictionary) – Attributes to query, can be a string, a list, or a dict
- id – The id of the object to act upon
- op – An operation to perform on the queried data, e.g., EQ, SET, LT,..
- attrop – Operation on multiple attributes, either PTL_AND, PTL_OR when an PTL_AND is used, only batch objects having all matches are returned, otherwise an OR is applied
- attempt (int) – The number of times this function has been called
- max_attempts (int or None) – The maximum number of attempts to perform.C{param_max_attempts}: 5
- interval – The interval time btween attempts. C{param_interval}: 1s
- count (bool) – If True, attrib will be accumulated using function counter
- extend – passed to the stat call
- offset (int) – the time to wait before the initial check. Defaults to 0.
- runas (str or None) – query as a given user. Defaults to current user
- msg (str or None) – Message from last call of this function, this message will be used while raising PtlExpectError.
Returns: True if attributes are as expected
Raises: PtlExpectError if attributes are not as expected
-
is_history_enabled
()¶ Short-hand method to return the value of job_history_enable
-
cleanup_jobs
(extend=None, runas=None)¶ Helper function to delete all jobs. By default this method will determine whether job_history_enable is on and will cleanup all history jobs. Specifying an extend parameter could override this behavior.
Parameters: runas (str or None) – Clean the job as
-
cleanup_reservations
(extend=None, runas=None)¶ Helper function to delete all reservations
-
cleanup_jobs_and_reservations
(extend='forcedeletehist')¶ Helper function to delete all jobs and reservations
Parameters: - extend – Optional extend parameter that is passed to delete. It defaults to ‘deletehist’ which is used in qdel and pbs_deljob() to force delete all jobs, including history jobs
- extend – str
-
update_attributes
(obj_type, bs)¶ Populate objects from batch status data
-
counter
(obj_type=None, attrib=None, id=None, extend=None, op=None, attrop=None, bslist=None, level=20, idonly=True, grandtotal=False, db_access=None, runas=None, resolve_indirectness=False)¶ Accumulate properties set on an object. For example, to count number of free nodes:
server.counter(VNODE,{'state':'free'})
Parameters: - obj_type – The type of object to query, one of the * objects
- attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary
- id – The id of the object to act upon
- extend – The extended parameter to pass to the stat call
- op (str or None) – The operation used to match attrib to what is queried. SET or None
- attrop – Operation on multiple attributes, either PTL_AND, PTL_OR
- bslist – Optional, use a batch status dict list instead of an obj_type
- idonly (bool) – if true, return the name/id of the matching objects
- db_access (str or dictionary) – credentials to access db, either a path to file or dictionary
- runas (str or None) – run as user
-
filter
(obj_type=None, attrib=None, id=None, extend=None, op=None, attrop=None, bslist=None, idonly=True, grandtotal=False, db_access=None, runas=None, resolve_indirectness=False)¶ Filter objects by properties. For example, to filter all free nodes:
server.filter(VNODE,{'state':'free'})
For each attribute queried, if idonly is True, a list of matching object names is returned; if idonly is False, then the value of each attribute queried is returned.
This is unlike Python’s built-in ‘filter’ that returns a subset of objects matching from a pool of objects. The Python filtering mechanism remains very useful in some situations and should be used programmatically to achieve desired filtering goals that can not be met easily with PTL’s filter method.
Parameters: - obj_type – The type of object to query, one of the * objects
- attrib (str or list or dictionary) – Attributes to query, can be a string, a list, a dictionary
- id – The id of the object to act upon
- extend – The extended parameter to pass to the stat call
- op (str or None) – The operation used to match attrib to what is queried. SET or None
- bslist (List or None) – Optional, use a batch status dict list instead of an obj_type
- idonly (bool) – if true, return the name/id of the matching objects
- db_access (str or dictionary) – credentials to access db, either path to file or dictionary
- runas (str or None) – run as user
-
logit
(msg, obj_type, attrib, id, level=20)¶ Generic logging routine for
IFL
commandsParameters: - msg (str) – The message to log
- obj_type – object type, i.e *
- attrib – attributes to log
- id (str or list) – name of object to log
- level – log level, defaults to
INFO
-
equivalence_classes
(obj_type=None, attrib={}, bslist=None, op=0, show_zero_resources=True, db_access=None, resolve_indirectness=False)¶ Parameters: - obj_type – PBS Object to query, one of *
- attrib (dictionary) – attributes to build equivalence classes out of.
- bslist (List) – Optional, list of dictionary representation of a batch status
- op – set to RESOURCES_AVAILABLE uses the dynamic amount of resources available, i.e., available - assigned, otherwise uses static amount of resources available
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
-
show_equivalence_classes
(eq=None, obj_type=None, attrib={}, bslist=None, op=0, show_zero_resources=True, db_access=None, resolve_indirectness=False)¶ helper function to show the equivalence classes
Parameters: - eq – equivalence classes as compute by equivalence_classes see equivalence_classes for remaining parameters description
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
-
whats_available
(attrib=None, jobs=None, resvs=None, nodes=None)¶ Returns what’s available as a list of node equivalence classes listed by availability over time.
Parameters: - attrib (List) – attributes to consider
- jobs – jobs to consider, if None, jobs are queried locally
- resvs – reservations to consider, if None, they are queried locally
- nodes – nodes to consider, if None, they are queried locally
-
show_whats_available
(wa=None, attrib=None, jobs=None, resvs=None, nodes=None)¶ helper function to show availability as computed by whats_available
Parameters: wa (Dictionary) – a dictionary of available attributes. see whats_available for a description of the remaining parameters
-
utilization
(resources=None, nodes=None, jobs=None, entity={})¶ Return utilization of consumable resources on a set of nodes
Parameters: - nodes (List) – A list of dictionary of nodes on which to compute utilization.Defaults to nodes resulting from a stat call to the current server.
- resources (List) – comma-separated list of resources to compute utilization on. The name of the resource is for example, ncpus or mem
- entity (Dictionary) – An optional dictionary of entities to
compute utilization of,
e.g. {'user':u1, 'group':g1, 'project'=p1}
The utilization is returned as a dictionary of percentage utilization for each resource.
Non-consumable resources are silently ignored.
-
create_vnodes
(name=None, attrib=None, num=1, mom=None, additive=False, sharednode=True, restart=True, delall=True, natvnode=None, usenatvnode=False, attrfunc=None, fname=None, vnodes_per_host=1, createnode=True, expect=True)¶ helper function to create vnodes.
Parameters: - name (str or None) – prefix name of the vnode(s) to create
- attrib – attributes to assign to each node
- num (int) – the number of vnodes to create. Defaults to 1
- mom – the MoM object on which the vnode definition is to be inserted
- additive (bool) – If True, vnodes are added to the existing vnode defs.Defaults to False.
- sharednode (bool) – If True, all vnodes will share the same host.Defaults to True.
- restart (bool) – If True the MoM will be restarted.
- delall (bool) – If True delete all server nodes prior to inserting vnodes
- natvnode (str or None) – name of the natural vnode.i.e. The node name in qmgr -c “create node <name>”
- usenatvnode (bool) – count the natural vnode as an allocatable node.
- attrfunc – an attribute=value function generator, see create_vnode_def
- fname (str or None) – optional name of the vnode def file
- vnodes_per_host (int) – number of vnodes per host
- createnode (bool) – whether to create the node via manage or not. Defaults to True
- expect (bool) – whether to expect attributes to be set or not. Defaults to True
Returns: True on success and False otherwise
-
create_moms
(name=None, attrib=None, num=1, delall=True, createnode=True, conf_prefix='pbs.conf_m', home_prefix='pbs_m', momhosts=None, init_port=15011, step_port=2)¶ Create MoM configurations and optionall add them to the server. Unique
pbs.conf
files are defined and created on each hosts on which MoMs are to be created.Parameters: - name (str or None) – Optional prefix name of the nodes to create. Defaults to the name of the MoM host.
- attrib – Optional node attributes to assign to the MoM.
- num (int) – Number of MoMs to create
- delall (bool) – Whether to delete all nodes on the server. Defaults to True.
- createnode (bool) – Whether to create the nodes and add them to the server.Defaults to True.
- conf_prefix (str) – The prefix of the PBS conf file.Defaults to pbs.conf_m
- home_prefix (str) – The prefix of the PBS_HOME directory. Defaults to pbs_m
- momhosts (List) – A list of hosts on which to deploy num MoMs.
- init_port (int) – The initial port number to start assigning
PBS_MOM_SERIVCE_PORT to. Default 15011
. - step_port (int) – The increments at which ports are allocated. Defaults to 2.
Note
Since PBS requires that PBS_MANAGER_SERVICE_PORT = PBS_MOM_SERVICE_PORT+1 The step number must be greater or equal to 2.
-
create_hook
(name, attrs)¶ Helper function to create a hook by name.
Parameters: - name (str) – The name of the hook to create
- attrs (str) – The attributes to create the hook with.
Returns: False if hook already exists
Raises: PbsManagerError, otherwise return True.
-
import_hook
(name, body, level=20)¶ Helper function to import hook body into hook by name. The hook must have been created prior to calling this function.
Parameters: - name (str) – The name of the hook to import body to
- body (str) – The body of the hook as a string.
Returns: True on success.
Raises: PbsManagerError
-
create_import_hook
(name, attrs=None, body=None, overwrite=True, level=20)¶ Helper function to create a hook, import content into it, set the event and enable it.
Parameters: - name (str) – The name of the hook to create
- attrs (str) – The attributes to create the hook with. Event and Enabled are mandatory. No defaults.
- body (str) – The hook body as a string
- overwrite – If True, if a hook of the same name already exists, bypass its creation. Defaults to True
Returns: True on success and False otherwise
-
evaluate_formula
(jobid=None, formula=None, full=True, include_running_jobs=False, exclude_subjobs=True)¶ Evaluate the job sort formula
Parameters: - jobid (str or None) – If set, evaluate the formula for the given jobid, if not set,formula is evaluated for all jobs in state Q
- formula – If set use the given formula. If not set, the server’s formula, if any, is used
- full (bool) – If True, returns a dictionary of job identifiers as keys and the evaluated formula as values. Returns None if no formula is used. Each job id formula is returned as a tuple (s,e) where s is the formula expression associated to the job and e is the evaluated numeric value of that expression, for example, if job_sort_formula is ncpus + mem a job requesting 2 cpus and 100kb of memory would return (‘2 + 100’, 102). If False, if a jobid is specified, return the integer value of the evaluated formula.
- include_running_jobs (bool) – If True, reports formula value of running jobs. Defaults to False.
- exclude_subjobs (bool) – If True, only report formula of parent job array
-
parse_server_limits
(server=None, db_access=None)¶ Parse all server limits
Parameters: - server (List) – list of dictionary of server data
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
-
parse_queue_limits
(queues=None, id=None, db_access=None)¶ Parse queue limits
Parameters: - queues (List) – list of dictionary of queue data
- id – The id of the queue to parse limit for. If None, all queue limits are parsed
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
-
parse_all_limits
(server=None, queues=None, db_access=None)¶ Parse all server and queue limits
Parameters: - server (List) – list of dictionary of server data
- queues (List) – list of dictionary of queue data
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
-
limits_info
(etype=None, ename=None, server=None, queues=None, jobs=None, db_access=None, over=False)¶ Collect limit information for each entity on which a
server/queue
limit is applied.Parameters: - etype (str or None) – entity type, one of u, g, p, o
- ename (str or None) – entity name
- server (List) – optional list of dictionary representation of server object
- queues (List) – optional list of dictionary representation of queues object
- jobs (List) – optional list of dictionary representation of jobs object
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
- over (bool) – If True, show only entities that are over their limit.Default is False.
Returns: A list of dictionary similar to that returned by a converted batch_status object, i.e., can be displayed using the Utils.show method
-
clusterize
(conf_file=None, hosts=None, import_jobs=False, db_creds_file=None)¶ Mimic a
pbs_diag
snapshot onto a set of hosts running a PBSserver
,``scheduler``, andMoM
.This method clones the following information from the diag:
Server attributes
Server resourcedef
Hooks
Scheduler configuration
Scheduler resource_group
Scheduler holiday file
Per Queue attributes
Nodes are copied as a vnode definition file inserted into each host’s MoM instance.
Currently no support for cloning the server ‘sched’ object, nor to copy nodes to multi-mom instances.
Jobs are copied over only if import_jobs is True, see below for details
Parameters: - asdiag (str) – Path to the pbs_diag snapshot to use
- conf_file – Configuration file for the MoM instance
- hosts (List) – List of hosts on which to clone the diag snapshot
- include_jobs (bool) – [Experimental] if True jobs from the pbs_diag are imported into the host’s database. There are several caveats to this option: The scripts are not imported The users and groups are not created on the local system.There are no actual processes created on the MoM for each job so operations on the job such as signals or delete will fail (delete -W force will still work)
- db_creds_file (str or None) – Path to file containing credentials to access the DB
-
class
ptl.lib.pbs_testlib.
EquivClass
(name, attributes={}, entities=[])¶ Equivalence class holds information on a collection of entities grouped according to a set of attributes :param attributes: Dictionary of attributes :type attributes: Dictionary :param entities: List of entities :type entities: List
-
add_entity
(entity)¶ Add entities
Parameters: entity (str) – Entity to add
-
show
(showobj=False)¶ Show the entities
Parameters: showobj (bool) – If true then show the entities
-
-
class
ptl.lib.pbs_testlib.
Resource
(name=None, type=None, flag=None)¶ PBS resource referenced by name, type and flag
Parameters: - name (str or None) – Resource name
- type – Type of resource
-
set_name
(name)¶ Set the resource name
-
set_type
(type)¶ Set the resource type
-
set_flag
(flag)¶ Set the flag
-
class
ptl.lib.pbs_testlib.
Holidays
¶ Descriptive calss for Holiday file.
-
class
ptl.lib.pbs_testlib.
Scheduler
(hostname=None, server=None, pbsconf_file=None, diagmap={}, diag=None, db_access=None, id='default', sched_priv=None)¶ Container of Scheduler related properties
Parameters: - hostname (str or None) – The hostname on which the scheduler instance is operating
- server – A PBS server instance to which this scheduler is associated
- pbsconf_file (str or None) – path to a PBS configuration file
- diagmap (Dictionary) – A dictionary of PBS objects (node,server,etc) to mapped files from PBS diag directory
- diag (str or None) – path to PBS diag directory (This will overrides diagmap)
- db_acccess – set to either file containing credentials
to DB access or dictionary containing
{'dbname':...,'user':...,'port':...}
-
setup_sched_priv
(sched_priv=None)¶ Initialize Scheduler() member variables on initialization or if sched_priv changes
-
initialise_service
()¶ initialise the scheduler object
-
isUp
()¶ Check for PBS scheduler up
-
signal
(sig)¶ Send a signal to PBS scheduler
-
get_pid
()¶ Get the PBS scheduler pid
-
all_instance_pids
()¶ Get the all pids for the instance
-
start
(sched_home=None, args=None, launcher=None)¶ Start the scheduler :param sched_home: Path to scheduler log and home directory :type sched_home: str :param args: Arguments required to start the scheduler :type args: str :param launcher: Optional utility to invoke the launch of the service :type launcher: str or list
-
stop
(sig=None)¶ Stop the PBS scheduler
Parameters: sig (str) – Signal to stop the PBS scheduler
-
restart
()¶ Restart the PBS scheduler
-
log_match
(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶ Match given
msg
in givenn
lines of Scheduler logParameters: - msg (str) – log message to match, can be regex also when
regexp
is True - id (str) – The id of the object to trace. Only used for tracejob
- n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
- tail (bool) – If true (default), starts from the end of the file
- allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
- regexp (bool) – If true msg is a Python regular expression. Defaults to False
- max_attempts (int) – the number of attempts to make to find a matching entry
- interval (int) – the interval between attempts
- starttime (int) – If set ignore matches that occur before specified time
- endtime (int) – If set ignore matches that occur after specified time
- level (int) – The logging level, defaults to INFO
- existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns: (x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type: tuple
Raises: PtlLogMatchError – When
existence
is True and givenmsg
is not found inn
line Or Whenexistence
is False and givenmsg
found inn
line.Note
The matching line number is relative to the record number, not the absolute line number in the file.
- msg (str) – log message to match, can be regex also when
-
pbs_version
()¶ Get the version of the scheduler instance
-
parse_sched_config
(schd_cnfg=None)¶ Parse a sceduling configuration file into a dictionary. Special handling of identical keys
(e.g., node_sort_key)
is done by appending a delimiter, ‘%’, between each value of the key. When printed back to file, each delimited entry gets written on a line of its own. For example, the python dictionary entry:{'node_sort_key': ["ncpus HIGH unusued" prime", "node_priority HIH" non-prime"]}
will get written as:
node_sort_key: "ncpus HIGH unusued" prime
node_sort_key: "node_priority HIGH" non-prime
Returns sched_config dictionary that gets reinitialized every time this method is called.
-
check_defaults
(config)¶ Check the values in argument config against default values
-
apply_config
(config=None, validate=True, path=None)¶ Apply the configuration specified by config
Parameters: - config – Configurations to set. Default: self. sched_config
- validate (bool) – If True (the default) validate that settings did not yield an error. Validation is done by parsing the scheduler log which, in some cases may be slow and therefore undesirable.
- path (str) – Optional path to file to which configuration is written. If None, the configuration is written to PBS_HOME/sched_priv/sched_config
Returns: True on success and False otherwise. Success means that upon applying the new configuration the scheduler did not emit an “Error reading line” in its log file.
-
set_sched_config
(confs={}, apply=True, validate=True)¶ set a
sched_config
propertyParameters: - confs (Dictionary) – dictionary of key value sched_config entries
- apply (bool) – if True (the default), apply configuration.
- validate (bool) – if True (the default), validate the configuration settings.
-
add_server_dyn_res
(custom_resource, script_body=None, res_file=None, apply=True, validate=True, dirname=None, host=None, perm=448, prefix='PtlPbsSvrDynRes', suffix='.scr')¶ Add a root owned server dynamic resource script or file to the scheduler configuration.
Parameters: - custom_resource (str) – The name of the custom resource to define
- script_body – The body of the server dynamic resource
- res_file (str or None) – Alternatively to passing the script body, use the file instead
- apply (bool) – if True (the default), apply configuration.
- validate (bool) – if True (the default), validate the configuration settings.
- dirname (str or None) – the file will be created in this directory
- host (str or None) – the hostname on which dyn res script is created
- perm – perm to use while creating scripts (must be octal like 0777)
- prefix (str) – the file name will begin with this prefix
- suffix (str) – the file name will end with this suffix
:return Absolute path of the dynamic resource script
-
unset_sched_config
(name, apply=True)¶ Delete a
sched_config
entryParameters: - name (str) – the entry to delete from sched_config
- apply (bool) – if True, apply configuration. Defaults to True
-
set_dedicated_time_file
(file)¶ Set the path to a dedicated time
-
revert_to_defaults
()¶ Revert scheduler configuration to defaults.
Returns: True on success, False otherwise
-
create_scheduler
(sched_home=None)¶ Start scheduler with creating required directories for scheduler :param sched_home: path of scheduler home and log directory :type sched_home: str
-
save_configuration
(outfile, mode='a')¶ Save scheduler configuration
Parameters: - outfile (str) – Path to a file to which configuration is saved
- mode (str) – mode to use to access outfile. Defaults to append, ‘a’.
Returns: True on success and False otherwise
-
load_configuration
(infile)¶ load configuration from saved file infile
-
get_resources
(exclude=[])¶ returns a list of allocatable resources.
Parameters: exclude (List) – if set, excludes the named resources, if they exist, from the resulting list
-
add_resource
(name, apply=True)¶ Add a resource to
sched_config
.Parameters: - name (str) – the resource name to add
- apply (bool) – if True, apply configuration. Defaults to True
Returns: True on success and False otherwise. Return True if the resource is already defined.
-
remove_resource
(name, apply=True)¶ Remove a resource to
sched_config
.Parameters: - name (str) – the resource name to remove
- apply (bool) – if True, apply configuration. Defaults to True
Returns: True on success and False otherwise
-
holidays_revert_to_default
(level=20)¶ Revert holidays file to default
-
holidays_parse_file
(path=None, obj=None, level=20)¶ Parse the existing holidays file
Parameters: - path (str or None) – optional path to the holidays file to parse
- obj – optional holidays object to be used instead of internal
Returns: The content of holidays file as a list of lines
-
holidays_set_day
(day_id, prime='', nonprime='', apply=True, obj=None, level=20)¶ Set prime time values for a day
Parameters: - day_id (str) – the day to be set (string)
- prime – the prime time value
- nonprime – the non-prime time value
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
Returns: The position
(0-7)
of the set day
-
holidays_get_day
(day_id, obj=None, level=20)¶ Parameters: - obj – optional holidays object to be used instead of internal
- day_id (str) – either a day’s name or “all”
Returns: A copy of info about a day/all set days
-
holidays_reposition_day
(day_id, new_pos, apply=True, obj=None, level=20)¶ Change position of a day
(0-7)
as it appears in the holidays fileParameters: - day_id (str) – name of the day
- new_pos – new position
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
Returns: The new position of the day
-
holidays_unset_day
(day_id, apply=True, obj=None, level=20)¶ Unset prime time values for a day
Parameters: - day_id (str) – day to unset (string)
- apply – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
Note
we do not unset the ‘valid’ field here so the entry will still be displayed but without any values
-
holidays_invalidate_day
(day_id, apply=True, obj=None, level=20)¶ Remove a day’s entry from the holidays file
Parameters: - day_id (str) – the day to remove (string)
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
-
holidays_validate_day
(day_id, apply=True, obj=None, level=20)¶ Make valid a previously set day’s entry
Parameters: - day_id (str) – the day to validate (string)
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
Note
The day will retain its previous position in the file
-
holidays_delete_entry
(entry_type, idx=None, apply=True, obj=None, level=20)¶ Delete
one/all
entries from holidays fileParameters: - entry_type (str) – ‘y’:year, ‘d’:day, ‘h’:holiday or ‘a’: all
- idx (str or None) – either a day of week (monday, tuesday etc.) or Julian date of a holiday
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
Returns: False if entry_type is invalid, otherwise True
Note
The day cannot be validated and will lose it’s position in the file
-
holidays_set_year
(new_year='', apply=True, obj=None, level=20)¶ Set the year value
Parameters: - newyear (str) – year value to set
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
-
holidays_unset_year
(apply=True, obj=None, level=20)¶ Unset the year value
Parameters: - apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
-
holidays_get_year
(obj=None, level=20)¶ Parameters: obj – optional holidays object to be used instead of internal Returns: The year entry of holidays file
-
holidays_add_holiday
(date=None, apply=True, obj=None, level=20)¶ Add a calendar holiday to the holidays file
Parameters: - date – Date value for the holiday
- apply (bool) – to reflect the changes to file
- obj – optional holidays object to be used instead of internal
-
holidays_get_holidays
(obj=None, level=20)¶ Parameters: obj – optional holidays object to be used instead of internal Returns: The list of holidays in holidays file
-
holidays_write_file
(content=None, out_path=None, hup=True, obj=None, level=20)¶ Write to the holidays file with content
given/generated
Parameters: - hup (bool) – SIGHUP the scheduler after writing the holidays file
- obj – optional holidays object to be used instead of internal
-
parse_dedicated_time
(file=None)¶ Parse the dedicated_time file and populate dedicated times as both a string dedicated_time array of dictionaries defined as
[{'from': datetime, 'to': datetime}, ...]
as well as a dedicated_time_as_str array with a string representation of each entryParameters: file (str or None) – optional file to parse. Defaults to the one under PBS_HOME/sched_priv
Returns: The dedicated_time list of dictionaries or None on error.Return an empty array if dedicated time file is empty.
-
clear_dedicated_time
(hup=True)¶ Clear the dedicated time file
-
add_dedicated_time
(as_str=None, start=None, end=None, hup=True)¶ Append a dedicated time entry. The function can be called in one of two ways, either by passing in start and end as time values, or by passing as_str, a string that gets appended to the dedicated time entries and formatted as follows, note that no check on validity of the format will be made the function uses strftime to parse the datetime and will fail if the strftime can not convert the string.
MM/DD/YYYY HH:MM MM/DD/YYYY HH:MM
Returns: True on success and False otherwise
-
valgrind
()¶ run scheduler instance through valgrind
-
alloc_to_execvnode
(chunks)¶ convert a resource allocation to an execvnode string representation
-
cycles
(start=None, end=None, firstN=None, lastN=None)¶ Analyze scheduler log and return cycle information
Parameters: - start – Optional setting of the start time to consider
- end – Optional setting of the end time to consider
- firstN – Optional setting to consider the given first N cycles
- lastN – Optional setting to consider only the given last N cycles
Parse fairshare data using
pbsfs
and populates fairshare_tree.If name or id are specified, return the data associated to that id.Otherwise return the entire fairshare tree
Set the fairshare usage associated to a given entity.
Parameters: - name (str or
PbsUser
or None) – The entity to set the fairshare usage of - usage – The usage value to set
- name (str or
Decay the fairshare tree through pbsfs
Compare two fairshare entities. Wrapper of
pbsfs -c e1 e2
Parameters: Returns: the name of the entity of higher priority or None on error
-
parse_resource_group
(hostname=None, resource_group=None)¶ Parse the Scheduler’s
resource_group
fileParameters: - hostname (str or None) – The name of the host from which to parse resource_group
- resource_group (str or None) – The path to a resource_group file
Returns: A fairshare tree
-
add_to_resource_group
(name, fairshare_id, parent, nshares, validate=True)¶ Add an entry to the resource group file
Parameters: - name (str or
PbsUser
) – The name of the entity to add - fairshare_id (int) – The numeric identifier of the entity to add
- parent (str) – The name of the parent group
- nshares (int) – The number of shares associated to the entity
- validate (bool) – if True (the default), validate the configuration settings.
- name (str or
-
job_formula
(jobid=None, starttime=None, max_attempts=5)¶ Extract formula value out of scheduler log
Parameters: - jobid (str or int) – Optional, the job identifier for which to get the formula.
- starttime – The time at which to start parsing the scheduler log
- max_attempts (int) – The number of attempts to search for formula in the logs
Returns: If jobid is specified, return the formula value associated to that job if no jobid is specified, returns a dictionary mapping job ids to formula
Object representation of the Scheduler’s resource_group file and pbsfs data
Parameters: hostname (str) – Hostname of the machine add node to the fairshare tree
Add an entry to the
resource_group
fileParameters: - name (str) – The name of the entity to add
- id (int) – The uniqe numeric identifier of the entity
- parent (str) – The name of the parent/group of the entity
- nshares (int) – The number of shares assigned to this entity
Returns: True on success, False otherwise
Return a node of the fairshare tree identified by either name or id.
Parameters: - name (str or None) – The name of the entity to query
- id – The id of the entity to query
Returns: The fairshare information of the entity when found, if not, returns None
Note
The name takes precedence over the id.
Object representation of the fairshare data as queryable through the command
pbsfs
.Parameters: - name (str or None) – Name of fairshare node
- nshares (int or None) – Number of shares
- usage – Fairshare usage
- perc – Percentage the entity has of the tree
-
class
ptl.lib.pbs_testlib.
MoM
(name=None, attrs={}, pbsconf_file=None, diagmap={}, diag=None, server=None, db_access=None)¶ Container for MoM properties. Provides various MoM operations, such as creation, insertion, deletion of vnodes.
Parameters: - name (str or None) – The hostname of the server. Defaults to calling pbs_default()
- attrs (Dictionary) – Dictionary of attributes to set, these will override defaults.
- pbsconf_file (str or None) – path to config file to parse for
PBS_HOME
,PBS_EXEC
, etc - diagmap (Dictionary) – A dictionary of PBS objects
(node,server,etc)
to mapped files from PBS diag directory - diag (str or None) – path to PBS diag directory (This will overrides diagmap)
- server – A PBS server instance to which this mom is associated
- db_acccess – set to either file containing credentials to DB access or dictionary containing {‘dbname’:…,’user’:…,’port’:…}
-
isUp
(max_attempts=20)¶ Check for PBS mom up
-
signal
(sig)¶ Send signal to PBS mom
-
get_pid
()¶ Get the PBS mom pid
-
all_instance_pids
()¶ Get all pids of a instance
-
start
(args=None, launcher=None)¶ Start the PBS mom
Parameters: - args (str or None) – Arguments to start the mom
- launcher (str or list or None) – Optional utility to invoke the launch of the service
-
stop
(sig=None)¶ Stop the PBS mom
Parameters: sig (str) – Signal to stop the PBS mom
-
restart
()¶ Restart the PBS mom
-
log_match
(msg=None, id=None, n=50, tail=True, allmatch=False, regexp=False, max_attempts=None, interval=None, starttime=None, endtime=None, level=20, existence=True)¶ Match given
msg
in givenn
lines of MoM logParameters: - msg (str) – log message to match, can be regex also when
regexp
is True - id (str) – The id of the object to trace. Only used for tracejob
- n (str or int) – ‘ALL’ or the number of lines to search through, defaults to 50
- tail (bool) – If true (default), starts from the end of the file
- allmatch (bool) – If True all matching lines out of then parsed are returned as a list. Defaults to False
- regexp (bool) – If true msg is a Python regular expression. Defaults to False
- max_attempts (int) – the number of attempts to make to find a matching entry
- interval (int) – the interval between attempts
- starttime (int) – If set ignore matches that occur before specified time
- endtime (int) – If set ignore matches that occur after specified time
- level (int) – The logging level, defaults to INFO
- existence (bool) – If True (default), check for existence of given msg, else check for non-existence of given msg.
Returns: (x,y) where x is the matching line number and y the line itself. If allmatch is True, a list of tuples is returned.
Return type: tuple
Raises: PtlLogMatchError – When
existence
is True and givenmsg
is not found inn
line Or Whenexistence
is False and givenmsg
found inn
line.Note
The matching line number is relative to the record number, not the absolute line number in the file.
- msg (str) – log message to match, can be regex also when
-
pbs_version
()¶ Get the PBS version
-
revert_to_defaults
(delvnodedefs=True)¶ Revert MoM configuration to defaults.
Remove epilogue and prologue
3.
Delete all vnode definitions HUP MoM
Parameters: delvnodedefs (bool) – if True (the default) delete all vnode definitions and restart the MoM Returns: True on success and False otherwise
-
save_configuration
(outfile, mode='a')¶ Save a MoM
mom_priv/config
Parameters: - outfile (str) – the output file to which onfiguration is saved
- mode (str) – the mode in which to open outfile to save configuration.
Returns: True on success, False on error
Note
first object being saved should open this file with ‘w’ and subsequent calls from other objects should save with mode ‘a’ or ‘a+’. Defaults to a+
-
load_configuration
(infile)¶ load configuration from saved file infile
-
is_cray
()¶ Returns True if the version of PBS used was built for Cray platforms
-
is_cpuset_mom
()¶ Check for cpuset mom
-
create_vnode_def
(name, attrs={}, numnodes=1, sharednode=True, pre='[', post=']', usenatvnode=False, attrfunc=None, vnodes_per_host=1)¶ Create a vnode definition string representation
Parameters: - name (str) – The prefix for name of vnode to create, name of vnode will be prefix + pre + <num> + post
- attrs (Dictionary) – Dictionary of attributes to set on each vnode
- numnodes (int) – The number of vnodes to create
- sharednode (bool) – If true vnodes are shared on a host
- pre (str) – The symbol preceding the numeric value of that vnode.
- post (str) – The symbol following the numeric value of that vnode.
- usenatvnode (bool) – use the natural vnode as the first vnode to allocate this only makes sense starting with PBS 11.3 when natural vnodes are reported as a allocatable
- attrfunc – function to customize the attributes, signature is (name, numnodes, curnodenum, attrs), must return a dict that contains new or modified attrs that will be added to the vnode def. The function is called once per vnode being created, it does not modify attrs itself across calls.
- vnodes_per_host (int) – number of vnodes per host
Returns: A string representation of the vnode definition file
-
parse_config
()¶ Parse mom config file into a dictionary of configuration options.
Returns: A dictionary of configuration options on success, and None otherwise
-
add_config
(conf={}, hup=True)¶ Add config options to mom_priv_config.
Parameters: - conf (Dictionary) – The configurations to add to
mom_priv/config
- hup (bool) – If True (default)
HUP
the MoM
Returns: True on success and False otherwise
- conf (Dictionary) – The configurations to add to
-
unset_mom_config
(name, hup=True)¶ Delete a mom_config entry
Parameters: - name (String) – The entry to remove from
mom_priv/config
- hup (bool) – if True (default)
HUP
the MoM
Returns: True on success and False otherwise
- name (String) – The entry to remove from
-
apply_config
(conf={}, hup=True, restart=False)¶ Apply configuration options to MoM.
Parameters: - conf (Dictionary) – A dictionary of configuration options to apply to MoM
- hup (bool) – If True (default) , HUP the MoM to apply the configuration
Returns: True on success and False otherwise.
-
get_vnode_def
(vnodefile=None)¶ Returns: A vnode def file as a single string
-
insert_vnode_def
(vdef, fname=None, additive=False, restart=True)¶ Insert and enable a vnode definition. Root privilege is required
Parameters: - vdef (str) – The vnode definition string as created by create_vnode_def
- fname (str or None) – The filename to write the vnode def string to
- additive (bool) – If True, keep all other vnode def files under config.d Default is False
- delete (bool) – If True, delete all nodes known to the server. Default is True
- restart (bool) – If True, restart the MoM. Default is True
-
has_vnode_defs
()¶ Check for vnode definition(s)
-
delete_vnode_defs
(vdefname=None)¶ delete vnode definition(s) on this MoM
Parameters: vdefname (str) – name of a vnode definition file to delete, if None all vnode definitions are deleted Returns: True if delete succeed otherwise False
-
has_pelog
(filename=None)¶ Check for prologue and epilogue
-
has_prologue
()¶ Check for prologue
-
has_epilogue
()¶ Check for epilogue
-
delete_pelog
()¶ Delete any prologue and epilogue files that may have been defined on this MoM
-
create_pelog
(body=None, src=None, filename=None)¶ create
prologue
andepilogue
files, functionality accepts either a body of the script or a source file.Returns: True on success and False on error
-
prologue
(body=None, src=None)¶ create prologue
-
epilogue
(body=None, src=None)¶ Create epilogue
-
action
(act, script)¶ Define action script. Not currently implemented
-
add_mom_dyn_res
(custom_resource, script_body=None, res_file=None, dirname=None, host=None, perm=448, prefix='PtlPbsMomDynRes', suffix='.scr')¶ Add a root owned mom dynamic resource script or file to the mom configuration
Parameters: - custom_resource (str) – The name of the custom resource to define
- script_body – The body of the mom dynamic resource
- res_file (str or None) – Alternatively to passing the script body, use the file instead
- dirname (str or None) – the file will be created in this directory
- host (str or None) – the hostname on which dyn res script is created
- perm – perm to use while creating scripts (must be octal like 0777)
- prefix (str) – the file name will begin with this prefix
- suffix (str) – the file name will end with this suffix
:return Absolute path of the dynamic resource script
-
class
ptl.lib.pbs_testlib.
Hook
(name=None, attrs={}, server=None)¶ PBS hook objects. Holds attributes information and pointer to server
Parameters: - name (str or None) – Hook name
- attrs (Dictionary) – Hook attributes
- server – Pointer to server
-
class
ptl.lib.pbs_testlib.
ResourceResv
(name, attrs={}, defaults={})¶ Generic PBS resource reservation, i.e., job or
advance/standing
reservation-
execvnode
(attr='exec_vnode')¶ PBS type execution vnode
-
exechost
()¶ PBS type execution host
-
resvnodes
()¶ nodes assigned to a reservation
-
classmethod
get_hosts
(exechost=None)¶ Returns: The hosts portion of the exec_host
-
get_vnodes
(execvnode=None)¶ Returns: The unique vnode names of an execvnode as a list
-
-
class
ptl.lib.pbs_testlib.
Job
(username=None, attrs={}, jobname=None)¶ PBS Job. Attributes and Resources
Parameters: - username (str or None) – Job username
- attrs (Dictionary) – Job attributes
- jobname (str or None) – Name of the PBS job
-
add_cray_vntype
(select=None)¶ Cray specific function to add vntype as
cray_compute
to each select chunkParameters: select (str or None) – PBS select statement
-
set_attributes
(a={})¶ set attributes and custom attributes on this job. custom attributes are used when converting attributes to CLI. In case of Cray platform if ‘Resource_List.vntype’ is set already then remove it and add vntype value to each chunk of a select statement.
Parameters: a (Dictionary) – Attribute dictionary
-
set_variable_list
(user=None, workdir=None)¶ Customize the
Variable_List
job attribute to<user>
-
set_sleep_time
(duration)¶ Set the sleep duration for this job.
Parameters: duration (int) – The duration, in seconds, to sleep
-
set_execargs
(executable, arguments=None)¶ Set the executable and arguments to use for this job
Parameters: - executable (str) – path to an executable. No checks are made.
- arguments (str or list or int) – arguments to executable.
-
create_script
(body=None, asuser=None, hostname=None)¶ Create a job script from a given body of text into a temporary location
Parameters: - body (str or None) – the body of the script
- asuser (str or None) – Optionally the user to own this script, defaults ot current user
- hostname (str or None) – The host on which the job script is to be created
-
create_subjob_id
(job_array_id, subjob_index)¶ insert subjob index into the square brackets of job array id
Parameters: - job_array_id (str) – PBS parent array job id
- subjob_index (int) – index of subjob
Returns: subjob id string
-
create_eatcpu_job
(duration=None)¶ Create a job that eats cpu indefinitely or for the given duration of time
-
class
ptl.lib.pbs_testlib.
Reservation
(username=None, attrs={})¶ PBS Reservation. Attributes and Resources
Parameters: attrs (Dictionary) – Reservation attributes
-
class
ptl.lib.pbs_testlib.
InteractiveJob
(job, cmd, host)¶ An Interactive Job thread
Interactive Jobs are submitted as a thread that sets the jobid as soon as it is returned by
qsub -I
, such that the caller can get back to monitoring the state of PBS while the interactive session goes on in the thread.The commands to be run within an interactive session are specified in the job’s interactive_script attribute as a list of tuples, where the first item in each tuple is the command to run, and the subsequent items are the expected returned data.
Implementation details:
Support for interactive jobs is currently done through the pexpect module which must be installed separately from PTL. Interactive jobs are submitted through
CLI
only, there is no API support for this operation yet.The submission of an interactive job requires passing in job attributes,the command to execute
(i.e. path to qsub -I)
and the hostnamewhen not impersonating:
pexpect spawns the
qsub -I
command and expects a prompt back, for each tuple in the interactive_script, it sends the command and expects to match the return value.when impersonating:
pexpect spawns
sudo -u <user> qsub -I
. The rest is as described in non- impersonating mode.-
run
()¶ Run the interactive job
-
-
class
ptl.lib.pbs_testlib.
Queue
(name=None, attrs={}, server=None)¶ PBS Queue container, holds attributes of the queue and pointer to server
Parameters: - name (str or None) – Queue name
- attrs (Dictionary) – Queue attributes
-
revert_to_defaults
()¶ reset queue attributes to defaults
-
class
ptl.lib.pbs_testlib.
PBSInitServices
(hostname=None, conf=None)¶ PBS initialization services
Parameters: - hostname (str or None) – Machine hostname
- conf (str or None) – PBS configuaration file
-
initd
(hostname=None, op='status', conf_file=None, init_script=None, daemon='all')¶ Run the init script for a given operation
Parameters: - hostname (str or None) – hostname on which to execute the init script
- op (str) – one of status, start, stop, restart
- conf_file (str or None) – optional path to a configuration file
- init_script (str or None) – optional path to a PBS init script
- daemon (str) – name of daemon to operate on. one of server, mom, sched, comm or all
-
restart
(hostname=None, init_script=None)¶ Run the init script for a restart operation
Parameters: - hostname (str or None) – hostname on which to execute the init script
- init_script (str or None) – optional path to a PBS init script
-
restart_server
(hostname=None, init_script=None)¶ Run the init script for a restart server
Parameters: - hostname (str or None) – hostname on which to restart server
- init_script (str or None) – optional path to a PBS init script
-
restart_mom
(hostname=None, init_script=None)¶ Run the init script for a restart mom
Parameters: - hostname (str or None) – hostname on which to restart mom
- init_script (str or None) – optional path to a PBS init script
-
restart_sched
(hostname=None, init_script=None)¶ Run the init script for a restart sched
Parameters: - hostname (str or None) – hostname on which to restart sched
- init_script (str or None) – optional path to a PBS init script
-
restart_comm
(hostname=None, init_script=None)¶ Run the init script for a restart comm
Parameters: - hostname (str or None) – hostname on which to restart comm
- init_script (str or None) – optional path to a PBS init script
-
start
(hostname=None, init_script=None)¶ Run the init script for a start operation
Parameters: - hostname (str or None) – hostname on which to execute the init script
- init_script (str or None) – optional path to a PBS init script
-
start_server
(hostname=None, init_script=None)¶ Run the init script for a start server
Parameters: - hostname (str or None) – hostname on which to start server
- init_script (str or None) – optional path to a PBS init script
-
start_mom
(hostname=None, init_script=None)¶ Run the init script for a start mom
Parameters: - hostname (str or None) – hostname on which to start mom
- init_script (str or None) – optional path to a PBS init script
-
start_sched
(hostname=None, init_script=None)¶ Run the init script for a start sched
Parameters: - hostname (str or None) – hostname on which to start sched
- init_script (str or None) – optional path to a PBS init script
-
start_comm
(hostname=None, init_script=None)¶ Run the init script for a start comm
Parameters: - hostname (str or None) – hostname on which to start comm
- init_script (str or None) – optional path to a PBS init script
-
stop
(hostname=None, init_script=None)¶ Run the init script for a stop operation
Parameters: - hostname (str or None) – hostname on which to execute the init script
- init_script (str or None) – optional path to a PBS init script
-
stop_server
(hostname=None, init_script=None)¶ Run the init script for a stop server
Parameters: - hostname (str or None) – hostname on which to stop server
- init_script (str or None) – optional path to a PBS init script
-
stop_mom
(hostname=None, init_script=None)¶ Run the init script for a stop mom
Parameters: - hostname (str or None) – hostname on which to stop mom
- init_script (str or None) – optional path to a PBS init script
-
stop_sched
(hostname=None, init_script=None)¶ Run the init script for a stop sched
Parameters: - hostname (str or None) – hostname on which to stop sched
- init_script (str or None) – optional path to a PBS init script
-
stop_comm
(hostname=None, init_script=None)¶ Run the init script for a stop comm
Parameters: - hostname (str or None) – hostname on which to stop comm
- init_script (str or None) – optional path to a PBS init script
-
status
(hostname=None, init_script=None)¶ Run the init script for a status operation
Parameters: - hostname (str or None) – hostname on which to execute the init script
- init_script (str or None) – optional path to a PBS init script
-
status_server
(hostname=None, init_script=None)¶ Run the init script for a status server
Parameters: - hostname (str or None) – hostname on which to status server
- init_script (str or None) – optional path to a PBS init script
-
status_mom
(hostname=None, init_script=None)¶ Run the init script for a status mom
Parameters: - hostname (str or None) – hostname on which to status mom
- init_script (str or None) – optional path to a PBS init script
-
status_sched
(hostname=None, init_script=None)¶ Run the init script for a status sched
Parameters: - hostname (str or None) – hostname on which to status sched
- init_script (str or None) – optional path to a PBS init script
-
status_comm
(hostname=None, init_script=None)¶ Run the init script for a status comm
Parameters: - hostname (str or None) – hostname on which to status comm
- init_script (str or None) – optional path to a PBS init script
-
switch_version
(hostname=None, version=None)¶ Switch to another version of PBS installed on the system
Parameters: - hostname (str or None) – The hostname to operate on
- version – version to switch
1.2. Utilities¶
1.2.1. pbs_cliutils¶
-
class
ptl.utils.pbs_cliutils.
CliUtils
¶ Command line interface utility
-
classmethod
get_logging_level
(level)¶ Get the logging levels
Parameters: level (str) – Name of the logging level
-
static
check_bin
(bin_name)¶ Check for command exist
Parameters: bin_name (str) – Command to be checked Returns: True if command exist else return False
-
static
expand_abs_path
(path)¶ Expand the path to absolute path
-
static
priv_ports_info
(hostname=None)¶ Return a list of privileged ports in use on a given host
Parameters: hostname (str or None) – The host on which to query privilege ports usage. Defaults to the local host
-
classmethod
1.2.2. pbs_covutils¶
-
class
ptl.utils.pbs_covutils.
LcovUtils
(cov_bin=None, html_bin=None, cov_out=None, data_dir=None, html_nosrc=False, html_baseurl=None)¶ Coverage Utils
Parameters: - cov_bin – Coverage binary
- html_bin – Coverage html binary
- cov_out (str or None) – Coverage output directory
- data_dir (str or None) – Coverage data directory
- html_nosrc – HTML reports without PBS source
- html_baseurl (str or None) – HTML base url
-
set_html_baseurl
(baseurl)¶ Set
HTML
base url
-
set_html_nosource
(nosource=False)¶ Set HTML nosource parameter to False.
-
set_coverage_bin
(cov_bin=None)¶ Set the coverage binary
Parameters: cov_binary – Coverage bin to set
-
set_genhtml_bin
(html_bin=None)¶ Set HTML generation utility.
Parameters: html_bin – HTML bin to set
-
set_coverage_out
(cov_out=None)¶ Set the coverage output directory
Parameters: cov_out – Coverage output directory path.
-
set_coverage_data_dir
(data=None)¶ Set the coverage data directory
Parameters: data – Data directory path Returns: True if file name ends with .gcno else return False
-
add_trace
(trace)¶ Add coverage trace
Parameters: trace – Coverage trace
-
create_coverage_data_files
(path)¶ Create .gcda counterpart files for every .gcno file and give it read/write permissions
-
initialize_coverage
(out=None, name=None)¶ Initialize coverage
Parameters: - out (str or None) – Output path
- name (str or None) – name of the command
-
capture_coverage
(out=None, name=None)¶ Capture the coverage parameters
-
zero_coverage
()¶ Zero the data counters. Note that a process would need to be restarted in order to collect data again, running
--initialize
will not get populate the data counters
-
merge_coverage_traces
(out=None, name=None, exclude=None)¶ Merge the coverage traces
-
generate_html
(out=None, html_out=None, html_nosrc=False)¶ Generate the
HTML
report
-
change_baseurl
(html_out=None, html_baseurl=None)¶ Change the
HTML
base url
-
summarize_coverage
(out=None)¶ Summarize the coverage output
1.2.3. pbs_dshutils¶
-
exception
ptl.utils.pbs_dshutils.
PbsConfigError
(message=None, rv=None, rc=None, msg=None)¶ Initialize PBS configuration error
-
exception
ptl.utils.pbs_dshutils.
PtlUtilError
(message=None, rv=None, rc=None, msg=None)¶ Initialize PTL Util error
-
class
ptl.utils.pbs_dshutils.
DshUtils
¶ PBS shell utilities
A set of tools to run commands, copy files, get process information and parse a PBS configuration on an arbitrary host
-
get_platform
(hostname=None, pyexec=None)¶ Get a local or remote platform info, essentially the value of Python’s sys.platform, in case of Cray it will return a string as “cray” for actual Cray cluster and “craysim” for Cray ALPS simulator
Parameters: - hostname (str or None) – The hostname to query for platform info
- pyexec (str or None) – A path to a Python interpreter to use to query a remote host for platform info
For efficiency the value is cached and retrieved from the cache upon subsequent request
-
get_uname
(hostname=None, pyexec=None)¶ Get a local or remote platform info in uname format, essentially the value of Python’s platform.uname :param hostname: The hostname to query for platform info :type hostname: str or None :param pyexec: A path to a Python interpreter to use to query
a remote host for platform infoFor efficiency the value is cached and retrieved from the cache upon subsequent request
-
get_os_info
(hostname=None, pyexec=None)¶ Get a local or remote OS info
Parameters: - hostname (str or None) – The hostname to query for platform info
- pyexec (str or None) – A path to a Python interpreter to use to query a remote host for platform info
Returns: a ‘str’ object containing os info
-
get_pbs_conf_file
(hostname=None)¶ Get the path of the pbs conf file. Defaults back to
/etc/pbs.conf
if unsuccessfulParameters: hostname (str or None) – Hostname of the machine Returns: Path to pbs conf file
-
parse_pbs_config
(hostname=None, file=None)¶ initialize
pbs_conf
dictionary by parsing pbs config fileParameters: file (str or None) – PBS conf file
-
set_pbs_config
(hostname=None, fin=None, fout=None, append=True, confs=None)¶ Set
environment/configuration
variables in apbs.conf
fileParameters: - hostname (str or None) – the name of the host on which to operate
- fin (str or None) – the input pbs.conf file
- fout (str or None) – the name of the output pbs.conf file, defaults
to
/etc/pbs.conf
- append (boolean) – whether to append to fout or not, defaults to True
- confs (Dictionary or None) – The
key/value
pairs to create
-
unset_pbs_config
(hostname=None, fin=None, fout=None, confs=None)¶ Unset
environment/configuration
variables in a pbs.conf fileParameters: - hostname (str or None) – the name of the host on which to operate
- fin (str or None) – the input pbs.conf file
- fout (str or None) – the name of the output pbs.conf file, defaults
to
/etc/pbs.conf
- confs (List or str or dict or None) – The configuration keys to unset
-
get_pbs_server_name
(pbs_conf=None)¶ Return the name of the server which may be different than
PBS_SERVER
,in order, this method looks atPBS_PRIMARY
,PBS_SERVER_HOST_NAME
, andPBS_LEAF_NAME
, andPBS_SERVER
-
parse_pbs_environment
(hostname=None, file='/var/spool/pbs/pbs_environment')¶ Initialize pbs_conf dictionary by parsing pbs config file
-
set_pbs_environment
(hostname=None, fin='/var/spool/pbs/pbs_environment', fout=None, append=True, environ=None)¶ Set the PBS environment
Parameters: - environ (dict or None) – variables to set
- hostname (str or None) – Hostname of the machine
- fin (str) – pbs_environment input file
- fout (str or None) – pbs_environment output file
- append (bool) – whether to append to fout or not, defaults defaults to true
-
unset_pbs_environment
(hostname=None, fin='/var/spool/pbs/pbs_environment', fout=None, environ=None)¶ Unset environment variables in a pbs_environment file
Parameters: - hostname (str or None) – the name of the host on which to operate
- fin (str) – the input pbs_environment file
- fout (str or None) – the name of the output pbs.conf file, defaults
to
/var/spool/pbs/pbs_environment
- environ (List or str or dict or None) – The environment keys to unset
-
parse_rhosts
(hostname=None, user=None)¶ Parse remote host
Parameters: - hostname (str or None) – Hostname of the machine
- user (str or None) – User name
-
set_rhosts
(hostname=None, user=None, entry={}, append=True)¶ Set the remote host attributes
Parameters: - entry (Dictionary) – remote hostname user dictionary
- append (boolean) – If true append key value else not
-
map_pbs_conf_to_cmd
(cmd_map={}, pconf={})¶ Map PBS configuration parameter to command
Parameters: - cmd_map (Dictionary) – command mapping
- pconf (Dictionary) – PBS conf parameter dictionary
-
get_current_user
()¶ helper function to return the name of the current user
-
check_user_exists
(username=None, hostname=None)¶ Check if user exist or not
Parameters: - username (str or None) – Username to check
- hostname (str or None) – Machine hostname
Returns: True if exist else return False
-
check_group_membership
(username=None, uid=None, grpname=None, gid=None)¶ Checks whether a user, passed in as username or uid, is a member of a group, passed in as group name or group id.
Parameters: - username (str or None) – The username to inquire about
- uid – The uid of the user to inquire about (alternative to username)
- grpname (str or None) – The groupname to check for user membership
- gid – The group id to check for user membership (alternative to grpname)
-
group_memberships
(group_list=[])¶ Returns all group memberships as a dictionary of group names and associated memberships
-
get_id_info
(user)¶ Return user info in dic format obtained by
"id -a <user>"
command for given userParameters: user (str) – The username to inquire about Returns: dic format: {
“uid”: <uid of given user>,”gid”: <gid of given user’s primary group>,
”name”: <name of given user>,
”pgroup”: <name of primary group of given user>,
”groups”: <list of names of groups of given user>
}
-
get_tempdir
(hostname=None)¶ Returns: The temporary directory on the given host Default host is localhost.
-
run_cmd
(hosts=None, cmd=None, sudo=False, stdin=None, stdout=-1, stderr=-1, input=None, cwd=None, env=None, runas=None, logerr=True, as_script=False, wait_on_script=True, level=18)¶ Run a command on a host or list of hosts.
Parameters: - hosts (str or None) – the name of hosts on which to run the command, can be a comma-separated string or a list. Defaults to localhost
- cmd (str or None) – the command to run
- sudo (boolean) – whether to run the command as root or not. Defaults to False.
- stdin – custom stdin. Defaults to PIPE
- stdout – custom stdout. Defaults to PIPE
- stderr – custom stderr. Defaults to PIPE
- input – input to pass to the pipe on target host, e.g. PBS answer file
- cwd – working directory on local host from which command is run
- env – environment variables to set on local host
- runas – run command as given user. Defaults to calling user
- logerr (boolean) – whether to log error messages or not. Defaults to True
- as_script (boolean) – if True, run the command in a script created as a temporary file that gets deleted after being run. This is used mainly to circumvent some implementations of sudo that prevent passing environment variables through sudo.
- wait_on_script (boolean) – If True (default) waits on process launched as script to return.
Returns: error, output, return code as a dictionary:
{'out':...,'err':...,'rc':...}
-
run_copy
(hosts=None, src=None, dest=None, sudo=False, uid=None, gid=None, mode=None, env=None, logerr=True, recursive=False, runas=None, preserve_permission=True, level=18)¶ copy a file or directory to specified target hosts.
Parameters: - hosts (str or None) – the host(s) to which to copy the data. Can be a comma-separated string or a list
- src (str or None) – the path to the file or directory to copy. If
src is remote,it must be prefixed by the
hostname.
e.g. remote1:/path,remote2:/path
- dest (str or None) – the destination path.
- sudo (boolean) – whether to copy as root or not. Defaults to False
- uid – optionally change ownership of dest to the specified user id,referenced by uid number or username
- gid – optionally change ownership of dest to the
specified group
name/id
- mode – optinoally set mode bits of dest
- env – environment variables to set on the calling host
- logerr – whether to log error messages or not. Defaults to True.
- recursive (boolean) – whether to copy a directory (when true) or a file.Defaults to False.
- runas (str or None) – run command as user
- preserve_permission – Preserve file permission while copying file (cp cmd with -p flag) Defaults to True
:type preserve_permission:boolean :param level: logging level, defaults to DEBUG :type level: int :returns: {‘out’:<outdata>, ‘err’: <errdata>, ‘rc’:<retcode>}
upon and None if no source file specified
-
run_ptl_cmd
(hostname, cmd, sudo=False, stdin=None, stdout=-1, stderr=-1, input=None, cwd=None, env=None, runas=None, logerr=True, as_script=False, wait_on_script=True, level=18)¶ Wrapper method of run_cmd to run PTL command
-
classmethod
set_sudo_cmd
(cmd)¶ set the sudo command
-
classmethod
set_copy_cmd
(cmd)¶ set the copy command
-
classmethod
set_rsh_cmd
(cmd)¶ set the remote shell command
-
is_localhost
(host=None)¶ Parameters: host (str or None) – Hostname of machine Returns: true if specified host (by name) is the localhost all aliases matching the hostname are searched
-
isdir
(hostname=None, path=None, sudo=False, runas=None, level=18)¶ Parameters: - hostname (str or None) – The name of the host on which to check for directory
- path (str or None) – The path to the directory to check
- sudo (boolean) – Whether to run the command as a privileged user
- runas (str or None) – run command as user
- level – Logging level
Returns: True if directory pointed to by path exists and False otherwise
-
isfile
(hostname=None, path=None, sudo=False, runas=None, level=18)¶ Parameters: - hostname (str or None) – The name of the host on which to check for file
- path (str or None) – The path to the file to check
- sudo (boolean) – Whether to run the command as a privileged user
- runas (str or None) – run command as user
- level – Logging level
Returns: True if file pointed to by path exists, and False otherwise
-
getmtime
(hostname=None, path=None, sudo=False, runas=None, level=18)¶ Parameters: - hostname (str or None) – The name of the host on which file exists
- path (str or None) – The path to the file to get mtime
- sudo (boolean) – Whether to run the command as a privileged user
- runas (str or None) – run command as user
- level – Logging level
Returns: Modified time of given file
-
listdir
(hostname=None, path=None, sudo=False, runas=None, fullpath=True, level=18)¶ Parameters: - hostname (str or None) – The name of the host on which to list for directory
- path (str or None) – The path to directory to list
- sudo (bool) – Whether to chmod as root or not. Defaults to False
- runas (str or None) – run command as user
- fullpath (bool) – Return full paths?
- level (int) – Logging level.
Returns: A list containing the names of the entries in the directory
-
chmod
(hostname=None, path=None, mode=None, sudo=False, runas=None, recursive=False, logerr=True, level=18)¶ Generic function of chmod with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- path (str or None) – the path to the file or directory to chmod
- mode – mode to apply as octal number like 0777, 0666 etc.
- sudo (boolean) – whether to chmod as root or not. Defaults to False
- runas (str or None) – run command as user
- recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
- logerr (boolean) – whether to log error messages or not. Defaults to True.
- level – logging level, defaults to INFOCLI2
Returns: True on success otherwise False
-
chown
(hostname=None, path=None, uid=None, gid=None, sudo=False, recursive=False, runas=None, logerr=True, level=18)¶ Generic function of chown with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- path (str or None) – the path to the file or directory to chown
- uid – uid to apply (must be either user name or uid or -1)
- gid – gid to apply (must be either group name or gid or -1)
- sudo (boolean) – whether to chown as root or not. Defaults to False
- recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
- runas (str or None) – run command as user
- logerr (boolean) – whether to log error messages or not. Defaults to True.
- level – logging level, defaults to INFOCLI2
Returns: True on success otherwise False
-
chgrp
(hostname=None, path=None, gid=None, sudo=False, recursive=False, runas=None, logerr=True, level=18)¶ Generic function of chgrp with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- path (str or None) – the path to the file or directory to chown
- gid – gid to apply (must be either group name or gid or -1)
- sudo (boolean) – whether to chgrp as root or not. Defaults to False
- recursive (boolean) – whether to chmod a directory (when true) or a file.Defaults to False.
- runas (str or None) – run command as user
- logerr (boolean) – whether to log error messages or not. Defaults to True.
- level – logging level, defaults to INFOCLI2
Returns: True on success otherwise False
-
which
(hostname=None, exe=None, level=18)¶ Generic function of which with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- exe (str or None) – executable to locate (can be full path also) (if exe is full path then only basename will be used to locate)
- level – logging level, defaults to INFOCLI2
-
rm
(hostname=None, path=None, sudo=False, runas=None, recursive=False, force=False, cwd=None, logerr=True, as_script=False, level=18)¶ Generic function of rm with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- path (str or None) – the path to the files or directories to remove for more than one files or directories pass as list
- sudo (boolean) – whether to remove files or directories as root or not.Defaults to False
- runas – remove files or directories as given user Defaults to calling user
- recursive (boolean) – remove files or directories and their contents recursively
- force (boolean) – force remove files or directories
- cwd – working directory on local host from which command is run
- logerr (boolean) – whether to log error messages or not. Defaults to True.
- as_script (boolean) – if True, run the rm in a script created as a temporary file that gets deleted after being run. This is used mainly to handle wildcard in path list. Defaults to False.
- level – logging level, defaults to INFOCLI2
Returns: True on success otherwise False
-
mkdir
(hostname=None, path=None, mode=None, sudo=False, runas=None, parents=True, cwd=None, logerr=True, as_script=False, level=18)¶ Generic function of mkdir with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- path (str or None) – the path to the directories to create for more than one directories pass as list
- mode – mode to use while creating directories (must be octal like 0777)
- sudo (boolean) – whether to create directories as root or not. Defaults to False
- runas – create directories as given user. Defaults to calling user
- parents (boolean) – create parent directories as needed. Defaults to True
- cwd (str or None) – working directory on local host from which command is run
- logerr (boolean) – whether to log error messages or not. Defaults to True.
- as_script (boolean) – if True, run the command in a script created as a temporary file that gets deleted after being run. This is used mainly to handle wildcard in path list. Defaults to False.
- level – logging level, defaults to INFOCLI2
Returns: True on success otherwise False
-
cat
(hostname=None, filename=None, sudo=False, runas=None, logerr=True, level=18)¶ Generic function of cat with remote host support
Parameters: - hostname (str or None) – hostname (default current host)
- filename (str or None) – the path to the filename to cat
- sudo (boolean) – whether to create directories as root or not. Defaults to False
- runas (str or None) – create directories as given user. Defaults to calling user
- logerr (boolean) – whether to log error messages or not. Defaults to True.
Returns: output of run_cmd
-
cmp
(hostname=None, fileA=None, fileB=None, sudo=False, runas=None, logerr=True)¶ Compare two files and return 0 if they are identical or non-zero if not
Parameters: - hostname (str or None) – the name of the host to operate on
- fileA (str or None) – the first file to compare
- fileB (str or None) – the file to compare fileA to
- sudo (boolean) – run the command as a privileged user
- runas (str or None) – run the cmp command as given user
- logerr (boolean) – whether to log error messages or not. Defaults to True.
-
useradd
(name, uid=None, gid=None, shell='/bin/bash', create_home_dir=True, home_dir=None, groups=None, logerr=True, level=18)¶ Add the user
Parameters: - name (str) – User name
- shell – shell to use
- create_home_dir (boolean) – If true then create home directory
- home_dir (str or None) – path to home directory
- groups – User groups
-
userdel
(name, del_home=True, force=True, logerr=True, level=18)¶ Delete the user
Parameters: - del_home (boolean) – If true then delete user home
- force (boolean) – If true then delete forcefully
-
groupadd
(name, gid=None, logerr=True, level=18)¶ Add a group
-
create_temp_file
(hostname=None, suffix='', prefix='PtlPbs', dirname=None, text=False, asuser=None, body=None, level=18)¶ Create a temp file by calling tempfile.mkstemp
Parameters: - hostname (str or None) – the hostname on which to query tempdir from
- suffix (str) – the file name will end with this suffix
- prefix (str) – the file name will begin with this prefix
- dirname (str or None) – the file will be created in this directory
- text (boolean) – the file is opened in text mode is this is true else in binary mode
- asuser (str or None) – Optional username or uid of temp file owner
- body (str or None) – Optional content to write to the temporary file
- level (int) – logging level, defaults to INFOCLI2
-
mkdtemp
(hostname=None, suffix='', prefix='PtlPbs', dir=None, uid=None, gid=None, mode=None, level=18)¶ Create a temp dir by calling
tempfile.mkdtemp
Parameters: - hostname (str or None) – the hostname on which to query tempdir from
- suffix (str) – the directory name will end with this suffix
- prefix (str) – the directory name will begin with this prefix
- dir (str or None) – the directory will be created in this directory
- uid – Optional username or uid of temp directory owner
- gid – Optional group name or gid of temp directory group owner
- mode – Optional mode bits to assign to the temporary directory
- level – logging level, defaults to INFOCLI2
-
parse_strace
(lines)¶ strace parsing. Just the regular expressions for now
-
1.2.4. pbs_fileutils¶
-
class
ptl.utils.pbs_fileutils.
FileUtils
(f, mode='head')¶ Utility to walk a file from
'head'
or'tail'
on the local filesystemParameters: - f (str) – File to process
- mode – One of FILE_HEAD or FILE_TAIL, which respectively set the file for processing from head or tail. Defaults to head.
-
get_file_descriptor
()¶ Return the file descriptor associated to the file being processed
-
set_buf_size
(bs=1024)¶ Set the buffer size to read blocks of file into
-
set_mode
(m=None)¶ Parameters: m – FILE_TAIL
if file to be tailed, andFILE_HEAD
to read from head
-
tell
()¶ Returns: The current file 'cursor'
-
get_size
()¶ Returns: The size of the file
-
get_num_lines
()¶ Returns: No of lines for the file
-
next
(n=1)¶ Get the next n lines of the file
Parameters: n (int) – the numer of lines to retrieve
-
get_line
(n=1)¶ Returns: The nth line from file
-
get_block
(from_n=1, to_n=1)¶ Returns: A block of lines between from_n
andto_n
-
next_head
(n=1)¶ Next line(s) from head
-
next_tail
(n=1)¶ Next line(s) from tail
-
head
(n=1)¶ Returns: n lines of head
-
tail
(n=1)¶ Returns: n lines of tail
1.2.5. pbs_logutils¶
-
class
ptl.utils.pbs_logutils.
PBSLogUtils
¶ Miscellaneous utilities to process log files
-
classmethod
convert_date_time
(dt=None, fmt=None)¶ convert a date time string of the form given by fmt into number of seconds since epoch (with possible microseconds). it considers the current system’s timezone to convert the datetime to epoch time
Parameters: - dt (str or None) – the datetime string to convert
- fmt (str) – Format to which datetime is to be converted
Returns: None if conversion fails
-
get_num_lines
(log, hostname=None, sudo=False)¶ Get the number of lines of particular log
Parameters: log (str) – the log file name
-
open_log
(log, hostname=None, sudo=False)¶ Parameters: - log (str) – the log file name to read from
- hostname (str or None) – the hostname from which to read the file
- sudo (boolean) – Whether to access log file as a privileged user.
Returns: A file instance
-
get_timestamps
(logfile=None, hostname=None, num=None, sudo=False)¶ Helper function to parse logfile
Returns: Each timestamp in a list as number of seconds since epoch
-
match_msg
(lines, msg, allmatch=False, regexp=False, starttime=None, endtime=None)¶ Returns (x,y) where x is the matching line y, or None if nothing is found.
Parameters: - allmatch (boolean) – If True (False by default), return a list of matching tuples.
- regexp (bool) – If True, msg is a Python regular expression. Defaults to False.
- starttime – If set ignore matches that occur before specified time
- endtime – If set ignore matches that occur after specified time
-
static
convert_resv_date_time
(date_time)¶ Convert reservation datetime to seconds
-
static
convert_hhmmss_time
(tm)¶ Convert datetime in hhmmss format to seconds
-
get_rate
(l=[])¶ Returns: The frequency of occurrences of array l The array is expected to be sorted
-
in_range
(tm, start=None, end=None)¶ Parameters: - tm – time to check within a provided range
- start – Lower limit for the time range
- end – Higer limit for the time range
Returns: True if time is in the range else return False
-
static
get_day
(tm=None)¶ Parameters: tm – Time for which to get a day
-
static
percentile
(N, percent)¶ Find the percentile of a list of values.
Parameters: - N (List) – A list of values. Note N MUST BE already sorted.
- percent (Float) – A float value from 0.0 to 1.0.
Returns: The percentile of the values
-
static
process_intervals
(intervals, groups, frequency=60)¶ Process the intervals
-
get_log_files
(hostname, path, start, end, sudo=False)¶ Parameters: - hostname (str) – Hostname of the machine
- path (str) – Path for the log file
- start – Start time for the log file
- end – End time for the log file
Returns: list of log file(s) found or an empty list
-
classmethod
-
class
ptl.utils.pbs_logutils.
PBSLogAnalyzer
(schedlog=None, serverlog=None, momlog=None, acctlog=None, genericlog=None, hostname=None, show_progress=False)¶ Utility to analyze the PBS logs
-
set_custom_match
(pattern, frequency=None)¶ Set the custome matching
Parameters: - pattern (str) – Matching pattern
- frequency (int) – Frequency of match
-
set_conditional_match
(conditions)¶ Set the conditional match
Parameters: conditions – Conditions for macthing
-
analyze_scheduler_log
(filename=None, start=None, end=None, hostname=None, summarize=True)¶ Analyze the scheduler log
Parameters: - filename (str or None) – Scheduler log file name
- start – Time from which log to be analyzed
- end – Time till which log to be analyzed
- hostname (str or None) – Hostname of the machine
- summarize (bool) – Summarize data parsed if True else not
-
analyze_server_log
(filename=None, start=None, end=None, hostname=None, summarize=True)¶ Analyze the server log
-
analyze_accounting_log
(filename=None, start=None, end=None, hostname=None, summarize=True)¶ Analyze the accounting log
-
analyze_mom_log
(filename=None, start=None, end=None, hostname=None, summarize=True)¶ Analyze the mom log
-
parse_conditional
(rec, start, end)¶ Match a sequence of regular expressions against multiple consecutive lines in a generic log. Calculate the number of conditional matching lines.
Example usage: to find the number of times the scheduler stat’ing the server causes the scheduler to miss jobs ending, which could possibly indicate a race condition between the view of resources assigned to nodes and the actual jobs running, one would call this function by setting re_conditional to
['Type 20 request received from Scheduler', 'Exit_status']
Which can be read as counting the number of times that the Type 20 message is preceded by anExit_status
message
-
analyze
(path=None, start=None, end=None, hostname=None, summarize=True, sudo=False)¶ Parse any log file. This method is not
context-specific
to each log file type.Parameters: - path (str or None) – name of
file/dir
to parse - start – optional record time at which to start analyzing
- end – optional record time after which to stop analyzing
- hostname (str or None) – name of host on which to operate. Defaults to localhost
- summarize (bool) – if True, summarize data parsed. Defaults to True.
- sudo (bool) – If True, access log file(s) as privileged user.
- path (str or None) – name of
-
analyze_logs
(schedlog=None, serverlog=None, momlog=None, acctlog=None, genericlog=None, start=None, end=None, hostname=None, showjob=False)¶ Analyze logs
-
-
class
ptl.utils.pbs_logutils.
PBSServerLog
(filename=None, hostname=None, show_progress=False)¶ Parameters: - filename (str or None) – Server log filename
- hostname (str or None) – Hostname of the machine
-
parse_runjob
(line)¶ Parse server log for run job records. For each record keep track of the job id, and time in a dedicated array
-
parse_endjob
(line)¶ Parse server log for run job records. For each record keep track of the job id, and time in a dedicated array
-
parse_nodeup
(line)¶ Parse server log for nodes that are up
-
parse_enquejob
(line)¶ Parse server log for enqued jobs
-
class
ptl.utils.pbs_logutils.
JobEstimatedStartTimeInfo
(jobid)¶ Information regarding Job estimated start time
-
add_estimate
(tm)¶ Add a job’s new estimated start time If the new estimate is now later than any preivous one, we add that difference to the drift time. If the new drift time is pulled earlier it is not added to the drift time.
drift time is a measure of
"negative perception"
that comes along a job being estimated to run at a later date than earlier"advertised"
.
-
-
class
ptl.utils.pbs_logutils.
PBSSchedulerLog
(filename=None, hostname=None, show_progress=False)¶ -
get_cycles
(start=None, end=None)¶ Get the scheduler cycles
Parameters: - start – Start time
- end – End time
Returns: Scheduling cycles
-
estimated_info_parsing
(line)¶ Parse Estimated start time information for a job
-
summarize_estimated_analysis
(estimated_jobs=None)¶ Summarize estimated job analysis
-
summary
(cycles=None, showjobs=False)¶ Scheduler log summary
-
-
class
ptl.utils.pbs_logutils.
PBSMoMLog
(filename=None, hostname=None, show_progress=False)¶ Container and Parser of a PBS
MoM
log-
summary
()¶ Mom log summary
-
-
class
ptl.utils.pbs_logutils.
PBSAccountingLog
(filename=None, hostname=None, show_progress=False)¶ Container and Parser of a PBS accounting log
-
enable_running_jobs_parsing
()¶ Enable parsing for running jobs
-
enable_utilization_parsing
(hostname=None, nodesfile=None, jobsfile=None)¶ Enable utilization parsing
Parameters: - hostname (str or None) – Hostname of the machine
- nodesfile (str or None) – optional file containing output of pbsnodes -av
- jobsfile (str or None) – optional file containing output of qstat -f
-
enable_job_info_parsing
()¶ Enable job information parsing
-
enable_accounting_workload_parsing
()¶ Enable accounting workload parsing
-
process_nodes_data
(hostname=None, nodesfile=None, jobsfile=None)¶ Get job and node information by stat’ing and parsing node data from the server. Compute the number of nodes and populate a list of running job ids on those nodes.
Parameters: - hostname (str or None) – The host to query
- nodesfile (str or None) – optional file containing output of pbsnodes -av
- jobsfile (str or None) – optional file containing output of qstat -f
The node data is needed to compute counts of nodes and cpus The job data is needed to compute the amount of resources requested
-
accounting_parsing
(rec, start, end)¶ Parsing accounting log
-
job_info
(rec)¶ PBS Job information
-
finished_jobs_nodes
(last=None, tm_range=None)¶ Parameters: - tm_range – a tuple of time where the first item is the start time
- last – If tm_range is None and last is specified, a time range from now till ‘last’ seconds from now is used as a range of time to consider and the second item is the end time to consider
Returns: A dictionary of jobs that ended in the time range as keys,and nodes (hostnames) on which those jobs were running as values.
-
summary
()¶ Accounting log summary
-
1.2.6. pbs_procutils¶
-
class
ptl.utils.pbs_procutils.
ProcUtils
¶ Utilities to query process information
-
get_ps_cmd
(hostname=None)¶ Get the ps command
Parameters: hostname (str or None) – hostname of the machine
-
get_proc_info
(hostname=None, name=None, pid=None, regexp=False)¶ Return process information from a process name, or pid, on a given host
Parameters: - hostname (str or none) – The hostname on which to query the process info. On Windows,only localhost is queried.
- name (str or None) – The name of the process to query.
- pid (int or None) – The pid of the process to query
- regexp (bool) – Match processes by regular expression. Defaults to True. Does not apply to matching by PID.
Returns: A list of ProcInfo objects, one for each matching process.
Note
If both, name and pid, are specified, name is used.
-
get_proc_state
(hostname=None, pid=None)¶ Returns: PID’s process state on host hostname On error the empty string is returned.
-
get_proc_children
(hostname=None, ppid=None)¶ Returns: A list of children PIDs associated to PPID
on host hostname.On error, an empty list is returned.
-
-
class
ptl.utils.pbs_procutils.
ProcInfo
(name=None, pid=None)¶ Process information reports
PID
,RSS
,VSZ
, Command and Time at which process information is collected
1.2.7. pbs_testsuite¶
-
ptl.utils.pbs_testsuite.
skip
(reason='Skipped test execution')¶ Unconditionally skip a test.
Parameters: reason (str or None) – Reason for the skip
-
ptl.utils.pbs_testsuite.
timeout
(val)¶ Decorator to set timeout value of test case
-
ptl.utils.pbs_testsuite.
checkModule
(modname)¶ Decorator to check if named module is available on the system and if not skip the test
-
ptl.utils.pbs_testsuite.
skipOnCray
(function)¶ Decorator to skip a test on a
Cray
system
-
ptl.utils.pbs_testsuite.
skipOnCpuSet
(function)¶ Decorator to skip a test on a CpuSet system
-
ptl.utils.pbs_testsuite.
requirements
(*args, **kwargs)¶ Decorator to provide the cluster information required for a particular testcase.
-
class
ptl.utils.pbs_testsuite.
PBSServiceInstanceWrapper
(*args, **kwargs)¶ A wrapper class to handle multiple service
(i.e., mom, server, scheduler)``instances as passed along through the test harness ``(pbs_benchpress)
.Returns an ordered dictionary of PBS service instances(i.e., mom/server/ scheduler)
Users may invoke PTL using pointers to multiple services per host, for example:
pbs_benchpress -p moms=hostA@/etc/pbs.conf,hostB,hostA@/etc/pbs.conf3
In such cases, the moms instance variable must be able to distinguish the
self.moms['hostA']
instances, each instance will be mapped to a unique configuration file-
keys
() → list of D's keys¶
-
itervalues
() → an iterator over the values of D¶
-
values
() → list of D's values¶
-
-
exception
ptl.utils.pbs_testsuite.
setUpClassError
¶
-
exception
ptl.utils.pbs_testsuite.
tearDownClassError
¶
-
class
ptl.utils.pbs_testsuite.
PBSTestSuite
(methodName='runTest')¶ Generic
setup
,teardown
, andlogging
functions to be used as parent class for most tests. Class instantiates:server object connected to localhost
scheduler objected connected to localhost
mom object connected to localhost
Custom parameters:
Parameters: - server – The hostname on which the PBS
server/scheduler
are running - mom – The hostname on which the PBS MoM is running
- servers – Colon-separated list of hostnames hosting a PBS server. Servers are then accessible as a dictionary in the instance variable servers.
- client – For CLI mode only, name of the host on which the PBS
client commands are to be run from. Format is
<host>@<path-to-config-file>
- moms – Colon-separated list of hostnames hosting a PBS MoM. MoMs are made accessible as a dictionary in the instance variable moms.
- comms – Colon-separated list of hostnames hosting a PBS Comm. Comms are made accessible as a dictionary in the instance variable comms.
- nomom=<host1>:<host2>... – expect no MoM on given set of hosts
- mode – Sets mode of operation to PBS server. Can be either
'cli'
or'api'
.Defaults to API behavior. - conn_timeout – set a timeout in seconds after which a pbs_connect IFL call is refreshed (i.e., disconnected)
- skip-setup – Bypasses setUp of PBSTestSuite (not custom ones)
- skip-teardown – Bypasses tearDown of PBSTestSuite (not custom ones)
- procinfo – Enables process monitoring thread, logged into
ptl_proc_info test metrics. The value can be set to
_all_ to monitor all PBS processes,including
pbs_server
,pbs_sched
,pbs_mom
, or a process defined by name. - revert-to-defaults=<True|False> – if False, will not revert to defaults.True by default.
- revert-hooks=<True|False> – if False, do not revert hooks to
defaults.Defaults to True.
revert-to-defaults
set to False overrides this setting. - del-hooks=<True|False> – If False, do not delete hooks. Defaults to False.``revert-to-defaults`` set to False overrides this setting.
- revert-queues=<True|False> – If False, do not revert queues to
defaults.Defaults to True.
revert-to-defaults
set to False overrides this setting. - revert-resources=<True|False> – If False, do not revert resources
to defaults. Defaults to True.
revert-to-defaults
set to False overrides this setting. - del-queues=<True|False> – If False, do not delete queues. Defaults to False.``revert-to-defaults`` set to Falseoverrides this setting.
- del-vnodes=<True|False> – If False, do not delete vnodes on MoM instances.Defaults to True.
- server-revert-to-defaults=<True|False> – if False, don’t revert Server to defaults
- comm-revert-to-defaults=<True|False> – if False, don’t revert Comm to defaults
- mom-revert-to-defaults=<True|False> – if False, don’t revert MoM to defaults
- sched-revert-to-defaults=<True|False> – if False, don’t revert Scheduler to defaults
- procmon – Enables process monitoring. Multiple values must be
colon separated. For example to monitor
server
,sched
, andmom
useprocmon=pbs_server:pbs_sched:pbs_mom
- procmon-freq – Sets a polling frequency for the process monitoring tool.Defaults to 10 seconds.
- test-users – colon-separated list of users to use as test users.
The users specified override the default users in the
order in which they appear in the
PBS_USERS
list. - default-testcase-timeout – Default test case timeout value.
- data-users – colon-separated list of data users.
- oper-users – colon-separated list of operator users.
- mgr-users – colon-separated list of manager users.
- root-users – colon-separated list of root users.
- build-users – colon-separated list of build users.
- clienthost – the hostnames to set in the MoM config file
-
classmethod
setUpClass
()¶ Hook method for setting up class fixture before running tests in the class.
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
classmethod
check_users_exist
()¶ Check whether the user is exist or not
-
classmethod
kicksched_action
(server, obj_type, *args, **kwargs)¶ custom scheduler action to kick a scheduling cycle when expectig a job state change
-
classmethod
parse_param
()¶ get test configuration parameters as a
comma-separated
list of attributes.Attributes may be
'='
separated key value pairs or standalone entries.Multi-property
attributes are colon-delimited.
-
classmethod
is_server_licensed
(server)¶ Check if server is licensed or not
-
classmethod
init_from_conf
(conf, single=None, multiple=None, skip=None, func=None)¶ Helper method to parse test parameters for`` mom/server/scheduler`` instances.
The supported format of each service request is:
hostname@configuration/path
For example:
pbs_benchpress -p server=remote@/etc/pbs.conf.12.0
initializes a remote server instance that is configured according to the remote file
/etc/pbs.conf.12.0
-
classmethod
init_servers
(init_server_func=None, skip=None)¶ Initialize servers
-
classmethod
init_comms
(init_comm_func=None, skip=None)¶ Initialize comms
-
classmethod
init_schedulers
(init_sched_func=None, skip=None)¶ Initialize schedulers
-
classmethod
init_moms
(init_mom_func=None, skip='nomom')¶ Initialize moms
-
classmethod
init_server
(hostname, pbsconf_file=None)¶ Initialize a server instance
Define custom expect action to trigger a scheduling cycle when job is not in running state
Returns: The server instance on success and None on failure
-
classmethod
init_comm
(hostname, pbsconf_file=None, server=None)¶ Initialize a Comm instance associated to the given hostname.
This method must be called after init_server
Parameters: - hostname (str) – The host on which the Comm is running
- pbsconf_file (str or None) – Optional path to an alternate pbs config file
- server (str) – The server name associated to the Comm
Returns: The instantiated Comm upon success and None on failure.
Return the instantiated Comm upon success and None on failure.
-
classmethod
init_scheduler
(hostname, pbsconf_file=None, server=None)¶ Initialize a Scheduler instance associated to the given server. This method must be called after
init_server
Parameters: - server (str) – The server name associated to the scheduler
- pbsconf_file (str or None) – Optional path to an alternate config file
- hostname (str) – The host on which Sched is running
Returns: The instantiated scheduler upon success and None on failure
-
classmethod
init_mom
(hostname, pbsconf_file=None, server=None)¶ Initialize a
MoM
instance associated to the given hostname.This method must be called after
init_server
Parameters: - hostname (str) – The host on which the MoM is running
- pbsconf_file (str or None) – Optional path to an alternate pbs config file
Returns: The instantiated MoM upon success and None on failure.
-
init_proc_mon
()¶ Initialize process monitoring when requested
-
revert_pbsconf
()¶ Revert contents of the pbs.conf file Also start/stop the appropriate daemons
-
revert_servers
(force=False)¶ Revert the values set for servers
-
revert_comms
(force=False)¶ Revert the values set for comms
-
revert_schedulers
(force=False)¶ Revert the values set for schedulers
-
revert_moms
(force=False)¶ Revert the values set for moms
-
revert_server
(server, force=False)¶ Revert the values set for server
-
revert_comm
(comm, force=False)¶ Revert the values set for comm
-
revert_scheduler
(scheduler, force=False)¶ Revert the values set for scheduler
-
revert_mom
(mom, force=False)¶ Revert the values set for mom
Parameters: - mom (MoM object) – the MoM object whose values are to be reverted
- force (bool) – Option to reverse forcibly
-
analyze_logs
()¶ analyze accounting and scheduler logs from time test was started until it finished
-
set_test_measurements
(mdic=None)¶ set dictionary of analytical results of the test in order to include it in test report
Parameters: mdic (dict) – dictionary with analytical data Returns: True on successful append or False on failure
-
add_additional_data_to_report
(datadic=None)¶ set dictionary that will be merged with the test report for the overall test run
Parameters: datadic (dict) – dictionary with analytical data Returns: True on succssful update or False on failure
-
start_proc_monitor
(name=None, regexp=False, frequency=60)¶ Start the process monitoring
Parameters: - name (str or None) – Process name
- regexp (bool) – Regular expression to match
- frequency (int) – Frequency of monitoring
-
stop_proc_monitor
()¶ Stop the process monitoring
-
skipTest
(reason=None)¶ Skip Test
Parameters: reason (str or None) – message to indicate why test is skipped
-
skip_test
(reason=None)¶ Skip Test
Parameters: reason (str or None) – message to indicate why test is skipped
-
tearDown
()¶ verify that
server
andscheduler
are up clean up jobs and reservations
-
classmethod
tearDownClass
()¶ Hook method for deconstructing the class fixture after running all tests in the class.
- server – The hostname on which the PBS
1.2.8. Plugins¶
1.2.8.1. ptl_test_data¶
1.2.8.2. ptl_test_db¶
-
exception
ptl.utils.plugins.ptl_test_db.
PTLDbError
(rv=None, rc=None, msg=None, post=None, *args, **kwargs)¶ PTL database error class
Parameters: - rv (str or None) – Return value for the database error
- rc (str or None) – Return code for the database error
- msg (str or None) – Error message
-
class
ptl.utils.plugins.ptl_test_db.
DBType
(dbtype, dbpath, dbaccess)¶ Base class for each database type Any type of database must inherit from me
Parameters: - dbtype (str) – Database type
- dbpath (str) – Path to database
- dbaccess (str) – Path to a file that defines db options
-
write
(data, logfile=None)¶ Parameters: - data – Data to write
- logfile (str or None) – Can be one of
server
,scheduler
,mom
,accounting
orprocs
-
close
(result=None)¶ Close the database
-
class
ptl.utils.plugins.ptl_test_db.
PostgreSQLDb
(dbtype, dbpath, dbaccess)¶ PostgreSQL type database
-
write
(data, logfile=None)¶ Parameters: - data – Data to write
- logfile (str or None) – Can be one of
server
,scheduler
,mom
,accounting
orprocs
-
close
(result=None)¶ Close the database
-
-
class
ptl.utils.plugins.ptl_test_db.
SQLiteDb
(dbtype, dbpath, dbaccess)¶ SQLite type database
-
write
(data, logfile=None)¶ Parameters: - data – Data to write
- logfile (str or None) – Can be one of
server
,scheduler
,mom
,accounting
orprocs
-
close
(result=None)¶ Close the database
-
-
class
ptl.utils.plugins.ptl_test_db.
FileDb
(dbtype, dbpath, dbaccess)¶ File type database
-
write
(data, logfile=None)¶ Parameters: - data – Data to write
- logfile (str or None) – Can be one of
server
,scheduler
,mom
,accounting
orprocs
-
close
(result=None)¶ Close the database
-
-
class
ptl.utils.plugins.ptl_test_db.
HTMLDb
(dbtype, dbpath, dbaccess)¶ HTML type database
-
write
(data, logfile=None)¶ Parameters: - data – Data to write
- logfile (str or None) – Can be one of
server
,scheduler
,mom
,accounting
orprocs
-
close
(result=None)¶ Close the database
-
-
class
ptl.utils.plugins.ptl_test_db.
JSONDb
(dbtype, dbpath, dbaccess)¶ JSON type database
-
write
(data, logfile=None)¶ Parameters: - data – Data to write
- logfile (str or None) – Can be one of
server
,scheduler
,mom
,accounting
orprocs
-
close
(result=None)¶ Close the database
-
-
class
ptl.utils.plugins.ptl_test_db.
PTLTestDb
¶ PTL Test Database Plugin
-
options
(parser, env)¶ Register command line options
-
set_data
(dbtype, dbpath, dbaccess)¶ Set the data
-
configure
(options, config)¶ Configure the plugin and system, based on selected options
Parameters: options – Configuration options for plugin
andsystem
-
process_output
(info={}, dbout=None, dbtype=None, dbaccess=None, name=None, logtype=None, summary=False)¶ Send analyzed log information to either the screen or to a database file.
Parameters: - info (Dictionary) – A dictionary of log analysis metrics.
- dbout (str or None) – The name of the database file to send output to
- dbtype – Type of database
- dbaccess – Path to a file that defines db options (PostreSQL only)
- name (str or None) – The name of the log file being analyzed
- logtype – The log type, one of
accounting
,schedsummary
,scheduler
,server
, ormom
- summary – If True output summary only
-
1.2.8.3. ptl_test_info¶
-
ptl.utils.plugins.ptl_test_info.
get_effective_reqs
(ts_reqs=None, tc_reqs=None)¶ get effective requirements at test case
-
class
ptl.utils.plugins.ptl_test_info.
PTLTestInfo
¶ Load test cases from given parameter
-
options
(parser, env)¶ Register command line options
-
set_data
(suites, list_test, showinfo, verbose, gen_ts_tree)¶ Set the data required for running the tests
Parameters: - suites – Test suites to run
- list_test – List of test to run
- gen_ts_tree – Generate test suite tree
-
configure
(options, config)¶ Configure the plugin and system, based on selected options
Parameters: options – Options to configure plugin and system
-
wantClass
(cls)¶ Is the class wanted?
-
wantFunction
(function)¶ Is the function wanted?
-
wantMethod
(method)¶ Is the method wanted?
-
1.2.8.4. ptl_test_loader¶
-
class
ptl.utils.plugins.ptl_test_loader.
PTLTestLoader
¶ Load test cases from given parameter
-
options
(parser, env)¶ Register command line options
-
set_data
(testgroup, suites, excludes, follow)¶ Set the data required for loading test data
Parameters: - testgroup – Test group
- suites – Test suites to load
- excludes – Tests to exclude while running
-
configure
(options, config)¶ Configure the
plugin
andsystem
, based on selected options
-
check_unknown
()¶ Check for unknown test suite and test case
-
prepareTestLoader
(loader)¶ Prepare test loader
-
is_already_allowed
(cls, method=None)¶ Parameters: method – Method to check Returns: True if method is already allowed else False
-
wantClass
(cls)¶ Is the class wanted?
-
wantFunction
(function)¶ Is the function wanted?
-
wantMethod
(method)¶ Is the method wanted?
-
1.2.8.5. ptl_test_runner¶
-
exception
ptl.utils.plugins.ptl_test_runner.
TimeOut
¶ Raise this exception to mark a test as timed out.
-
exception
ptl.utils.plugins.ptl_test_runner.
TCThresholdReached
¶ Raise this exception to tell that tc-failure-threshold reached
-
class
ptl.utils.plugins.ptl_test_runner.
TestLogCaptureHandler
¶ Log handler for capturing logs which test case print using logging module
-
class
ptl.utils.plugins.ptl_test_runner.
PtlTestRunner
(stream=<open file '<stdout>', mode 'w'>, descriptions=True, verbosity=3, config=None)¶ Test runner that uses
PtlTestResult
to enable errorClasses, as well as providing hooks for plugins to override or replace the test output stream, results, and the test case itself.-
run
(test)¶ Overrides to provide plugin hooks and defer all output to the test result class.
-
-
class
ptl.utils.plugins.ptl_test_runner.
PTLTestRunner
¶ PTL Test Runner Plugin
-
options
(parser, env)¶ Register command line options
-
configure
(options, config)¶ Configure the plugin and system, based on selected options
-
prepareTestRunner
(runner)¶ Prepare test runner
-
prepareTestResult
(result)¶ Prepare test result
-
startTest
(test)¶ Start the test
-
stopTest
(test)¶ Stop the test
-
addError
(test, err)¶ Add error
-
addFailure
(test, err)¶ Add failure
-
addSuccess
(test)¶ Add success
-
1.2.8.6. ptl_test_tags¶
Decorator that adds tags to classes or functions or methods
Look up an tag on a
method/function
. If the tag isn’t found there, looking it up in the method’s class, if any.
Object that can act as context dictionary for eval and looks up names as attributes on a method/function and its class.
Load test cases from given parameter
Register command line options
Configure the plugin and system, based on selected options.
attr and eval_attr may each be lists.
self.attribs will be a list of lists of tuples. In that list, each list is a group of attributes, all of which must match for the rule to match.
Verify whether a method has the required tags The method is considered a match if it matches all tags for any tag group.
Prepare test runner
Accept the class if its subclass of TestCase and has at-least one test case
Accept the function if its tags match.
Accept the method if its tags match.
1.2.8.7. ptl_report_json¶
-
class
ptl.utils.plugins.ptl_report_json.
PTLJsonData
(command)¶ The intent of the class is to generate json format of PTL test data
-
get_json
(data, prev_data=None)¶ Method to generate test data in accordance to json schema
Parameters: - data (dict) – dictionary of a test case details
- prev_data (dict) – dictionary of test run details that ran before the current test
:returns a formatted dictionary of the data
-