xcollections

pyslurm.xcollections

Custom Collection utilities

MultiClusterMap

Mapping of Multi-Cluster Data for a Collection.

TL;DR

If you have no need to write Multi-Cluster capable code and just work on a single Cluster, Collections inheriting from this Class behave just like a normal dict.

This class enables collections to hold data from multiple Clusters if applicable. For quite a few Entities in Slurm it is possible to gather data from multiple Clusters. For example, with sacct, you can easily query Jobs running on different Clusters - provided your Cluster is joined in a Federation or simply part of a multi Cluster Setup.

Collections like pyslurm.db.Jobs inherit from this Class to enable holding such data from multiple Clusters. Internally, the data is structured in a dict like this (with pyslurm.db.Jobs as an example):

data = {
    "LOCAL_CLUSTER": {
        1: pyslurm.db.Job(1),
        2: pyslurm.db.Job(2),
        ...
    },
    "OTHER_REMOTE_CLUSTER": {
        100: pyslurm.db.Job(100),
        101, pyslurm.db.Job(101)
        ...
    },
    ...
}

When a collection inherits from this class, its functionality will basically simulate a standard dict - with a few extensions to enable multi-cluster code. By default, even if your Collections contains Data from multiple Clusters, any operation will be targeted on the local Cluster data, if available.

For example, with the data from above:

job = data[1]

job would then hold the instance for pyslurm.db.Job(1) from the LOCAL_CLUSTER data.

Alternatively, data can also be accessed like this:

job = data["OTHER_REMOTE_CLUSTER"][100]

Here, you are directly specifying which Cluster data you want to access, and you will get the instance for pyslurm.db.Job(100) from the OTHER_REMOTE_CLUSTER data.

Similarly, every method (where applicable) from a standard dict is extended with multi-cluster functionality (check out the examples on the methods)

add(item) method descriptor

An Item to add to the collection

Note that a collection can only hold its specific type. For example, a collection of pyslurm.db.Jobs can only hold pyslurm.db.Job objects. Trying to add anything other than the accepted type will raise a TypeError.

Parameters:

Name	Type	Description	Default
item	Any	Item to add to the collection.	required

Raises:

Type	Description
TypeError	When an item with an unexpected type not belonging to the collection was added.

Examples:

Add a pyslurm.db.Job instance to the pyslurm.db.Jobs collection.

>>> import pyslurm
>>> jobs = pyslurm.db.Jobs()
>>> job = pyslurm.db.Job(1)
>>> jobs.add(job)
>>> print(jobs)
pyslurm.db.Jobs({1: pyslurm.db.Job(1)})

clear() method descriptor

Clear the collection

clusters() method descriptor

Return a View of all the Clusters in this collection

Returns:

Type	Description
ClustersView	View of Cluster keys

Examples:

Iterate over all Cluster-Names the Collection contains:

>>> for cluster in collection.clusters()
...     print(cluster)

copy() method descriptor

Return a Copy of this instance.

get(key, default=None) method descriptor

Get the specific value for a Key

This behaves like dict's get method, with the difference that you can additionally pass in a 2-tuple in the form of (cluster, key) as the key, which can be helpful if this collection contains data from multiple Clusters.

If just a key without notion of the Cluster is given, access to the local cluster data is implied. If this collection does however not contain data from the local cluster, the first cluster detected according to next(iter(self.keys())) will be used.

Examples:

Get a Job from the LOCAL_CLUSTER

>>> job_id = 1
>>> job = data.get(job_id)

Get a Job from another Cluster in the Collection, by providing a 2-tuple with the cluster identifier:

>>> job_id = 1
>>> job = data.get(("REMOTE_CLUSTER", job_id))

items() method descriptor

Return a View of all the Values in this collection

Returns:

Type	Description
ItemsView	View of all Items

Examples:

Iterate over all Items from all Clusters:

>>> for key, value in collection.items()
...     print(key, value)

Iterate over all Items from all Clusters with the name of the Cluster additionally provided:

>>> for cluster, key, value in collection.items().with_cluster()
...     print(cluster, key, value)

keys() method descriptor

Return a View of all the Keys in this collection

Returns:

Type	Description
KeysView	View of all Keys

Examples:

Iterate over all Keys from all Clusters:

>>> for key in collection.keys()
...     print(key)

Iterate over all Keys from all Clusters with the name of the Cluster additionally provided:

>>> for cluster, key in collection.keys().with_cluster()
...     print(cluster, key)

pop(key, default=None) method descriptor

Remove key from the collection and return the value

This behaves like dict's pop method, with the difference that you can additionally pass in a 2-tuple in the form of (cluster, key) as the key, which can be helpful if this collection contains data from multiple Clusters.

If just a key without notion of the Cluster is given, access to the local cluster data is implied. If this collection does however not contain data from the local cluster, the first cluster detected according to next(iter(self.keys())) will be used.

popitem() method descriptor

Remove and return a (key, value) pair as a 2-tuple

to_json(multi_cluster=False) method descriptor

Convert the collection to JSON.

Returns:

Type	Description
str	JSON formatted string from json.dumps()

update(data={}, **kwargs) method descriptor

Update the collection.

This functions like dict's update method.

values() method descriptor

Return a View of all the Values in this collection

Returns:

Type	Description
ValuesView	View of all Values

Examples:

Iterate over all Values from all Clusters:

>>> for value in collection.values()
...     print(value)

BaseView

Base View for all other Views

KeysView

Bases: BaseView

A simple Keys View of a collection

When iterating, this yields all the keys found from each Cluster in the collection. Note that unlike the KeysView from a dict, the keys here aren't unique and may appear multiple times.

If you indeed have multiple Clusters in a collection and need to tell the keys apart, use the with_cluster() function.

with_cluster() method descriptor

Return a Multi-Cluster Keys View.

Returns:

Type	Description
MCKeysView	Multi-Cluster Keys View.

MCKeysView

Bases: BaseView

A Multi-Cluster Keys View

Unlike KeysView, when iterating over an MCKeysView instance, this will yield a 2-tuple in the form (cluster, key).

Similarly, when checking whether this View contains a Key with the in operator, a 2-tuple must be used in the form described above.

ItemsView

Bases: BaseView

A simple Items View of a collection.

Returns a 2-tuple in the form of (key, value) when iterating.

Similarly, when checking whether this View contains an Item with the in operator, a 2-tuple must be used.

with_cluster() method descriptor

Return a Multi-Cluster Items View.

Returns:

Type	Description
MCItemsView	Multi-Cluster Items View.

MCItemsView

Bases: BaseView

A Multi-Cluster Items View.

This differs from ItemsView in that it returns a 3-tuple in the form of (cluster, key, value) when iterating.

Similarly, when checking whether this View contains an Item with the in operator, a 3-tuple must be used.

ValuesView

Bases: BaseView

A simple Value View

When iterating over an instance of this View, this will yield all values from all clusters.

ClustersView

Bases: BaseView

A simple Cluster-Keys View

When iterating over an instance of this View, it will yield all the Cluster names of the collection.