seaflow.schedulers package

Submodules

seaflow.schedulers.dag module

Module Introduction

This is a directed acyclic graph management class, which mainly manages directed acyclic graphs, including CRUD operations related to directed acyclic graphs and scheduling operations based on directed acyclic graphs; Main technologies: Networkx, descriptor technology, Kahn hierarchical algorithm, decorator technology and magic method of Operator overloading, prepare and new construction process.

  • Design mode:

    nothing

  • Key points:

    1. Networkx

    2. Descriptor technology

    3. Kahn algorithm

    4. Prepare and New Construction Process Magic Method

    5. Decorator Technology

    6. Operator overloading

    7. Type meta

    8. Call magic method

    9. Get magic method

    10. Deque technology

  • Main functions:

    1. CRUD Operation of Directed acyclic graph

    2. Scheduling operations of Directed acyclic graph

Usage examples

Class Description

class seaflow.schedulers.dag.DAGManagerDict(*args, **kwargs)

Bases: OrderedDict

Class Introduction:

Define a DAG diagram dedicated management Data dictionary and inherit the ordered dictionary.

Attribute Function:

Define a method for initializing DAG management dictionaries.

class seaflow.schedulers.dag.DAGer(scheduler=<seaflow.schedulers.dag.KahnScheduler object>)

Bases: object

Class Introduction:

This is a Directed acyclic graph control class. Its main function is to provide the management of DAG, computing engine management and scheduling management. The main technologies are based on DAG technology of Networkx, Ray based computing environment, decorator technology and Operator overloading.

Attribute Function:

Define an initial attribute method, mainly used to load the calculation engine and scheduling engine when creating control objects.

Parameters:

  • dag (object) - DAG Graph Management Objects.

  • scheduler (object) - Scheduling Engine Object.

add_edge(method='single', start_node_name=None, end_node_name=None, data_obj=None, data_dict_list=None, *args, **kwargs)

Method Function:

Define a method for adding edges.

Parameters:

  • start_node_name (str) - Start Node Name.

  • end_node_name (str) - End Node Name.

  • data_obj (object) - Node Data Object.

  • method (str) - Add Method.

  • data_dict_list (list) - List of Data dictionary.

Returns:

  • result (str) - Return result text.

add_node(method='single', node_name=None, data_obj=None, data_dict_list=None, *args, **kwargs)

Method Function:

Define a method for adding nodes.

Parameters:

  • node_name (str) - node name.

  • data_obj (obj) - node data object.

  • method (str) - Add Method.

  • data_dict_list (list) - List of Data dictionary.

Returns:

  • result (str): Return result text.

dispath_algorithm_with_kahn()

Method Function:

Define a scheduling method using Kahn algorithm, where yield technique is used to return a generator.

Returns:

  • run_list_iter (iter) - kahn run list iter.

draw()

Method Function:

Define a method for displaying legends.

get_edge_attributes(start_node_name=None, end_node_name=None, attr_name=None, *args, **kwargs)

Method Function:

Define a method for obtaining edge attributes.

Parameters:

  • start_node_name (str) - Start Node Name.

  • end_node_name (str) - End Node Name.

  • attr_name (str) - Attribute Name.

Returns:

attr_data (object) - Attribute data.

get_edge_neighbor(node_name=None, direction=None, *args, **kwargs)

Method Function:

Define a method for accessing neighboring edges of target nodes.

Parameters:

  • node_name (str) - Node name.

  • direction (str) - Access direction.

Returns:

  • edges_list (list) - Edges list.

get_function_obj(function_name)

Method Function:

Define a method to obtain the Function object.

Parameters:

  • function_name (str) - Function name.

Returns:

  • function_obj (object) - Function object.

get_graph_obj()

Method Function:

Define a method for obtaining graph objects.

Returns:

  • graph_obj (object) - Graph object, based on networkx technology.

get_node_attributes(node_name=None, attr_name=None, *args, **kwargs)

Method Function:

Define a method for obtaining node attributes.

Parameters:

  • node_name (str) - Node name.

  • attr_name (str) - Attribute Name.

Returns:

  • atrr_data (object) - Attribute data.

get_node_neighbor(node_name=None, direction=None, *args, **kwargs)

Method Function:

Define a method for accessing neighboring nodes of the target node.

Parameters:

  • node_name (str) - Node name.

  • direction (str) - Access direction.

Returns:

  • nodes_list (list) - Nodes list.

init_job_queue_with_kahn()

Method Function:

Define a method for initializing priority scheduling queues using the Kahn algorithm.

Returns:

  • result (str) - Return result text.

kahnclassification_algorithm()

Method Function:

Define a DAG graph layering algorithm.

Returns:

  • kahnclassification_list (list) - Kahn Rating List.

modify_edge(start_node_name=None, end_node_name=None, data_obj=None, *args, **kwargs)

Method Function:

Define a method for modifying edges.

Parameters:

  • start_node_name (str) - Start Node Name.

  • end_node_name (str) - End Node Name.

  • data_obj (object) - Node Data Object.

Returns:

  • result (str) - Return result text.

modify_node(node_name=None, data_obj=None, *args, **kwargs)

Method Function:

Define a method for modifying nodes.

Parameters:

  • node_name (str) - Node name.

  • data_obj (object) - Node data object.

Returns:

  • result (str) - Return result text.

query_edge(start_node_name=None, end_node_name=None, *args, **kwargs)

Method Function:

Method of Defining a Query Graph Edge.

Parameters:

  • start_node_name (str) - Start Node Name.

  • end_node_name (str) - End Node Name.

Returns:

  • data_dict (str) - Data Dict.

query_node(node_name=None, *args, **kwargs)

Method Function:

Method for defining a query graph node.

Parameters:

  • node_name (str) - Node name.

Returns:

  • data_dict (dict) - Data Dict.

register(controller_obj)

Method Function:

Define a decorator method for registering functions, using the decorator function technique with parameters.

Parameters:

  • controller_obj (object) - controller object.

Returns:

  • decorate (function) - Decorator Function object.

remove_edge(method='single', start_node_name=None, end_node_name=None, edges_name_list=None, *args, **kwargs)

Method Function:

Define a method for deleting edges.

Parameters:

  • method (str) - Delete Method.

  • edge_name (str) - Edge Name.

  • edges_name_list (list) - Edge Name List.

Returns:

  • result (str) - Return result text.

remove_node(method='single', node_name=None, nodes_name_list=None, *args, **kwargs)

Method Function:

Define a method for deleting nodes.

Parameters:

  • method (str) - Delete Method.

  • node_name (str) - node name.

  • nodes_name_list (list) - Node Name List.

Returns:

  • result (str) - Return result text.

show_edges()

Method Function:

A method for defining the edge situation of a presentation graph manager.

Returns:

  • show_edges_list (List) - Show Edge List.

show_nodes()

Method Function:

Define a method to display the node situation of the graph manager.

Returns:

  • show_nodes_list (List) - Display node list.

toposort_algorithm()

Method Function:

Define a method for Topological sorting.

Returns:

  • toposort_list (list) - Topological sorting List.

class seaflow.schedulers.dag.DAGerMeta(name, bases, attrs_dict, *args, **kwargs)

Bases: type

Class Introduction:

This is a control metaclass of Ensemble learning calculation chart. Its main function is to manage the generation of Ensemble learning calculation chart control class. Its main technology__Prepare__,__New__.

Method Function:

Overload a class creation method to implement DAG management objects based on Networkx.

Parameters:

  • name (str) - class name.

  • bases (tuple) - Base calss tuple.

  • attrs_dict (dict) - A namespace variable defined within a class, starting from__Prepare__ Method return access.

Returns:

  • instance (object) - Create an instance of the class (actually an object).

instance

alias of DAGer

class seaflow.schedulers.dag.DirectedAcyclicGraph(name)

Bases: object

Class Introduction:

This is a DAG management class, mainly used to manage DAG, mainly using Networkx technology and descriptor protocols.

Attribute Function:

Define an initialization method, mainly used to define the DAG manager name.

Parameters:

  • name (str) - name.

  • graph (DAGManagerDict) - DAG Manager Dictionary Object.

class seaflow.schedulers.dag.KahnScheduler

Bases: object

Class Introduction:

This is a task scheduler class, with main functions based on DAG calculation graphs and scheduling calculations using a computing engine. The main techniques include hierarchical queues and scheduling algorithms.

Attribute Function:

Define an initialization method for generating work queues.

seaflow.schedulers.pipeline module

Module Introduction

This is a scheduling management class that mainly combines nodes to form execution processes; Main technical internal methods and magic methods.

  • Design mode:

    nothing

  • Key points:

    1. Attribute function

  • Main functions:

    1. Function node management

Usage examples

Class Description

class seaflow.schedulers.pipeline.Pipeline(nodes)

Bases: object

Class Introduction:

This is a process management class that mainly combines nodes to form execution processes; Main technical internal methods and magic methods.

Attribute Function:

Define an initialization method for process pipeline class attributes.

Parameters:

  • nodes (list) - Node Object List (Ordered).

  • _nodes_by_name (list) - Node Name List (Ordered).

  • _nodes_by_input (orderdict) - Node Input List (Ordered).

  • _nodes_by_output (orderdict) - Node output list (ordered).

inputs()

Method Function:

Define a method for enumerating input parameters in a process pipeline.

Returns:

  • inputs (orderdict) - Ordered Dictionary of Input Parameters.

names()

Method Function:

Define a method for enumerating node names in a process pipeline.

Returns:

  • names (list) - Node Name List.

outputs()

Method Function:

Method of defining a grid to list output parameters in a process pipeline.

Returns:

  • outputs (orderdict) - Output Parameter Ordered Dictionary

Module contents