DagStream

DagStream is the Python package in order to manage relationship between functions, especially for data-preprocessing process for machine learning applications.

Key Features

  • Simple method to define dag relationship

  • Quick Visualization by mermaid

Definition of Dag

DagStream class convert your functions into dag nodes.

import dagstream

def funcA():
   print("funcA")
   return 10

def funcB(val_from_A: int):
   print(f"funcB, received {val_from_A} from A")

def funcC(val_from_A: int):
   print(f"funcC, received {val_from_A} from A")
   return 20

def funcD(val_from_C: int):
   print(f"funcD, received {val_from_C} from C")

def funcE():
   print("funcE")

def funcF():
   print("funcF")


stream = dagstream.DagStream()
# convert to functional nodes
A, B, C, D, E, F = stream.emplace(funcA, funcB, funcC, funcD, funcE, funcF)

# define relationship betweeen functional nodes

# A executes before B and C
# output of A passed to B and C
A.precede(B, C, pipe=True)

# E executes after B, C and D
E.succeed(B, C, D)

# D executes after C
# D receives output of C
D.succeed(C, pipe=True)

F.succeed(E)

Relationship between functions are like below.

stateDiagram direction LR state "funcA" as state_0 state "funcB" as state_1 state "funcC" as state_2 state "funcD" as state_3 state "funcE" as state_4 state "funcF" as state_5 [*] --> state_0 state_0 --> state_1: Pipe state_0 --> state_2: Pipe state_1 --> state_4 state_2 --> state_4 state_2 --> state_3: Pipe state_3 --> state_4 state_4 --> state_5 state_5 --> [*]

License

the Apache License, Version 2.0 (the “License”)

Contents

Indices and tables