Skip to main content

dataflow_template

dataflow

To get started:​

Dynamically pull and run​

from hamilton import dataflows, driver
# downloads into ~/.hamilton/dataflows and loads the module -- WARNING: ensure you know what code you're importing!
dataflow_template = dataflows.import_module("dataflow_template", "example_dataflow_template")
dr = (
    driver.Builder()
    .with_config({})  # replace with configuration as appropriate
    .with_modules(dataflow_template)
    .build()
)
# If you have sf-hamilton[visualization] installed, you can see the dataflow graph
# In a notebook this will show an image, else pass in arguments to save to a file
# dr.display_all_functions()
# Execute the dataflow, specifying what you want back. Will return a dictionary.
result = dr.execute(
    [dataflow_template.CHANGE_ME, ...],  # this specifies what you want back
    inputs={...}  # pass in inputs as appropriate
)

Use published library version​

pip install sf-hamilton-contrib --upgrade  # make sure you have the latest
from hamilton import dataflows, driver
# Make sure you've done - `pip install sf-hamilton-contrib --upgrade`
from hamilton.contrib.user.example_dataflow_template import dataflow_template
dr = (
    driver.Builder()
    .with_config({})  # replace with configuration as appropriate
    .with_modules(dataflow_template)
    .build()
)
# If you have sf-hamilton[visualization] installed, you can see the dataflow graph
# In a notebook this will show an image, else pass in arguments to save to a file
# dr.display_all_functions()
# Execute the dataflow, specifying what you want back. Will return a dictionary.
result = dr.execute(
    [dataflow_template.CHANGE_ME, ...],  # this specifies what you want back
    inputs={...}  # pass in inputs as appropriate
)

Modify for your needs​

Now if you want to modify the dataflow, you can copy it to a new folder (renaming is possible), and modify it there.

dataflows.copy(dataflow_template, "path/to/save/to")

Purpose of this module​

Template module to show what needs to be filled out.

Configuration Options​

This module can be configured with the following options: [list options]

Limitations​

Write limitations/assumptions/known issues here.

Source code​

__init__.py
# --- START NOTICES (optional)
# --- END NOTICES
# --- START IMPORT SECTION
import logging

logger = logging.getLogger(__name__)

from hamilton import contrib

with contrib.catch_import_errors(__name__, __file__, logger):
    # non-hamilton imports go here
    pass

# hamilton imports go here; check for required version if need be.

# --- END IMPORT SECTION

# --- START HAMILTON DATAFLOW


# --- END HAMILTON DATAFLOW
# --- START MAIN CODE
if __name__ == "__main__":
    # Code to create an imaging showing on DAG workflow.
    # run as a script to test Hamilton's execution
    import __init__ as MODULE_NAME

    from hamilton import base, driver

    dr = driver.Driver(
        {},  # CONFIG: fill as appropriate
        MODULE_NAME,
        adapter=base.DefaultAdapter(),
    )
    # saves to current working directory creating dag.png.
    dr.display_all_functions("dag", {"format": "png", "view": False})
# --- END MAIN CODE

Requirements​

# put non-hamilton requirements here