CODEWITHSUNDEEP
airflowtfx
natielle
natielle

Reputation: 426

How to configure optional component with TFX?

I would like to run one optional custom component with TFX and Airflow. Is it possible?

This optional component is used to send updates to another system, if this optional component fails, it will not be able to interrupt the execution of the main tasks.

I would like something like this:

# fictitious example

from tfx.orchestration import pipeline
from my_custom_component_1 import MyCustomComponent1
from my_custom_component_optional import MyCustomComponentOptional
from my_custom_component_2 import MyCustomComponent2
from utils import PIPELINE_NAME, PIPELINE_ROOT, get_ml_metadata_conn_config

pipeline_name = PIPELINE_NAME
pipeline_root = PIPELINE_ROOT

components = [
    MyCustomComponent1(),
    MyCustomComponentOptional(),
    MyCustomComponent2()
]

pipeline.Pipeline(
    pipeline_name=pipeline_name,
    pipeline_root=pipeline_root,
    components=components,
    enable_cache=True,
    metadata_connection_config=get_ml_metadata_conn_config(),
)

I'm using these versions:

apache-airflow = 1.10.15
tfx = 0.22.0
tensorflow = "2.2.0"

Can someone help me?

Upvotes: 0

Views: 35

Answers (1)

Pritam Dodeja
Pritam Dodeja

Reputation: 326

This is possible, the output of this component should not be the input to any other component. Ideally you have another component with similar inputs that you can use to start building this component. The executor of this component where the communication would happen. I believe there's a slack component out there that might be a good starting point.

Upvotes: 1

Related Questions