Disclaimer that Iām currently new to the field of DE in general and Iām learning airflow for the first time
I have a pipeline written in python thatās a single class where each method is a task, and they communicate with each other mainly via class variables. From what I understand, xcoms are airflows way of passing variables between tasks
The problem I can see myself running into is that I use a copious amount of āmetadataā in the form of these variables (I even pass dataframes this way instead of returning a dataframe from the methods)
I havenāt experimented much, but would I be able to convert any python object to an xcom, or would it be better to sort of rewrite the logic so that these tasks actually return values?
Iām very new to airflow, so any help is appreciated
Subreddit
Post Details
- Posted
- 1 year ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/dataenginee...