Updated specific locations to be searchable, take a look at Las Vegas as an example.
1
Converting an object-oriented pipeline to Airflow, how to handle a copious amount of instance variables?
Post Flair (click to view more posts with a particular flair)
Post Body

Disclaimer that Iā€™m currently new to the field of DE in general and Iā€™m learning airflow for the first time

I have a pipeline written in python thatā€™s a single class where each method is a task, and they communicate with each other mainly via class variables. From what I understand, xcoms are airflows way of passing variables between tasks

The problem I can see myself running into is that I use a copious amount of ā€œmetadataā€ in the form of these variables (I even pass dataframes this way instead of returning a dataframe from the methods)

I havenā€™t experimented much, but would I be able to convert any python object to an xcom, or would it be better to sort of rewrite the logic so that these tasks actually return values?

Iā€™m very new to airflow, so any help is appreciated

Author
Account Strength
100%
Account Age
7 years
Verified Email
Yes
Verified Flair
No
Total Karma
42,675
Link Karma
23,655
Comment Karma
18,693
Profile updated: 2 days ago
Posts updated: 6 months ago

Subreddit

Post Details

We try to extract some basic information from the post title. This is not always successful or accurate, please use your best judgement and compare these values to the post title and body for confirmation.
Posted
1 year ago