This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
1
Design a dynamic data pipeline in airflow
Post Body
I intent to create a dag that would run few tasks in the sequence defined by the json payload sent to rest api.
I have created a stackoverflow post here : https://stackoverflow.com/questions/75190990/design-airflow-pipeline-to-perform-sequential-tasks-based-on-payload-sent-to-air
- I cannot create dynamic dags since the internal tool i am using does not allow it.
- Creating dynamic tasks has a limitation that old dag runs also has the recent dag structure. This makes it complicated to debug failures
- Using a branch and loop design(as described in SO post) will not be considered acyclic and will not represent a pipeline.
Please go through the post and add a comment or suggest a design for my use case.
Author
Account Strength
80%
Account Age
4 years
Verified Email
Yes
Verified Flair
No
Total Karma
38
Link Karma
21
Comment Karma
17
Profile updated: 6 days ago
Posts updated: 1 year ago
Subreddit
Post Details
We try to extract some basic information from the post title. This is not
always successful or accurate, please use your best judgement and compare
these values to the post title and body for confirmation.
- Posted
- 2 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/apache_airf...