Essentially what the title says. I have an end to end pipeline that at a high level is * extract data * do some python transformations * export data
I want to add a 4th step of
- run the relevant dbt models
The thing is that I’m running into problems with invoking it programmatically, as the pipeline file is in a different directory than the dbt project (so it doesn’t detect the project yaml file) and I can’t use absolute paths to specify which model to run, which I need to do because the pipeline is in a different directory
This is for a personal project, so I am trying to implement dbt just for the sake of learning it. The thing is though, that the pipeline runs automatically on a cron job, and I need the model to be run at the time of the pipeline
Is this the wrong usage pattern for dbt? I suspect it is because it’s not straightforward to do so far
Subreddit
Post Details
- Posted
- 1 year ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/dataenginee...