This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
7
DMS -> S3 (CDC & Full) to SCD2 with DBT?
Post Flair (click to view more posts with a particular flair)
Post Body
holy acronyms, batman!
AWS DMS spits out "full" extract files to s3, as well as change data capture (CDC) files (usually different paths)
The CDC files look like this:
op,id,name,ts
I,33,jim,timestamp
U,44,frank,timestamp
D,55,bob,timestamp
where I = insert, U = update, D = delete
Questions:
- is there a name for the above file format? (to ease searching)
- how do I get from full cdc file to an SCD2 table with DBT?
Do I combine "full" and "cdc" to get "current state", and then snapshot "current state" against the SCD2 table?
Or is there some query magic I'm not figuring out?
Thanks
Author
Account Strength
70%
Account Age
1 year
Verified Email
Yes
Verified Flair
No
Total Karma
5,772
Link Karma
623
Comment Karma
5,149
Profile updated: 5 days ago
Posts updated: 6 days ago
Subreddit
Post Details
We try to extract some basic information from the post title. This is not
always successful or accurate, please use your best judgement and compare
these values to the post title and body for confirmation.
- Posted
- 2 months ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/dataenginee...