Coming soon - Get a detailed view of why an account is flagged as spam!
view details

r/dataengineering

219,197 subscribers
News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Notice: This subreddit is not actively monitored, so the posts here are only a small subset of what exists on reddit.

Browse by flair

Help Career Discussion Meme Blog Open Source
New Top (24hr) Top (Week) Top (Month)
to r/dataengineering by saaggy_peneer
1 month ago
DMS -> S3 (CDC & Full) to SCD2 with DBT?
Discussion
to r/dataengineering by saaggy_peneer
1 month ago
DBT incremental merge with DMS CDC?
Discussion
to r/dataengineering by saaggy_peneer
3 months ago
Preview thumbnail for Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 image
Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2
Open Source
to r/dataengineering by FigTraditional1201
3 weeks ago
Google sheets —-> ???? ——-> PowerBI
Help
to r/dataengineering by FigTraditional1201
3 weeks ago
Database solution?
Help
to r/dataengineering by NFeruch
7 months ago
Using pandas in a data-intensive application - what the best, RAM-efficient, alternative?
Discussion
to r/dataengineering by cantseemelol
8 months ago
Cost of ADF using Stored Procedures vs pure ADF
Help
to r/dataengineering by NFeruch
7 months ago
Database design for my software app - how to handle changing fields from API response?
Discussion
to r/dataengineering by izotAcario
4 months ago
Newbie question: What solutions should I use to suggest my users posts based on their preferences?
Help
to r/dataengineering by NFeruch
7 months ago
Is there such thing as a “product data engineer” who isn’t just a slave to the reporting team?
Discussion
to r/dataengineering by NFeruch
7 months ago
Why can’t I pass context between tasks in Airflow/Dagster?
Discussion
to r/dataengineering by NFeruch
7 months ago
Critique my data pipeline (personal project). Are Airflow/Dagster incompatible?
Help
to r/dataengineering by NFeruch
7 months ago
Is doing a "dry run" of a data pipeline - so that changes to the database are either rolled back or not committed, for testing purposes - normal practice in DE?
Discussion
to r/dataengineering by NFeruch
7 months ago
What is the best SQL environment you have ever worked in?
Discussion
to r/dataengineering by NFeruch
7 months ago
Preview thumbnail for Keep in mind the following when reading about anything tech online lol image
Keep in mind the following when reading about anything tech online lol
Meme
to r/dataengineering by IllustriousCorgi9877
1 month ago
Kind of DE job to apply to
Career
in Delaware
to r/dataengineering by FigTraditional1201
3 weeks ago
MS SQL / AZURE SQL data base?
Discussion
to r/dataengineering by NFeruch
7 months ago
How can a data engineer contribute at a hackathon?
Career
to r/dataengineering by LoneSysAdm
7 months ago
ADF for Consuming REST API's
Help
to r/dataengineering by NFeruch
7 months ago
ELI5, what are the main differences between software engineers and data engineers?
Discussion
to r/dataengineering by CutSubstantial7320
8 months ago
How do you "know" your architecture is correct?
Discussion
to r/dataengineering by chock-a-block
11 months ago
Apache Airflow PostgreSQL not Committing
Help
to r/dataengineering by NFeruch
7 months ago
What is the difference between a data engineer, a data platform engineer, and a software engineer - data platform?
Discussion
to r/dataengineering by NFeruch
7 months ago
Converting an object-oriented pipeline to Airflow, how to handle a copious amount of instance variables?
Help
to r/dataengineering by BitterCustard26
11 months ago
Load SDMX format into a db
Help