I (45M) have been working on customer dashboards for my most recent job. Its boring AF, you can only make so many Tableau dashboards before wanting to punch yourself in the face. I can continue to do this, make decent $ at it, but man, work has to have something more fun than just the $ that I make.

I want to get back into data engineering - but I am not sure which types of jobs I should apply to, I don't have luck getting interviews.

My background is in RDBMS data warehousing, and building data warehouse applications (series of stored procedures that accomplished a very complicated and data intensive accounting process with slowly changing rules). Creating logical and physical data models is something I am good at. I understand, write, and tune SQL quite well, and have many years job experience doing these tasks. I ran and operated things on Teradata, and can tune their sharded MPP pretty well. All of these were orchestrated in UC4, I haven't used tools like Airflow, but aren't they the same?

I was a user of Impala, but never engineered anything on Hadoop - only consumed data there.

I've built a Snowflake Data Warehouse on top of AWS S3, and was minimally involved in creating data pipelines to those AWS buckets (I built firehose jobs that partitioned our data). Parsing JSON events and creating a real-time ETL - I have done this stuff pretty well.

Where I feel I lack, and interviewers know it:

I have NO on the job experience with Python, Spark, Databricks, Azure. People look for my experience with SQL Server, and I'm like... Isn't it the same as these others? I've not configured RedShift before, I've only consumed data there, so no experience standing up files as a datalake via Redshift - I've only ingested files into Snowflake (not even used files as external tables).

I have built fairly complicated for-fun Python projects at home, but otherwise I have NO on the job experience there. I downloaded a local instance of Spark, and did some stupid stuff with it (weeee!!!!), but I've never seen jobs in production on Spark.

What types of jobs should I be applying for? How do I go about get practical (preferably on the job) experience in the things I am lacking? I feel my skills are atrophied, no longer relevant to jobs people are hiring for, and am struggling figuring out how to bridge the gap to these newer technologies.