I'm 2 yoe Backend Engineer. I've worked mostly on Microservices developed using Spring/Java aimed at solving enterprise-y problems. I very much enjoy working on all aspects of development, from requirements gathering to implementing CI/CD and infrastructure provisioning on public cloud providers.

Trouble is, the place I work at has taken a hard pivot towards being an "AI and data company". What this means for me is that the kind of enterprisey , backend work I like to do has been gradually phased out with more data engineering specific workload.

I'm being made to skill up on Big Data tech, specifically pyspark to work on ELT/ETL related tasks.

I strongly dislike Data engineering/processing and don't really want to specialise in it or make a career out of it. Should I quit(very risky in current market) or is there some light at the end of the tunnel where knowledge of ELT/ETL/pyspark could help my long term prospects in Backend Engineering in some way?

If there's no benefit, I feel like I would be wasting an year or so working on something which is totally unrelated to my career goals and long term interests.