Senior Data Engineer

Senior Data Engineer | Cloudera → Cloud Migration Specialist

We're hiring a Senior Data Engineer to lead the delivery of a modern, cloud-native data platform, supporting advanced analytics and machine learning.

This is a hands on senior role focused on migrating legacy Cloudera (CDH/HDP) environments to Snowflake or Databricks-with a strong emphasis on re architecting workloads for the cloud, not just lifting and shifting.

What You'll Be Doing

• Leading end-to-end Cloudera → Snowflake or Databricks migrations

• Re-engineering Hive / Impala workloads into scalable, cloud-native solutions

• Migrating and optimising large-scale datasets from HDFS to cloud platforms

• Designing and building high-performance data pipelines using Spark / PySpark

• Implementing modern ingestion patterns (CDC, APIs, Reverse ETL)

• Leveraging platform-specific features such as: Snowflake: Snowpipe, Streams, Tasks

Databricks: Auto Loader, Delta Live Tables, Unity Catalog

• Playing a key role in shaping the architecture of a greenfield data platform

What We're Looking For

• Proven experience delivering Cloudera → Snowflake or Databricks migrations

• Experience with Snowflake or Databricks in production environments

• Strong Python engineering skills and experience with Spark / PySpark

• Experience refactoring legacy ETL pipelines into cloud-native architectures (not "lift-and-shift")

• Solid understanding of distributed systems and modern data architectures

• Experience working with AWS or GCP

Key Requirements

• This role requires candidates who can hit the ground running

• We're specifically looking for engineers with practical experience using Snowflake or Databricks features

• Candidates with only Hadoop/Cloudera backgrounds without exposure to the target platforms are unlikely to be considered

Why Join?

• Work on a greenfield data platform with real ownership

• Be part of a major transformation programme

• Use cutting-edge tools across Snowflake, Databricks & cloud ecosystems

• Join a highly technical, delivery-focused team

Spectrum IT Recruitment (South) Limited is acting as an Employment Agency in relation to this vacancy.