Senior Databricks Developer - Government Health Sector

About the Role

We are seeking a Senior Databricks Developer remote to join a vital leader in the Government Health sector. This role is dedicated to optimizing public health outcomes through a modern, cloud-native data platform. As a Senior Databricks Developer, you will play a crucial role in architecting and building highly scalable data pipelines using PySpark/Scala within the Databricks Lakehouse Platform.

What You'll Do

• Lead the design and development of large-scale, resilient, and performant ETL/ELT data pipelines using PySpark/Scala within Databricks notebooks and jobs.
• Architect and manage the Delta Lake environment, focusing on data ingestion, quality enforcement, and schema evolution for complex public health datasets.
• Optimize Databricks clusters, notebooks, and Spark jobs for cost-efficiency and performance, specifically targeting bottlenecks in high-volume batch and streaming workloads.
• Define and enforce data governance practices within the Lakehouse, utilizing Unity Catalogue for centralized metadata and access control, adhering to government standards.
• Collaborate closely with government analysts and data scientists to transition analytical models and research findings into scalable, production-ready pipelines.
• Champion CI/CD and MLOps practices for Databricks notebooks and workflows, utilizing tools like Azure DevOps or Jenkins.
• Mentor and guide junior engineers on Databricks development standards, Spark optimization, and modern data engineering practices.

Requirements

• 6+ years of progressive professional experience in Data Engineering, with at least 3 years dedicated to developing solutions on the Databricks Platform.
• Expert-level proficiency in PySpark and/or Scala for distributed data processing.
• Mandatory hands-on experience with Delta Lake architecture, including DLT, time travel, and VACUUM operations.
• Deep understanding of cloud infrastructure (Azure preferred) and how Databricks integrates with cloud storage (ADLS Gen2) and services.
• Expert proficiency in SQL and dimensional modeling principles.
• Proven experience with CI/CD, Infrastructure as Code (e.g., Terraform), and Databricks command-line tools for automation.
• Exceptional communication and problem-solving skills, with the ability to analyze complex requirements and design resilient solutions in a highly regulated environment.

What We Offer

• Competitive salary range of AUD 120,000 - 150,000 per year.
• Visa sponsorship available for the right candidate.
• Opportunity to work on impactful projects in the Government Health sector.
• Flexible work arrangements with a remote-first approach.
• Professional development opportunities and mentorship.