DataOps Engineer - Azure Databricks & Unity Catalog

About the role

Mars Data Platform Services (DPS) is transforming its enterprise data platform across Azure Databricks and Unity Catalog into an automated, self-service product, with an intelligent, metadata-driven ingestion engine that configures and scales pipelines automatically. The DataOps Engineer leads the technical evolution of the data inflow and landing zone ecosystem, turning pipeline creation, data onboarding, storage provisioning and data archiving into automated, configuration-driven processes.

Key responsibilities

• Build and scale the metadata-driven ingestion engine: design and enhance the central metadata framework so new data sources need only configuration updates, and ensure the engine orchestrates ingestion, schema evolution and merge logic reliably.
• Lead landing zone reliability and data quality safeguards: maintain high-throughput landing zones across cloud storage and Unity Catalog, and build automated safeguards that quarantine schema errors or corrupted data.
• Automate Unity Catalog assets and long-term archiving: automate the creation of storage locations, schemas and tables, and implement lifecycle policies for time-based retention and migration to low-cost archival storage.
• Champion DataOps standards and operational enablement: drive CI/CD testing and deployment, create reusable deployment templates, monitoring dashboards and operational playbooks for 24/7 support teams.

Required profile

• 4+ years of hands-on data engineering experience building batch and streaming pipelines with Python/PySpark, SQL and cloud data warehouses or lakehouses (Azure Databricks / Delta Lake).
• Proven track record of config-driven, metadata-driven frameworks where pipeline generation, schema creation, Delta merge logic and scheduling are driven by central control tables.
• Programmatic Unity Catalog and landing zone architecture: provisioning Databricks Unity Catalog assets (Volumes, External Locations, Managed/External Tables) via code/APIs alongside Azure Data Lake Storage Gen2 landing zones.
• DataOps and automation practices: treating pipelines as software products with CI/CD, Git, automated testing and modular packaging such as Databricks Asset Bundles or Terraform.

Nice to have

• Declarative pipeline tools such as Delta Live Tables or dbt.
• Automated data retention schedules and long-term cloud archival storage.
• Familiarity with supporting operational support teams and Managed Service Providers.