Senior Lakehouse Solution Architect & Platform Administrator (Azure Databricks)

Senior Lakehouse Solution Architect & Platform Administrator (Azure Databricks)

6-Month Contract-to-hire

100% Remote

40 hrs/week

Start Date: 01/12/26

Job Description

Our client is building a mission-critical Azure Databricks Lakehouse to support enterprise clinical, financial, operational, and analytical workloads. We are seeking a Senior Lakehouse Solution Architect & Platform Administrator to serve as a foundational member of the Lakehouse engineering team and provide hands-on technical leadership during the platform’s critical stabilization and hardening phase.

This role is architect-first and execution-oriented. The architect will own Lakehouse platform design, administration, reliability, and security while partnering closely with a small core team and mentoring distributed data engineers across the enterprise. The initial mandate is clear: deliver a hardened, stable, observable, and recoverable Minimal Viable Lakehouse capable of supporting 24/7 healthcare operations. Feature expansion will follow only after platform stability is assured.

This is a high-impact opportunity to shape the long-term data, analytics, and AI foundation for a major healthcare system.

Responsibilities

• Serve as the primary architect and administrator for the Azure Databricks Lakehouse, including Unity Catalog, workspaces, clusters/warehouses, storage integrations, and access controls
• Design and execute a Lakehouse hardening roadmap, focusing on:
• High availability and disaster recovery readiness
• Backup and restore patterns for data and metadata
• Job orchestration, observability, logging, alerting, and platform SLOs
• Proactive data quality and validation frameworks
• Define and enforce naming standards, environment strategy, and catalog structures to support lineage, governance, and long-term maintainability
• Partner with Enterprise Architecture and Security to ensure alignment on identity, access, networking, encryption, and compliance (HIPAA/PHI)
• Provide architectural oversight for ingestion and transformation pipelines (ADF → landing → raw → curated; dbt; Databricks Jobs)
• Guide engineers on resilient ingestion patterns, incremental processing, retry/recovery strategies, and pipeline observability
• Own and evolve CI/CD and Infrastructure as Code for the Lakehouse:
• Databricks deployment patterns (e.g., Asset Bundles or equivalent)
• Azure DevOps or GitHub-based pipelines
• Terraform-managed Databricks and Azure resources
• Support FinOps and platform guardrails, including cost tagging, usage monitoring, and cost-efficient compute patterns
• Mentor core and distributed data engineers through code reviews, architectural guidance, documentation, and best-practice enablement

Required Qualifications

• 5+ years of experience in data platform engineering, cloud engineering, or solution architecture roles
• Strong hands-on expertise with Azure Databricks, including:
• Unity Catalog administration
• Clusters and SQL/Python compute
• Databricks Jobs, notebooks, and governance patterns
• Experience with Azure Data Factory for orchestration and ingestion
• Strong SQL and Python skills
• Experience with CI/CD pipelines (Azure DevOps or GitHub Enterprise)
• Working experience with Terraform or other IaC frameworks
• Demonstrated experience designing or supporting platform observability, monitoring, and reliability
• Strong collaboration skills; comfortable partnering with Security, Enterprise Architecture, and distributed engineering teams

Preferred Qualifications

• Experience operating data platforms in healthcare or regulated environments (HIPAA/PHI)
• Deeper experience with Terraform, Kubernetes, or advanced Databricks governance patterns
• Familiarity with FinOps practices for cloud data platforms
• Exposure to OHDSI / OMOP, Purview, or enterprise data cataloging tools
• Experience modernizing or stabilizing legacy ingestion pipelines