Senior Databricks Data Engineer

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Databricks Data Engineer in United Kingdom.

In this role, you will design, build, and optimize enterprise-scale data pipelines on Azure Databricks, supporting structured, semi-structured, and unstructured data. You will work closely with data architects, security teams, and business stakeholders to implement best practices for data governance, security, and high-performance data processing. This position involves hands-on development of Delta Lake architectures, CI/CD pipelines, and orchestrated workflows to deliver reliable, scalable data products. Operating in a fully remote, collaborative environment, you will have the opportunity to influence the overall data platform strategy while solving complex technical challenges. Your work will enable analytics, reporting, and AI initiatives across the enterprise. The role combines technical depth, architectural insight, and operational excellence, offering strong growth potential in cloud data engineering.

Accountabilities

• Design, develop, and optimize ETL/ELT data pipelines using Azure Databricks (Python, PySpark, SQL, Delta Lake).
• Configure and maintain Databricks workspaces, clusters, jobs, repositories, and workflow schedules for multi-team data product delivery.
• Implement and enforce data governance and security best practices, including access controls, lineage, and auditing frameworks.
• Build and maintain Delta Lake architectures with medallion (bronze/silver/gold) layer structures.
• Integrate Databricks pipelines with Azure Data Platform services, ensuring reliable orchestration, observability, and CI/CD automation.
• Collaborate with data architects, data owners, and cross-functional teams to align platform solutions with enterprise standards.
• Optimize pipeline performance, compute cost, and system efficiency through code-level and cluster-level tuning strategies.
  
  

Requirements

Interested in remote work opportunities in Data Science? Discover Data Science Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.

• Minimum 5 years of professional experience delivering Azure Databricks solutions in enterprise environments.
• Strong expertise in Databricks components: Workspaces, Notebooks, Jobs, Workflows, Repos, Unity Catalog, Delta Lake, Delta Live Tables, and MLflow.
• Solid knowledge of Azure Data Platform services: ADLS Gen2, Azure Key Vault, Azure Monitor, Azure Log Analytics, Azure Entra ID/RBAC, Terraform provider a plus.
• Experience implementing data security and governance frameworks including access controls, masking, row-level security, ABAC, governed tags, credential management, lineage, and auditability.
• Proficiency in Python, SQL, PySpark, Git, Spark performance tuning, and distributed computing concepts.
• Familiarity with AI/ML lifecycle and MLflow model management.
• Experience working in Agile or DevOps-oriented teams, with strong analytical, problem-solving, and communication skills.
• Fluency in Portuguese and English.
  
  

Benefits

• Competitive compensation aligned with experience.

Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.

• Fully remote work environment.
• Delivery of work equipment suited to the role and responsibilities.
• Comprehensive benefits plan.
• Opportunity to work with expert teams on high-impact, large-scale projects.
• Exposure to long-term, strategic client initiatives in diverse industries.
  
  

Why Apply Through Jobgether?

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.