AI Summary
Lead complex data engineering teams, manage capacity and growth projections, and develop optimal data pipeline architecture. Drive optimization, testing, and tooling to improve solution quality. Expertise in Databricks, PySpark, and large distributed systems.
Key Highlights
Manage end-to-end delivery of data engineering projects
Develop and manage capacity and growth projection forecasts
Create and maintain optimal data pipeline architecture
Drive optimization, testing, and tooling to improve solution quality
Lead teams that build and operate high-volume distributed systems
Technical Skills Required
Benefits & Perks
Job Description
Primary Skills
- Databricks
- PySpark
- ETL
- DWH
Responsibilities
- Manage end to end delivery by Investigating problem areas, working cross-functionally with product manager & other stakeholders
- Follow the Agile development methodology; think strategically and execute methodically
- Develop and manage capacity and growth projection forecasts of the environment within budgets
- Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements
- Drive optimization, testing and tooling to improve quality of solutions
- Manage teams that build and operate high volume distributed systems in a SaaS environment
- Great at devising efficient processes that increase velocity and quality
Desired Experience/ Skills:
- 10+ years of relevant experience
- At least 5 years of experience building and leading highly complex, technical engineering teams.
- Strong hands-on experience in Data-bricks
- Experience managing distributed teams preferred.
- Strong technical experience in large distributed systems, Data Warehousing, Data Lake at scale Project management skills: financial/budget management, scheduling and resource management experience with medium and large-scale projects
- Comfortable working with ambiguity and multiple stakeholders.
- Comfortable working cross functionality with product management and directly with customers; ability to deeply understand product and customer personas.
- Architecture Design Experience for Cloud and Non-cloud platforms
- Expertise on Azure Cloud platform
- Knowledge on orchestrating workloads on cloud
- Ability to set and lead the technical vision while balancing business drivers
- Strong experience with PySpark, Python programming
- Proficiency with APIs, containerisation and orchestration is a plus
Other Skills/Qualifications:
Bachelor's and/or master’s degree in computer science or equivalent experience.
Strong communication, analytical and problem-solving skills with a high attention to detail.