Senior Data Engineer - Real-Time Data Platforms

At Aviato Consulting, you will be the architect of our clients' real time data future, building the high throughput, low latency streaming platforms on Google Cloud. Here's what joining our team means for you:

• Innovate with Real-Time Data Services: You'll be at the forefront of streaming technology, building sophisticated, event-driven data pipelines that process massive volumes of data in real-time for our enterprise clients.
• Deep Dive into Google Cloud: As a core part of your role, you'll gain unmatched expertise in GCP's streaming services, designing and deploying robust data processing solutions with Dataflow and Pub/Sub. You will become a true expert in building and scaling streaming platforms on GCP.
• Shape Our Data Practice: Your expertise will be vital in defining our best practices for real-time data engineering. You'll have a direct impact on how we design, build, and monitor our streaming solutions, contributing significantly to our growing knowledge base.

What You'll Do:

• Architect and Build Mission Critical Streaming Pipelines: Design, implement, and manage high-volume, real-time data pipelines for core operational systems
• Develop and Deploy Scalable Streaming Solutions: Create and maintain resilient and efficient streaming jobs using Dataflow Streaming and Apache Spark to handle complex data processing, transformations, and enrichments in real-time.
• Utilize Pub/Sub to build decoupled, high-throughput data ingestion and distribution systems that are resilient, scalable, and built for enterprise reliability.
• Proactively monitor, troubleshoot, and optimize streaming pipelines to guarantee low latency, high availability, and data integrity for critical operational data.
• Use Infrastructure as Code (IaC) with Terraform to provision and manage all streaming data infrastructure, ensuring consistent and automated environments.

Essential Skills

• At least 4+ years of experience in a Data Engineering role with a strong focus on streaming data.
• Streaming Expertise: Deep, hands on knowledge of Dataflow Streaming and the underlying Apache Beam model.
• Messaging and Eventing Mastery: Proven experience building and scaling solutions with Pub/Sub or similar messaging queues (e.g., Kafka).
• Distributed Processing Skills: Strong experience with distributed data processing frameworks, particularly Apache Spark (Streaming or Batch).
• Strong Python or Java Skills: Proven experience in building data processing logic and automation scripts.

Preferred Skills

• Advanced Windowing and Triggering: Deep experience with advanced data processing techniques in Beam or Spark Streaming.
• Performance Tuning: Experience in performance tuning and optimizing large-scale Dataflow or Spark jobs.
• Kubernetes and GKE: Experience with containerization and managing data workloads on Google Kubernetes Engine (GKE).
• GCP Certification: Professional Data Engineer or similar.

What We Offer (Beyond the Standard):

• Continuous Learning: We are committed to your growth, providing opportunities to deepen your GCP and streaming data skills.
• Equity Opportunity: Become a part-owner of Aviato (after 6 months) and share in our success.
• Remote Flexibility: Work remotely from anywhere in India, aligning with IST to collaborate effectively with our clients.
• Direct Contribution: Your ideas and contributions will be valued and have a tangible impact on our streaming data practices.

Ready to be a key player in our growing Data Engineering practice and build the future of real-time data platforms for enterprise clients?