Job Description
Iris's Direct client, a leading Logistics solutions and services provider is looking to hire a Sr. Data Engineer. This role will be 100% remote in Canada and will be Full Time T4 with Iris Software.
MUST HAVE OVER 10 yrs of IT Experience.
Job Description / Requirements
- Create and Optimize data pipelines for performance and efficiency.
- Work with Databricks clusters and configuration management tools.
- Use appropriate tools in the cloud data lake development and deployment. Developing/implementing cloud infrastructure to support current and future business needs.
- Provide technical expertise and ownership in the diagnosis and resolution of issues.
- Ensure all cloud solutions exhibit a higher level of cost efficiency, performance, security, scalability, and reliability.
- Manage cloud data lake development and deployment on AWS /Databricks.
MUST HAVES:
- Experience & Proficiency with Databricks platform - Delta Lake storage, Spark (PySpark, Spark SQL). Must be well versed with Databricks Lakehouse, Unity Catalog concept and its implementation in enterprise environments.
- Familiarity of data design pattern - medallion architecture to organize data in a Lakehouse.
- Experience & Proficiency with AWS Data Services – S3, Glue, Athena, Redshift etc.| Airflow scheduling
- Proficiency in SQL and experience with relational databases.
- Proficiency in at least one programming language (e.g., Python, Spark, Pyspark, Java) for data processing and scripting.
- Good understanding of data principles, Cloud Data Lake design & development including data ingestion, data modeling and data distribution.
Regards,
Anurag Dang
www.irissoftware.com
https://www.linkedin.com/in/anuragdang/