Senior Data Engineer

This Senior Data Engineer role involves administering and enhancing CAE's enterprise data platform, including Data Warehouses and Data Lakes, ensuring optimal performance and scalability. Key responsibilities include designing and maintaining robust data pipeline architecture using SQL and big data technologies like Hadoop, Spark, and Kafka. The engineer will establish ETL infrastructure, implement DevOps for CICD, integrate diverse data sources, and create backend solutions for analysts. The role also focuses on process enhancements, stakeholder partnership, security compliance, and setting up observability systems, contributing to innovative data and AI products.

Must Have

• Administer and enhance enterprise data platform (DWH, Data Lakes).
• Design, implement, and maintain robust data pipeline architecture.
• Establish and oversee ETL infrastructure from diverse sources.
• Implement DevOps practices for continuous integration and deployment.
• Integrate and manage data from multiple sources, including external and IoT.
• Create and maintain backend data solutions for analysts and data scientists.
• Identify and implement process enhancements, automating manual tasks.
• Partner with stakeholders to resolve technical issues and support infrastructure.
• Ensure compliance with data architecture and security standards.
• Set up observability and monitoring systems for data pipeline reliability.
• Minimum 6 years industry experience with data, coding, and scripting.
• Solid knowledge of CS fundamentals in algorithms and data structures.
• Strong experience with Terraform.
• Experience with big data tools: Hadoop, Spark, Kafka.
• Experience with relational SQL and NoSQL databases.
• Experience with automated data pipeline and workflow management tools.
• Experience with Microsoft cloud services: Azure, Databricks, Fabric, Purview.
• Experience with stream-processing systems: Storm, Spark-Streaming.
• Familiarity with Kubernetes for container orchestration.
• Proficiency in Power BI for data visualization and reporting.

Good to Have

• Master’s degree in Computer Science, Engineering, or related field.

Perks & Benefits

• Comprehensive benefits supporting professional and personal well-being.

Passionate about your field? Looking for a work environment where you can meet challenges, innovate, use state-of-the-art tools and collaborate with experts from various sectors? Want to join a worldclass company and industry leader and be proud to contribute to our projects and mission? Come join our team and share your passion with us!

Key Responsibilities:

We are seeking a highly skilled and motivated Data Platform Engineer to join our team. This role involves administering and enhancing our enterprise data platform, including Data Warehouses (DWH) and Data Lakes, ensuring optimal performance and scalability. The ideal candidate will have extensive experience with Terraform and a strong background in various data technologies.

Responsibilities:

• Administer and enhance the enterprise data platform, including Data Warehouses (DWH) and Data Lakes, ensuring optimal performance and scalability.
• Design, implement, and maintain robust and reliable data pipeline architecture to meet business needs, utilizing SQL and big data technologies such as Hadoop, Spark, and Kafka.
• Establish and oversee infrastructure for efficient extraction, transformation, and loading (ETL) of data from diverse sources, ensuring adherence to best practices and standards.
• Implement DevOps practices for continuous integration and continuous deployment (CICD) of data systems, enhancing agility and efficiency.
• Integrate and manage data from multiple sources, including external and IoT data, ensuring data is a valuable enterprise asset.
• Create and maintain backend data solutions for analysts and data scientists, facilitating insightful data analysis and decision-making.
• Identify and implement process enhancements, such as automating manual tasks and optimizing data delivery, to improve efficiency and scalability.
• Partner with stakeholders, including product teams, data experts, and architects, to resolve technical issues and support data infrastructure needs.
• Ensure compliance with data architecture and security standards, maintaining data protection and privacy.
• Set up observability and monitoring systems to track data pipeline reliability and provide prompt operational support in case of incidents.
• Work with executive, CX, design, and development teams to support data-related technical needs and propose innovative ideas for data and AI products.
• Inspire and motivate colleagues by sharing knowledge, contributing to a collaborative work environment, and promoting an Agile mindset.
• Stay connected with industry trends and advancements, suggesting innovative ideas and improvements for data strategies and delivery processes.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or related field. A Master’s is a plus.
• A minimum of 6 years industry experience working with data, coding, and scripting (Python, Java, Scala, SQL, JavaScript, Bash), design, and testing.
• Solid knowledge of CS fundamentals in algorithms and data structures.
• Strong experience with Terraform.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Experience with big data tools: Hadoop, Spark, Kafka.
• Experience with relational SQL and NoSQL databases, including SQL Server, PostgreSQL, and CosmosDB.
• Experience with automated data pipeline and workflow management tools: DevOps, ARM, Azure Data Factory, Airflow.
• Experience with Microsoft cloud services: Azure, Databricks, Fabric, Purview
• Experience with stream-processing systems: Storm, Spark-Streaming.
• Familiarity with Kubernetes for container orchestration.
• Proficiency in Power BI for data visualization and reporting.