Principal Data Engineer

Autodesk seeks a principal engineer and technical leader for the Data Acquisition team, focusing on designing robust and resilient data pipelines for analytical abstracts, enrichment, and metadata about design content. These datasets are crucial for AI/ML projects. The role involves collaborating with Product Managers, Program/Project managers, Analysts, Platform Architecture, and the Engineering Team to ensure modern, scalable, and reliable data pipelines. Responsibilities include designing, architecting, and implementing data pipelines, automating data flows, developing workflows with Apache Spark, Airflow, AWS EMR, and Snowflake, deploying solutions via CI/CD, and troubleshooting code-level problems.

Must Have

• Design, architect, solution, and implement data pipelines for enterprise consumption.
• Map and automate data flows from multiple sources.
• Develop workflows for Apache Spark applications using Airflow and AWS EMR.
• Deploy solutions via automated CI/CD processes for batch data pipelines.
• Develop, execute, and document software solutions for data collection, processing, transformation, and reporting.
• Interpret and translate business needs into technical requirements.
• Troubleshoot code level problems quickly and efficiently.
• Participate in code and document reviews.
• Strong experience in Python/PySpark, Spark/Spark SQL, Apache Iceberg/Airflow, Amazon EMR, Lambda, S3, Snowflake/Snowflake SQL, dbt, Fivetran, Hive, Hadoop.
• Extensive experience across layers of data engineering and pipelines.
• Strong experience in distributed and cloud architecture patterns.
• Ability to provide detailed architecture artifacts.
• Firm understanding of programming paradigms and data structures.
• Experience building and deploying data applications with AWS.
• Experience implementing CI/CD and familiarity with TDD/BDD.
• Strong experience designing and using data lakes and data warehouses.

Good to Have

• Familiarity with AI/Machine Learning workflows and technologies.
• Airflow (Astronomer is a plus).

Perks & Benefits

• Competitive compensation package
• Annual cash bonuses
• Stock grants
• Comprehensive benefits package
• Culture of belonging where everyone can thrive

Position Overview

Autodesk seeks a strong principal engineer and technical leader within the Data Acquisition team, designated with designing new robust and resilient data pipelines for analytical abstracts, enrichment, and metadata about design content. These datasets form the basis for data selection and understanding for various AI/ML projects at Autodesk.

This role collaborates with Product Managers, Program/Project managers, Analysts, Platform Architecture, and the Engineering Team to ensure data pipelines and delivery uses modern, scalable, reliable methods. Communication skills are critical, and you must be comfortable distilling complex technical information to a non-technical audience as a technical leader.

We work with a group of highly collaborative individuals who are excited to solve challenges, and implement cloud based, resilient, available, reliable solutions. We are comfortable with several languages and paradigms and aim to choose the technology which is most appropriate for the task. We endeavor to automate our tests and deployments end to end. We seek to mitigate existing issues and balance tech debt with the delivery of value to Autodesk. If you can identify with these traits, we are looking forward to working with you.

Responsibilities

• Designing, architecting, solutioning, and implementing data pipelines for enterprise consumption
• Mapping and automating data flows, from multiple sources
• Develop workflows to run Apache Spark applications using Airflow and AWS EMR; may also work on Snowflake applications using Airflow and dbt
• Deploy solutions via automated CI/CD processes to execute data pipelines running in batch
• Developing, executing, and documenting software solutions to address complex data collection, processing, transformation, and reporting issues
• Interpreting and translating business needs into technical requirements
• Troubleshooting code level problems quickly and efficiently as needed
• Participating in code and document reviews

Minimum Qualifications

• Strong experience in several of the following: Pytho / PySpark, Spark / Spark SQL, Apache Iceberg/Airflow (Astronomer is a plus), Amazon EMR, Lambda, S3, Snowflake / Snowflake SQL, dbt, Fivetran, Hive, Hadoop
• Extensive experience with a strong focus across layers of data engineering and pipelines
• Strong experience in distributed and cloud architecture patterns
• Ability to provide detailed architecture artifacts (L1 / L2 diagrams, sequence diagrams, etc.)
• Firm understanding of programming paradigms, their usage, as well as data structure
• Effective communication and collaboration skills, understanding how to tailor your message to your audience
• Ability to fully develop a feature across all data and/or application layers from inception
• Experience building and deploying data applications with AWS
• Experience implementing CI/CD and familiarity TDD/BDD. Preferably concrete examples of each
• Strong experience designing and using data lakes and data warehouses
• Powerful sense of ownership with a bias for action when challenges arise
• Ability to work independently and collaboratively across an organization.
• Desire to create, investigate and follow best practices
• Familiarity with AI/Machine Learning workflows and technologies is a plus
• Eagerness to learn and share knowledge with a good attitude

Learn More

About Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.

We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.

When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Salary transparency

Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Diversity & Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/diversity-and-belonging