Data Engineer II

Microsoft’s Cloud business is expanding, and the Cloud Supply Chain (CSCP) organization is responsible for enabling the infrastructure underlying this growth including AI! CSCP’s vision is to empower customers to achieve more by delivering Cloud Capacity Differentiated at Scale. Our mission is to deliver capacity for all cloud services predictably through intelligent systems driven by continuous learning and a world class organization. The CSCP Organization is responsible for traditional supply chain functions such as Plan, Source, Make, Deliver, but also manages supportability (spares), decommissioning and disposition of Data center assets worldwide. We deliver the core infrastructure and foundational technologies for Microsoft's over 200 online businesses including Bing, MSN, Office 365, Xbox Live, Skype, OneDrive and the Microsoft Azure platform for external customers. Our infrastructure is comprised of a large global portfolio of more than 200 datacenters supporting services for more than 1 billion customers in over 90 countries worldwide.

As part of the Decision Sciences Engineering team in Azure, we are at the heart of Microsoft’s cloud data strategy.  Our mission is to deliver the data for our customers & partners by combining the best of Data science and Engineering culture of delivering platforms and services on Azure Data platforms. Our Culture : Growth mindset – we practice a learn it all vs a know-it-all attitude. Wins are cheered, failures are celebrated and viewed as learning opportunities.  Diverse and Inclusive – every voice matters and is appreciated, where you can express yourself. Leadership oriented – each of us are leaders who strive to create clarity, generate energy, and deliver success. We are looking for a Data Engineer, who can demonstrate the following traits:  • A team player who is self-starter, customer obsessed, highly inquisitive and biased towards action. • Comfortable with cold start problems, thrive in ambiguity with an eye for detail. • Delivers results through innovation and persistence.  • Demonstrate depth in areas of Data Engineering, especially depth in Spark, • Demonstrates a product mindset and looks to deliver re-usable components that can be deployed as-is or into existing pipelines. • Have a flair for Artificial Intelligence/Machine Learning technology.

Minimum Qualifications

• Bachelor's Degree in Computer Science, Data Science, Mathematics or equivalent field OR Master's Degree in Computer Science, Data Science, Mathematics or equivalent • A successful history of architecting, building, and maintaining end-to-end, high-throughput data lake systems and their supporting services. • Minimum of 8+ years of hands-on development experience designing and building data lakes/datawarehouses • 7+ years of experience developing Big Data applications in the cloud using technologies such as Kusto, Databricks, Spark, Azure Data Factory, Azure ML, Azure Synapse or equivalent. • 8+ years of hands-on coding experience on either Scala, Spark SQL, pyspark, good to have optimized Query writing, Performance Tuning, Troubleshooting & development experience. • 8+ years of hands-on data pipelines and ETL/ELT experience • 5+ years of experience in building OLAP systems like Azure Analysis Services, Power BI etc. • Strong understanding of Data modeling including Dimensional and relational modeling.

Preferred Qualifications

• Proficient working knowledge of cloud-computing environments in the data stack - Azure preferred.  • Utilizing a variety of data stores, including data lakes/warehouses, relational database systems, in-memory caches, and searchable document databases. • Developing for continuous integration and automated deployments.

#cscpjobs

Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Understanding of modern data architecture patterns and concepts like Data mesh, Lambda Architecture, Delta lake etc. Understanding existing systems and lay out the path for a more modern and highly scalable data platform. • In depth understanding of Spark internals and passion for data engineering at scale. Identify, design, and implement process improvements to optimize our data delivery, and re-designing infrastructure and pipelines to achieve greater scalability. • Design, develop, and maintain data pipelines and back-end services for real-time decisioning, reporting, optimization, data collection, and related functions. • Ensure that all features embody engineering best practices, efficient, scalable architecture, elegant design, and high-quality implementation. • Collaborate with stakeholders, including Product, Executive and Domain teams to support them on data-related issues and data infrastructure needs. • Seek and incorporate continuous feedback to improve the efficiency of the business outcomes. • A team player who is self-starter, customer obsessed, highly inquisitive and biased towards action while engaging everyone in team. • Comfortable with cold start problems, thrive in ambiguity with an eye for detail. • Delivers results through innovation and persistence.  • A person with an eye for details and ‘can-do’ attitude. • Provide mentorship and guidance to other team members.