Software Engineer II

 
The Bing Metrics team is looking for passionate full stack developers and data scientists to work on a new generation of metrics and quality control for the entire Bing search landscape. The team ensures that Bing shows high-quality, error-free, and authoritative results using a variety of different approaches. We routinely query petabytes of user activity data to uncover potential issues in user interactions with the search engine. We build complex pipelines including crowd judging and leverage the power of large language models (LLMs) to verify our suspicions. LLMs allow us to evaluate the quality of search results at multiple levels: query, answer, whole page and generate insights for the teams who are responsible for this experience.

As a part of an international and distributed team you will be responsible for identifying issues and implementing search quality metrics within Bing Search. The job provides you with the opportunity to work with multiple teams across the entire Bing organization (>80 different teams) and greatly influence search engine relevance and search result quality. We are an established core team in Bing with very high visibility and impact.

We are looking for a talented engineer who is detail oriented, with a passion to work with large scale computing, loves to design complex data pipelines built on top of LLM models, create new tools for running multi-step prompts to evaluate search engine quality and generate actionable insights for teams. If your blood boils when you see bad search results and you wish you could do something about them, this is the job for you!

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Required Qualifications:

• Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  
  • OR equivalent experience.
• 2+ years of experience in writing automation code with Chromium and Puppeteer.
• 2+ years of experience in developing solutions on Azure, utilizing Functions, WebJobs, Cloud Services, Azure Database, and Queues.
• 2+ years of experience with SQL, T-SQL, SQL Server.

Additional or Preferred Qualifications:

• 3+ years of experience with modern web development [HTML, JavaScript, jQuery].
• Experience in testing and relevance evaluation.
• Experience or deep interest in Large-Language Models (ChatGPT).
• Experience in (big) data and data analytics.

• Build tools and pipelines with Bing Logs using Big Data platforms.
• Design and implement E2E pipelines (from data collection, evaluation and result display).
• Design and implement tools for LLM models, engineer prompts for textual and multi-model LLMs for data processing and insight generation.
• Design and implement creative visualization for your results.