Principal Machine Learning Operations Developer

Job Description 

Position Overview 

We are looking for an experienced Principal Software Engineer to join our platform team focusing on AI/ML Platform (AMP). This team builds and maintains central components to fast track the development new ML/AI models such as model development studio, feature store, model serving, model observability.   Ideal candidate would have background in MLOps, Data engineering, DevOps with the experience of building high scale deployment architectures, observability. As an important contributor to our engineering team, you will help shape the future of our AI/ML capabilities, delivering solutions that inspire value for our organization. You will report to a manager. 

Responsibilities 

• System design: You will design, implement and manage software systems for the AI/ML Platform, orchestrating the full ML development lifecycle for the partner teams. 

• Mentoring: Spreading your knowledge, sharing best practices, doing design reviews to step up the expertise at the team level.  

• Multi-cloud architecture: Define components which leverages strengths from multiple cloud platforms (e.g., AWS, Azure) to optimize performance, cost, and scalability 

• AI/ML observability: You will build systems for monitoring performance of AI/ML models and finding insights on the underlying data such as drift detection, data fairness/bias, anomalies  

• ML Solution Deployment: You will develop tools for building and deploying ML artifacts in production environments, facilitating a smooth transition from development to deployment 

• Big Data Management: Automate and orchestrate tasks related to managing big data transformation and processing, building large-scale data stores for ML artifacts 

• Scalable Services: Design and implement low-latency, scalable prediction, and inference services to support the diverse needs of our users 

• Cross-Functional Collaboration: Collaborate across diverse teams, including machine learning researchers, developers, product managers, software architects, and operations, fostering a collaborative and cohesive work environment 

• End-to-end ownership: You will take the end-to-end ownership of the components and work with other engineers in the team including design, architecture, implementation, rollout, onboarding support to partner teams, production on-call support, testing/verification, investigations etc. 

Minimum Qualifications 

• Educational Background: Bachelors degree in Computer Science or equivalent practical experience 

• Experience: Over 8 years of experience in software development and engineering, delivering production systems and services 

• Prior experience of working with MLOps team at the intersection of the expertise across ML model deployments, DevOps, data engineering.  

• Hands-on skills: Ability to fluently translate the design into high quality code in golang, python, Java.  

• Knowledge of DevOps practices, containerization, orchestration tools such as CI/CD, Terraform, Docker, Kubernetes, Gitops 

• Demonstrated knowledge of distributed data processing frameworks, orchestrators, and data lake architectures using technologies such as Spark, Airflow, iceberg/ parquet formats.  

• Prior collaborations with Data science teams to deploy their models, setting up ML observability for inference level monitoring.  

• Exposure for building RAG based applications by collaborating with other product teams, Data scientists/AI engineers.  

• Demonstrated creative problem-solving skills with the ability to break down problems into manageable components 

• Knowledge of Amazon AWS and/or Azure cloud for solutioning large scale application deployments.  

• Excellent communication and collaboration skills, fostering teamwork and effective information exchange 

Preferred Qualifications 

• Experience of integrating with third party vendors  

• Experience in latency optimization with the ability to diagnose, tune, and enhance the efficiency of serving systems 

• Familiarity with tools and frameworks for monitoring and managing the performance of AI/ML models in production (e.g., MLflow, Kubeflow, TensorBoard) 

• Familiarity with distributed model training/inference pipelines using (KubeRay or equivalent) 

• Exposure to leveraging GPU computing for AI/ML workloads, including experience with CUDA, OpenCL, or other GPU programming tools, to significantly enhance model training and inference performance 

• Exposure to ML libraries such as PyTorch, TensorFlow, XGBoost, Pandas, and ScikitLearn