Data Engineer

Description

• Design, build, and maintain scalable ETL/ELT pipelines to integrate data from diverse sources, optimizing for performance and cost efficiency.
• Leverage Databricks and other modern data platforms to manage, transform, and process data for ML and AI models, supporting both real-time and batch processing workflows.
• Work with cross-functional teams to implement data solutions that support model training, monitoring, and production deployment.
• Collaborate with a cybersecurity research team to understand emerging threats and develop solutions that leverage advanced data analytics.
• Design and develop innovative prompts and instruction sets to enhance our autonomous LLM-based labeling platform.
• Optimize prompts to generate high-quality, coherent, and contextually relevant responses.
• Collaborate with software and data engineers to integrate ML/LLM techniques into production systems.

• 3+ years of experience in data engineering, including cloud-based data solutions
• Proven expertise in implementing large-scale data solutions.
• Proficiency in Python. PySpark is a plus.
• Experience with cloud and big data technologies such as Databricks and Azure Data factory.
• Experience with prompt engineering techniques.
• Experience with vector databases and embedding techniques is a plus.
• Experience with MLOps is a plus.
• Strong analytical and problem-solving skills, with the ability to evaluate and interpret complex data.
• Excellent communication and collaboration skills, with the ability to work effectively in a multidisciplinary team.
• Proven track record of delivering high-quality results in a fast-paced and dynamic environment.