About the job
Responsibilities:- Participate in efforts to design, build, and develop products in Big Data technologies.
- Build applications using Python, SQL, Databricks and AWS;
- Working understanding of Agile, Scrum, Design Thinking, and Lean Startup principles.
- Proactively identify and implement opportunities to automate tasks and develop reusable frameworks.
- Adhere to standard methodologies for coding, testing, and designing reusable code/component.
- Participate in sprint planning meetings and provide estimations on technical implementation.
- Contributed to the exploration and understanding of new tools and techniques and propose improvements to the data pipeline.
- Collaborate with the other engineering team members to ensure all services are reliable, maintainable, and well-integrated into existing platforms.
- Review functional and technical designs to identify areas of risk and/or missing requirements.
- Conduct complex data analysis and report on results.
- Prepare data for prescriptive and predictive modeling.
- Explore ways to enhance data quality and reliability.
- 5-7 years’ experience with data engineering using various AWS Technologies like S3, Athena, Glue, Data Lake, pySpark.
- Ability to learn quickly, be organized and detail oriented.
- Understanding of Spark, Hive, Kafka, Kinesis, Spark Streaming.
- Good experience in SQL Programming
- Experience in Software Engineering and Development.
- Understanding of database schema design.
- Proficient in the coding languages Python
