Lead Data Engineer - Python, AWS, ETL
Job Summary
Job Description
We are seeking an experienced AWS Lead Data Engineer to design, build, and manage scalable data pipelines and platforms on AWS. This role involves hands-on expertise with AWS services such as Athena, Glue, Step Functions, Lambda, S3, and Lake Formation. Key responsibilities include developing ETL/ELT pipelines, building data lakes and warehouses, implementing data governance and security, and working with data scientists and business stakeholders. Strong Python, SQL, and AWS data services experience is required.
Must have:
- AWS data services: Glue, Athena, Step Functions, Lambda, Lake Formation, S3, EC2, Aurora, EBS/EFS, CloudFront
- PySpark, Python, SQL, PL/SQL proficiency
- ETL/ELT processes and data warehousing concepts
- Data modeling experience
- AWS orchestration/workflow management tools
- Debugging and performance tuning skills
Good to have:
- CDK
- GitHub
- Postgresql
- S3
- Glue
7 skills required
Job Details
Lead Data Engineer
- Python, SQL/PLSQL, AWS,Postgresql,S3, Glue Good to have : CDK, GitHub
- Location: Chicago, IL - Hybrid with 2-3 days on-site in a week
Job Description We are looking for an experienced AWS Lead Data Engineer to design, build, and manage robust, scalable, and high-performance data pipelines and data platforms on AWS. The ideal candidate will have a strong foundation in ETL fundamentals, data modeling, and modern data architecture, with hands-on expertise across a broad spectrum of AWS services including Athena, Glue, Step Functions, Lambda, S3, and Lake Formation.
Key Responsibilities:
- Design and implement scalable ETL/ELT pipelines using AWS Glue, Spark (PySpark), and Step Functions.
- Work with structured and semi-structured data using Athena, S3, and Lake Formation to enable efficient querying and access control.
- Develop and deploy serverless data processing solutions using AWS Lambda and integrate them into pipeline orchestration.
- Perform advanced SQL and PL/SQL development for data transformation, analysis, and performance tuning. Build data lakes and data warehouses using S3, Aurora, and Athena.
- Implement data governance, security, and access control strategies using AWS tools including Lake Formation, CloudFront, EBS/EFS, and IAM. Develop and maintain metadata, lineage, and data cataloging capabilities.
- Participate in data modeling exercises for both OLTP and OLAP environments.
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights. Monitor, debug, and optimize data pipelines for reliability and performance.
Required Skills & Experience:
- Strong experience with AWS data services: Glue, Athena, Step Functions, Lambda, Lake Formation, S3, EC2, Aurora, EBS/EFS, CloudFront.
- Proficient in PySpark, Python, SQL (basic and advanced), and PL/SQL.
- Solid understanding of ETL/ELT processes and data warehousing concepts.
- Familiarity with modern data platform fundamentals and distributed data processing.
- Experience in data modeling (conceptual, logical, physical) for analytical and operational use cases.
- Experience with orchestration and workflow management tools within AWS.
- Strong debugging and performance tuning skills across the data stack.
Know more about:
Know what it’s like to work and grow at Brillio: https://www.brillio.com/join-us/
Equal Employment Opportunity Declaration
Brillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.
#LI-SR1
Similar Jobs
IGT
West Greenwich, Rhode Island, United States (On-Site)
• 1 Month ago
QuinStreet
Pune, Maharashtra, India (On-Site)
• 2 Weeks ago
Blinkhealth
Pittsburgh, Pennsylvania, United States (On-Site)
• 2 Months ago
Qualcomm
Hyderabad, Telangana, India (On-Site)
• 2 Months ago
.png%3F1725396992&w=256&q=50)
Get notifed when new similar jobs are uploaded
Similar Skill Jobs
Scopely
Tel Aviv-Yafo, Tel Aviv District, Israel (On-Site)
• 2 Months ago
Get notifed when new similar jobs are uploaded
Jobs in Chicago, Illinois, United States
Inkittt
San Francisco, California, United States (On-Site)
• 11 Months ago
Apple
Sparks, Nevada, United States (On-Site)
• 2 Weeks ago
Palo Alto Networks
Santa Clara, California, United States (On-Site)
• 1 Month ago
InnoPhase IoT
San Diego, California, United States (On-Site)
• 2 Months ago
Rippling
San Francisco, California, United States (On-Site)
• 3 Months ago
Get notifed when new similar jobs are uploaded
Data Analysis Jobs
%3Aformat(webp)&w=256&q=50)
Get notifed when new similar jobs are uploaded
