Senior Data Engineer with Python and Databricks
4 Months ago • All levels
Data Analysis
Job Description
As a Senior Data Engineer, you will collaborate with data scientists and analysts to understand data needs and build efficient data pipelines. Your responsibilities include designing, implementing, and maintaining data pipelines for data ingestion, processing, and transformation using cloud services and other technologies. You will create and maintain data solutions in Azure using services like Data Factory, Synapse, Databricks/Spark, Polars/Pandas, and Fabric. Ensuring data quality, integrity, and dependability by implementing data validation and cleansing procedures is crucial, as is enhancing the scalability, efficiency, and cost-effectiveness of data pipelines. Furthermore, you will be responsible for monitoring and resolving data pipeline issues, ensuring data consistency and availability.
Must Have:
- Create and maintain data pipelines.
- Implement data validation and cleansing procedures.
- Work with Azure services like Data Factory, Synapse, and Databricks/Spark.
- Work together with data scientists and analysts to understand the needs for data and create effective data pipelines
- Solution design, implement, and maintain data pipelines for data ingestion, processing, and transformation using could services and other technologies
- Create and maintain data solutions IN Azure, using any of the following: Data Factory, Synapse, Databricks/Spark, Polars/Pandas, Fabric
- Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
- Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
- Monitoring and resolving data pipeline problems, ensuring consistency and availability of the data.
Set alerts for more jobs like Senior Data Engineer with Python and Databricks
Set alerts for new jobs by endava
Set alerts for new Data Analysis jobs in Romania
Set alerts for new jobs in Romania
Set alerts for Data Analysis (Remote) jobs
