Senior Data Engineer with Python and Databricks

As a Senior Data Engineer, you will collaborate with data scientists and analysts to understand data needs and build efficient data pipelines. Your responsibilities include designing, implementing, and maintaining data pipelines for data ingestion, processing, and transformation using cloud services and other technologies. You will create and maintain data solutions in Azure using services like Data Factory, Synapse, Databricks/Spark, Polars/Pandas, and Fabric. Ensuring data quality, integrity, and dependability by implementing data validation and cleansing procedures is crucial, as is enhancing the scalability, efficiency, and cost-effectiveness of data pipelines. Furthermore, you will be responsible for monitoring and resolving data pipeline issues, ensuring data consistency and availability.

Must Have

• Create and maintain data pipelines.
• Implement data validation and cleansing procedures.
• Work with Azure services like Data Factory, Synapse, and Databricks/Spark.

• Work together with data scientists and analysts to understand the needs for data and create effective data pipelines
• Solution design, implement, and maintain data pipelines for data ingestion, processing, and transformation using could services and other technologies
• Create and maintain data solutions IN Azure, using any of the following: Data Factory, Synapse, Databricks/Spark, Polars/Pandas, Fabric 
• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
• Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
• Monitoring and resolving data pipeline problems, ensuring  consistency and availability of the data.