Lead I - Data Engineering - Azure Data Engineer

Role Description

Job Summary

The Azure Data Engineer is responsible for designing, developing, and maintaining scalable data pipelines and analytics solutions on Microsoft Azure. The role focuses on building robust ETL/ELT workflows, integrating data from multiple sources, optimizing data processing, and enabling reliable, high-performance data platforms using Azure Databricks, Azure Data Factory, and related Azure services.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory.
• Build scalable data processing solutions using Azure Databricks and Apache Spark.
• Develop and maintain data transformation workflows using PySpark, Spark SQL, and SQL.
• Ingest data from databases, APIs, cloud storage, and flat files into Azure data platforms.
• Implement data quality checks, monitoring, logging, and error-handling mechanisms.
• Integrate Azure services including Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL Database.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with data architects, business stakeholders, and engineering teams to deliver data solutions.
• Implement CI/CD pipelines and deployment automation for data engineering workflows.
• Ensure data security, governance, and compliance with organizational standards.
  
  

Must Have Skills

• Strong hands-on experience with Azure Databricks.
• Strong experience with Azure Data Factory (ADF).
• Expertise in Apache Spark, PySpark, and Spark SQL.
• Strong SQL development and query optimization skills.
• Experience with Python for data engineering.
• Knowledge of ETL/ELT concepts and data warehousing.
• Experience working with Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics.
• Understanding of Delta Lake architecture.
• Experience with Git-based version control and CI/CD practices.
• Familiarity with REST APIs and data integration patterns.
• Experience with performance tuning and optimization of Spark workloads.
  
  

Good to Have Skills

• Experience with Azure DevOps for CI/CD pipelines.
• Knowledge of Azure Functions, Logic Apps, or Event Hub.
• Exposure to Data Lakehouse architecture.
• Experience with monitoring and logging tools.
• Knowledge of data governance, security, and compliance best practices.
• Experience working in Agile/Scrum environments.
• Microsoft Azure Data Engineer certification is an added advantage.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent experience.
  
  

Skills

azure data factory,sql,databricks,python,performance tuning,