Data Engineer

Data Engineer – Roles & Responsibilities

Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS Glue, Lambda, S3, Redshift, and Apache Spark.
• Build and optimize ETL/ELT workflows using PySpark, Python, and DBT for large-scale data processing.
• Develop and maintain complex Apache Airflow DAGs and AWS Step Functions for workflow orchestration.
• Create high-performance analytical SQL queries, stored procedures, window functions, and optimize database performance.
• Design and implement real-time data ingestion solutions using Apache Kafka and AWS Kinesis.
• Develop infrastructure using Terraform and AWS CloudFormation following Infrastructure-as-Code (IaC) best practices.
• Build and maintain CI/CD pipelines using GitHub Actions or similar DevOps tools.
• Design scalable data models and support enterprise data warehouse and analytics initiatives.
• Collaborate with Data Scientists, Product Managers, Business Analysts, and Application Development teams.
• Monitor data quality, pipeline reliability, and resolve production issues.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Ensure data governance, security, and compliance across AWS environments.
• Utilize AI-assisted development tools such as Claude Code or equivalent to improve engineering productivity.
• Support code reviews, technical documentation, and knowledge sharing across engineering teams.
• Experience with Databricks is highly preferred.

Required Skills

• AWS S3, Glue, Lambda, SQS, SNS, EventBridge, IAM
• Redshift
• Apache Spark, PySpark
• Python
• DBT
• Apache Airflow
• AWS Step Functions
• Kafka
• AWS Kinesis
• Terraform
• AWS CloudFormation
• GitHub Actions
• Analytical SQL
• Databricks (Preferred)
• Claude Code or similar AI coding tools