Senior Data Engineer

Trantor · India

Data Engineer – AWS Data Platform

Job Summary

We are looking for a highly skilled and motivated data engineer with strong expertise in AWS data services to join our data platform team. The ideal candidate will have hands-on experience designing scalable data pipelines, workflow orchestration frameworks, and large-scale data migration solutions .

This role will be responsible for building robust cloud-native data engineering solutions on AWS , migrating datasets from legacy systems and data warehouses, and ensuring secure and efficient data processing pipelines across distributed environments.

Key Responsibilities

AWS Data Pipeline Development

• Design and implement scalable ETL/ELT data pipelines using AWS Glue, AWS Lambda, and AWS S3 .
• Build and maintain high-performance data ingestion frameworks for processing large-scale datasets.
• Implement data pipelines for data warehousing and analytics platforms such as AWS Redshift .
• Optimize storage and querying strategies using AWS S3 data lakes .

Data Workflow Orchestration

• Develop and maintain data workflow orchestration frameworks using tools such as Apache Airflow or AWS Step Functions .
• Automate complex workflows including data ingestion, transformation, validation, and loading processes .
• Build reusable and configurable workflows to support multiple data processing use cases.

Data Migration & Integration

• Lead data migrations from legacy data warehouse technologies to modern AWS data platforms.
• Perform data migration from RDBMS systems (e.g., MySQL, SQL Server, Oracle) to AWS S3 or AWS Redshift .
• Design scalable migration frameworks for large datasets with minimal downtime.
• Integrate data sources from enterprise applications and external systems.

Data Security & Governance

• Implement secure data pipelines using AWS security best practices .
• Manage access control and data governance using AWS IAM and Lake Formation .
• Ensure data encryption, access management, and compliance across all data platforms.

Performance Optimization & Monitoring

• Monitor data pipelines and troubleshoot performance issues.
• Optimize ETL workflows for scalability, reliability, and cost efficiency.
• Implement logging, monitoring, and alerting mechanisms for data pipelines.

Required Skills & Qualifications (Must Have)

• 5+ years of experience in Data Engineering or Data Platform development
• Strong hands-on experience with:
• AWS
• AWS Glue
• AWS S3
• AWS Lambda
• Experience with Data Workflow Orchestration tools such as Apache Airflow or AWS Step Functions
• Experience performing data migrations from other data warehouse technologies
• Experience performing data migrations from RDBMS systems to AWS S3 or AWS Redshift
• Strong expertise in Python and SQL for building scalable data pipelines
• Solid understanding of ETL/ELT concepts, data partitioning, and distributed data processing
• Experience working with version control systems such as GitLab or Bitbucket
• Strong debugging, analytical thinking, and problem-solving skills
• Basic understanding of Object-Oriented Programming concepts

Industry Knowledge & Experience

• Experience building cloud-native data engineering solutions on AWS
• Experience with data warehouse architectures and large-scale analytics platforms
• Hands-on experience with data extraction, transformation, and migration frameworks
• Experience working in high-volume data environments such as FinTech, analytics platforms, or enterprise data systems

Good to Have Skills

• IBM Cognos
• AWS Athena
• AWS Lake Formation
• AWS Redshift
• AWS Glue Data Catalog
• AWS SageMaker
• AWS IAM

Soft Skills

• Strong communication skills to present technical solutions and recommendations to stakeholders
• Ability to work cross-functionally in a fast-paced and evolving environment
• Detail-oriented with a proactive approach to identifying and solving data platform challenges
• Ability to collaborate effectively with data scientists, analysts, and platform engineering teams