Sr. data engineer

Data Engineer – Data Platform
Location: Bangalore
Experience: 5+ Years
Role Overview
We are looking for a hands-on Data Engineer to design, develop, and optimize scalable data platforms and data pipelines. The ideal candidate will have strong expertise in Python, SQL, Py Spark/Spark, Databricks , and modern data engineering practices.
Candidates should have hands-on experience with at least one cloud platform – AWS, Azure, or GCP. Experience with Big Query, Lakehouse architecture, Delta Lake, Airflow, Kafka, and CDC will be highly valued.
Key Responsibilities
Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, Py Spark, and Apache Spark.
Build and optimize data processing solutions using Databricks and cloud-native data services.
Develop batch and real-time data ingestion pipelines using technologies such as Airflow, Kafka, CDC , and other streaming/integration frameworks.
Implement modern Lakehouse and Medallion Architecture using Delta Lake or similar technologies.
Develop reliable data pipelines for structured and semi-structured data from multiple sources.
Implement data quality, validation, reconciliation, monitoring, logging, and error-handling mechanisms.
Optimize Spark jobs and data pipelines for performance, scalability, reliability, and cloud cost efficiency.
Work with cloud storage, compute, databases, and data services across AWS, Azure, or GCP.
Implement engineering best practices including Git, CI/CD, unit testing, integration testing, and code reviews.
Troubleshoot data pipeline failures and resolve performance and data-quality issues.
Collaborate with Data Architects, Solution Architects, Analysts, Developers, and Business Stakeholders to understand requirements and deliver scalable data solutions.
Contribute to technical design, documentation, coding standards, and continuous improvement of the data platform.
Must-Have Skills
5+ years of hands-on experience in Data Engineering.
Strong programming skills in Python.
Strong expertise in SQL and data manipulation.
Hands-on experience with Py Spark / Apache Spark.
Strong hands-on experience with Databricks.
Good understanding of ETL/ELT concepts, data pipelines, data modeling, and distributed data processing.
Experience with Lakehouse Architecture, Medallion Architecture, and Delta Lake.
Hands-on experience with at least one cloud platform: AWS / Azure / GCP.
Experience with one or more cloud data/storage services such as:
AWS: S3, Glue, EMR, Redshift
Azure: ADLS, Data Factory, Synapse
GCP: GCS, Big Query, Dataflow
Exposure to Apache Airflow or other workflow orchestration tools.
Experience with Kafka, CDC, or real-time/streaming data pipelines.
Good understanding of data quality, monitoring, testing, and pipeline optimization.
Experience with Git and CI/CD pipelines.
Strong analytical, troubleshooting, and problem-solving skills.
Good communication and collaboration skills.
Good-to-Have Skills
Apache Iceberg / Big Lake
dbt
Databricks Unity Catalog
Kafka / Structured Streaming
Change Data Capture (CDC) tools
Infrastructure as Code (Ia C) – Terraform or similar
Cloud security, IAM, networking, and access management
Cloud cost optimization
Data governance and metadata management
Experience working with data warehouses and dimensional data modeling
Preferred Technical Stack