Data Engineer – Python & Pyspark (AWS)
Exp: 6+ yr
s Location : Chennai Client Place Guind
y Work Mode : All 5 day
s
Role Overvi
ew We're seeking a Senior Data Engineer with strong hands-on expertise in Python, Py Spark, and AWS to design, build, and operate scalable data pipelines for analytics and operational workloads. You will own end-to-end data engineering — from ingestion to delivery — across batch and near-real-time patterns, with a focus on pipeline reliability, data quality, and cloud-native engineerin
g. This is an engineering role — you will build and own production systems, not just develop queries or reports. Individual contributor expected to influence architecture, drive technical decisions, and mentor peers without formal people management responsibilitie
s.
Key Responsibilit
ies Design, build, and maintain scalable ETL/ELT pipelines using Python and Py Spark to ingest data from files, APIs, RDBMS, No SQL, and message queues into AWS data sto res Develop and optimize Py Spark jobs for large-scale data processing, transformation, and aggregation on AWS EMR or Databri cks Build and manage AWS-native data workflows using S3, Glue, Lambda, Redshift, and Step Functions; manage scheduling, dependencies, retries, and S LAs Write production-grade Python for data transformations, validations, orchestration, and API integrations — not notebook-level script ing Implement robust data quality controls: schema validation, deduplication, referential integrity checks, and anomaly detect ion Collaborate on data modeling (star/snowflake, data vault, medallion architecture) and define data contracts and line age Build CI/CD for data pipelines (tests, linting, packaging) using Git Hub Actions or AWS Code Pipeline; maintain documentation and runbo oks Troubleshoot and remediate production issues; drive continuous improvement in performance, reliability, and c
ost
Required Qualificat
ions6–8 years of hands-on data engineering experience with a strong focus on Python, Py Spark, and AWSProduction-grade Python skills — modular code, logging, error handling, REST/Graph QL API integration; not scripting or notebook-only work Strong Py Spark expertise — job optimization, partitioning, caching, broadcast joins, performance tuning at s cale Hands-on AWS experience: S3, Glue, Lambda, Redshift, EMR, Step Functions, IAMAdvanced SQL skills — complex joins, window functions, query optimization; Oracle or Postgre SQL experience is a plus Solid understanding of ETL/ELT patterns, data modeling, and data quality framew orks Experience with workflow orchestration: Apache Airflow, AWS Step Functions, or equiva lent CI/CD experience: Git Hub Actions, AWS Code Pipeline, Azure Dev Ops, or Git La b CIStrong engineering mindset — you build it, you own it, you operat
e it Nice-to-
Have Experience with Kafka, Kinesis, or Event Hubs for event stre aming Familiarity with dbt for ELT and data te sting Great Expectations or Deequ for data qu ality Docker / Kubernetes for containerized data ser vices Knowledge of data security patterns: PII masking, tokenization, column/row-level sec urity Informatica, ODI, or SSIS exp
osure Education & Certifica
tions Bachelor's/Master's in Computer Science, Engineering, or equi valent AWS Certified Data Engineer, AWS Solutions Architect, Databricks, or Snowflake certifications are a plus
