Data engineer | iit madras

Job Title: Data Engineer

Location & Work Mode: Hyderabad (On-Site) · Open to Relocation

Employment Type: Full-time

Experience Level: 2–3 years

About the Role: As a foundational member of the Statistical Experiments and Machine Learning Lab at the Department of Data Science and AI, IIT Madras, you will take full ownership of data reliability and pipeline scalability for NIRDPR (National Institute of Rural Development and Panchayati Raj). Your goal is to ensure engineering integrity across all scientific datasets on AWS, serving as a dependable guardian of data correctness and experimental freshness for NIRDPR rural policy projects.

Key Responsibilities

Reliability Engineering: Build and maintain highly dependable pipelines that guarantee data accuracy and availability for downstream analytics. Pipeline Scalability: Develop modular and efficient ETL processes capable of scaling alongside increasing data volumes without compromising performance. Data Integrity Audits: Enforce strict data quality tests and monitoring scripts to catch anomalies early and maintain absolute trust in the data. Foundational Support: Maintain the core Lab engineering standards on AWS that allow for consistent, predictable data flows across all NIRDPR public policy projects. SLA Adherence: Take personal accountability for meeting data delivery timelines, ensuring stakeholders have the fresh datasets they need.

Must-Have Requirements

Expert-level SQL and production-grade Python/Scala proficiency. Solid understanding of distributed computing (Apache Spark) and how to debug a slow, resource-heavy job. Strong grasp of data modeling (Dimensional modeling, OLTP vs OLAP). Deep commitment to system reliability, data integrity, and high-performance engineering standards in large-scale data environments. Strong problem-solving mindset with the ability to translate complex technical constraints into clear, non-technical business outcomes for stakeholders. Proficiency with AI coding assistants (e.g., Cursor, Cloud Code, Git Hub Copilot) for rapid development and demo creation. Data modeling expertise and performance optimization of data systems. Experience with big data tools (e.g., Spark, Kafka, Flink, Streaming, Docker/Containers). Solution Architecture experience and proficiency in AWS cloud architecture.

Preferred / Nice-to-Have

Streaming (Kafka, Flink, or Spark Structured Streaming). Modern platform (e.g., AWS-based solutions like Redshift, Glue, or EMR), dbt, orchestration (Airflow / Dagster), data-quality/observability tooling. Containers (Docker), basic CI/CD and Ia C. Data format expertise (e.g., Delta Lake, Apache formats).

Education Degree in CS, math, engineering, or related field.