Data Engineer - GCP

We are seeking a highly skilled GCP Data Engineer with 5+ years of experience designing and building scalable, production-grade data platforms. This role requires deep expertise in Google Cloud Platform, Python, PySpark, SQL, and distributed data processing across the complete data lifecycle—including ingestion, transformation, orchestration, storage, quality, and serving.The ideal candidate is a hands-on engineer with strong experience building batch and real-time streaming pipelines, developing reusable Python data frameworks, optimizing large-scale workloads, and designing high-performance data models in BigQuery. The engineer will help establish reliable, secure, observable, and cost-efficient data solutions using modern GCP architecture and engineering practices.

Do not wait to apply after reading this description a high application volume is expected for this opportunity.
Key ResponsibilitiesDesign, develop, and maintain enterprise-scale batch and streaming data pipelines using Python, PySpark, Dataflow, Dataproc, Pub/Sub, BigQuery, and Cloud ComposerBuild modular, reusable, and testable Python frameworks for data ingestion, transformation, validation, and pipeline automationDevelop high-performance data-processing solutions using PySpark, Spark SQL, and Apache BeamIntegrate data from relational and NoSQL databases, REST APIs, flat files, cloud storage, third-party platforms, and event streamsBuild real-time streaming pipelines using Pub/Sub and Dataflow, including windowing, triggers, watermarking, late-arriving data, deduplication, and delivery semanticsDesign event-driven architectures supporting real-time analytics, operational reporting, and downstream data productsCreate and maintain workflow orchestration using Cloud Composer and Apache Airflow, including reusable operators, dependency management, retries, alerting, and recoveryDesign scalable BigQuery schemas using dimensional, normalized, denormalized, and lakehouse modeling patternsOptimize BigQuery workloads through effective partitioning, clustering, materialized views, query tuning, and cost-management strategiesTune Spark workloads by resolving data skew, excessive shuffling, inefficient joins, memory constraints, partitioning issues, and small-file problemsDevelop incremental loading, change data capture, and high-volume data-processing solutionsImplement automated data-quality checks, reconciliation, schema validation, lineage, logging, monitoring, and alertingDevelop Python-based data services and APIs using frameworks such as FastAPIApply security best practices using IAM, service accounts, encryption, secret management, and least-privilege accessBuild and maintain CI/CD pipelines for data applications using Cloud Build, GitHub Actions, or equivalent toolsProvision and manage GCP data infrastructure using TerraformCollaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable data solutionsCreate technical documentation, data mappings, operational runbooks, and architecture diagramsParticipate in code reviews and promote Python, SQL, GCP, and data-engineering best practices
Required Qualifications5+ years of hands-on experience in data engineering, including building production-grade data pipelines and platformsStrong programming expertise in Python, including object-oriented programming, reusable modules, exception handling, logging, testing, packaging, and performance optimizationAdvanced experience with PySpark, Spark SQL, and Apache SparkStrong knowledge of GCP data services, particularly BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, and Cloud StorageAdvanced SQL skills, including complex joins, window functions, CTEs, query optimization, and large-scale data transformationDeep expertise in BigQuery, including schema design, partitioning, clustering, performance tuning, workload management, and cost optimizationHands-on experience building batch and streaming pipelines with Dataflow and Apache Beam using PythonStrong experience designing and operating Spark workloads on DataprocExperience developing and managing Airflow DAGs using Cloud ComposerExperience designing Pub/Sub topics, subscriptions, schemas, retry policies, dead-letter queues, xzbcwsv and deduplication strategiesStrong understanding of distributed processing concepts, including partitioning, shuffling, parallelism, fault tolerance, and delivery guaranteesSolid understanding of Spark internals, including execution plans, DAGs, Catalyst optimization, memory management, and Spark UI troubleshootingExperience with dimensional data modeling, including star and snowflake schemas and SCD Type 1 and Type 2Experience processing large-scale partitioned datasets and implementing incremental data loadsExperience with data formats such as Parquet, Avro, ORC, JSON, and DeltaStrong understanding of streaming concepts, including event time, processing time, windowing, triggers, watermarking, and exactly-once versus at-least-once processingExperience implementing data-quality, observability, reconciliation, and production-monitoring frameworksProficiency with Git, automated testing, Agile delivery, and CI/CD practices
Preferred QualificationsExperience building Python APIs and microservices using FastAPI or FlaskWorking knowledge of Terraform and infrastructure-as-code practicesExperience with CDC technologies and patternsFamiliarity with Dataform or dbt for SQL-based data transformationsKnowledge of GCP security, networking, IAM, Secret Manager, and data-governance practicesGoogle Cloud Professional Data Engineer certification is preferred