Data Engineer – Tier 1 Data Team – NYC Investment Manager
Mondrian Alpha · New York, New York, United States
FULL TIMEpermanent
Job Description
About the RoleEvery strategy, every research notebook, every risk model at the firm reads from data this engineer's pipelines produce. The role owns the Spark and Databricks infrastructure that turns raw market, reference, and alternative data into something research and portfolio teams can trust without double-checking it. When this pipeline is fast and clean, research moves at the speed of ideas. When it isn't, every team downstream feels it the same day.
Scroll down the page to see all associated job requirements, and any responsibilities successful candidates can expect.
What You'll DoDesign, build, and optimize large-scale ETL pipelines in Spark and Databricks, processing high-volume market and alternative dataOwn performance at scale: partitioning, caching, cluster sizing, and cost, not just correctnessBuild the Delta Lake architecture that gives research and quant teams a single, reliable source of truthPartner with quants and researchers to turn their data requirements into production-grade, reusable pipelinesSet and enforce data quality and lineage standards across the Databricks environmentDrive the migration of legacy batch jobs onto a modern, scalable Spark architecture
Must-haves4+ years of hands-on Spark experience in production, with deep fluency in Databricks specifically. This is non-negotiable for the roleStrong PySpark or Scala, plus the SQL to matchReal experience with Delta Lake, job orchestration, and cluster performance tuning at scaleTrack record of taking a slow or fragile pipeline and making it fast and boring, in the best senseComfortable being the go-to Spark expert in the building, not just a contributor on someone else's pipeline
Nice-to-havesExperience with streaming data (Structured Streaming, Kafka) alongside batchExposure to financial market or tick data specificallyFamiliarity with MLflow or feature stores built on top of DatabricksAWS or Azure infrastructure experience supporting a Databricks environment
Why This RoleThis is the most technically demanding data seat in the building, and it's built that way on purpose. The firm has invested seriously in its Databricks environment and is looking for someone to take full ownership of it, not inherit someone else's half-finished setup. Comp is top of market, reflecting how central this infrastructure is to everything the fund does. xzbcwsv For a Spark engineer who wants their pipeline to be the thing every research team depends on and never has to think about, this is that.
Scroll down the page to see all associated job requirements, and any responsibilities successful candidates can expect.
What You'll DoDesign, build, and optimize large-scale ETL pipelines in Spark and Databricks, processing high-volume market and alternative dataOwn performance at scale: partitioning, caching, cluster sizing, and cost, not just correctnessBuild the Delta Lake architecture that gives research and quant teams a single, reliable source of truthPartner with quants and researchers to turn their data requirements into production-grade, reusable pipelinesSet and enforce data quality and lineage standards across the Databricks environmentDrive the migration of legacy batch jobs onto a modern, scalable Spark architecture
Must-haves4+ years of hands-on Spark experience in production, with deep fluency in Databricks specifically. This is non-negotiable for the roleStrong PySpark or Scala, plus the SQL to matchReal experience with Delta Lake, job orchestration, and cluster performance tuning at scaleTrack record of taking a slow or fragile pipeline and making it fast and boring, in the best senseComfortable being the go-to Spark expert in the building, not just a contributor on someone else's pipeline
Nice-to-havesExperience with streaming data (Structured Streaming, Kafka) alongside batchExposure to financial market or tick data specificallyFamiliarity with MLflow or feature stores built on top of DatabricksAWS or Azure infrastructure experience supporting a Databricks environment
Why This RoleThis is the most technically demanding data seat in the building, and it's built that way on purpose. The firm has invested seriously in its Databricks environment and is looking for someone to take full ownership of it, not inherit someone else's half-finished setup. Comp is top of market, reflecting how central this infrastructure is to everything the fund does. xzbcwsv For a Spark engineer who wants their pipeline to be the thing every research team depends on and never has to think about, this is that.
Details
| Company | Mondrian Alpha |
| Location | New York, New York, United States |
| Type | FULL TIME |
| Niche | finance |
| Experience | permanent |