Lead Data Engineer

Proprietary Trading Firm · India

Experience Level: 5 - 7+ Years

Preferred Industry: High-Frequency Trading (HFT) or Hedge Funds

Key Responsibilities:

• Handle the entire data delivery process, including data procurement, vendor discussions, process setup, pipeline design and deployment (with logging and alerts), and database insertion, optimizing for minimal data delivery cycles.
• Build applications that simplify data querying for quantitative developers.
• Work with multiple datasets to deliver standardized data outputs.
• Break down large, abstract tasks into manageable deliverables within set deadlines.
• Identify business requirements, communicate them effectively to end users, and clarify technical details as needed.
• Distinguish between user needs and wants to prioritize effectively.
• Proactively explore and propose new ideas.
• Quickly assume responsibilities from senior staff to support the team.
• Adhere to safe data handling practices.

Essential Skills:

• Strong command of Python, Bash, and SQL in a Linux environment.
• Understanding of low-level system design for efficient data delivery applications.
• Ability to explain technical concepts to non-technical stakeholders.
• Hands-on experience with ClickHouseDB and its integrations with Python, Bash, and C++.
• Familiarity with Jira and Agile methodologies.
• Proficiency with Git and version control workflows.
• Comfortable developing over SSH.
• Knowledge of market, alternative, and supporting datasets with strong technical comprehension.
• Experience leading a team of at least 4 people and delivering robust data solutions.

Ideal Skills:

• Experience handling 1+ TB of daily data processing.
• Ability to build highly efficient and resource-optimized data pipelines.
• Up to date with hardware and software trends relevant to data engineering.
• Ability to write efficient and maintainable code without relying on LLMs.
• Experience working in fast-paced and high-availability proprietary trading environments.

Nice-to-Have Skills:

• Experience with serialization, lock-free queues, and multithreading in C++.
• Knowledge of multiprocessing and asynchronous programming in Python.
• Familiarity with various data processing frameworks.
• Experience with Azure data services.

Needful Qualities:

• Well-organized, detail-oriented; able to multi-task and keep track of various deadlines in a fast-paced, high-demand environment.
• Highest level of integrity, empathy, and character.
• Mature and thoughtful, with the ability to operate in a collaborative, team-oriented culture.

Why Join Us?

• Be part of a fast-growing, innovative firm at the forefront of financial technology.
• Work on challenging, high-impact projects with opportunities for rapid learning and career growth.
• Competitive compensation package with performance-based incentives.
• A dynamic work environment with exposure to cutting-edge technologies in trading.