Data Engineer
4–6 years experience / Chennai
We are looking for a skilled Data Engineer to design, build, and maintain scalable data pipelines and platforms. You will work closely with data scientists, analysts, and business stakeholders to ensure reliable, high-quality data across our ecosystem.
Responsibilities
· Design and develop robust ETL/ELT pipelines using PySpark and SQL for large-scale data processing.
· Build, manage, and optimise data workflows on Databricks, including notebooks, jobs, and cluster configurations.
· Collaborate with analytics and ML teams to deliver clean, well-documented datasets.
· Monitor pipeline performance, troubleshoot failures, and ensure data quality and integrity.
· Define and implement data governance and best practices across the data platform.
· Participate in design reviews and contribute to architectural decisions.
Requirements
· 4–6 years of hands-on experience in data engineering or a related role.
· Strong proficiency in PySpark for large-scale distributed data processing.
· Advanced SQL skills — query optimisation, window functions, CTEs, and complex joins.
· Solid experience with Databricks — Delta Lake, workflows, Unity Catalog preferred.
· Experience working with cloud platforms (AWS/Azure).
Core skills
PySpark/SQL/Databricks/AWS / Azure /Python/Git
Nice to have
· Experience with streaming frameworks such as Apache Kafka or Spark Streaming.
· Exposure to ML pipelines and feature engineering workflows.
Requirements
· 4–6 years of hands-on experience in data engineering or a related role.
· Strong proficiency in PySpark for large-scale distributed data processing.
· Advanced SQL skills — query optimisation, window functions, CTEs, and complex joins.
· Solid experience with Databricks — Delta Lake, workflows, Unity Catalog preferred.
· Experience working with cloud platforms (AWS/Azure).