Data Engineer

We are seeking an experienced and highly motivated Data Engineer to design, build, and operate reliable data pipelines and analytics-ready datasets that power reporting and business decision-making. In this role, you will develop and optimize ETL/ELT processes using Python, Azure Databricks, and SQL, integrating data across platforms including Microsoft SQL Server and PostgreSQL. You will partner with analysts, data consumers, and engineering teams to translate business requirements into scalable, well-governed data products, applying strong engineering practices around testing, monitoring, performance, and data quality, while using AI-assisted development tools and generative AI to enhance productivity and solution quality. We are looking for someone with hands-on experience integrating AI capabilities into enterprise data workflows and a strong understanding of responsible AI principles, including data privacy, fairness, and explainability.

Key duties and responsibilities:

• Data Pipeline Development (ETL/ELT): Design, build, and maintain batch and/or incremental pipelines using Python, Azure Databricks, and SQL to ingest, transform, and curate data for analytics and downstream applications.

• Orchestration, Scheduling & Reliability: Automate and schedule pipeline execution, implement idempotent processing, and build monitoring/alerting and operational runbooks to ensure reliable, observable data products.

• Production Operations Support: Own day-to-day production support for data pipelines and reporting workflows, including incident triage and resolution, root-cause analysis, and proactive monitoring to meet SLAs and ensure data reliability.

• Databricks Engineering: Develop and optimize notebooks and jobs; implement reusable libraries, parameterized workflows, and cluster/job configurations; apply performance tuning techniques (partitioning, caching, query optimization) as appropriate.

• SQL performance optimization: Optimize schemas, tables, views, and SQL logic across Microsoft SQL Server and PostgreSQL. Troubleshoot performance issues and ensure data integrity through constraints, indexing, and query tuning.

• Data Quality & Controls: Define and implement validation rules, reconciliation checks, and automated tests to detect anomalies, enforce SLAs, and improve trust in reporting and analytics outputs.

• AI Enablement & Productivity: Use AI-assisted development tools and generative AI to improve engineering productivity, data pipeline quality, and documentation while maintaining strong validation and governance practices.