07/10/2026
Job Description: Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks and PySpark Build robust ETL/ELT workflows for batch and near real‑time data processing Write optimized and efficient SQL queries for data transformation, validation, and analysis Work with large-scale distributed datasets using Spark-based processing frameworks Perform data ingestion from multiple sources (APIs, files, databases) into data lake/lakehouse architecture Ensure data quality, integrity, and consistency through validations and monitoring Optimize pipeline performance and cost (partitioning, caching, query tuning, etc.) Collaborate with cross-functional teams (Data Ops, Business, Analytics) to deliver data solutions Troubleshoot pipeline failures and implement proactive monitoring and alerting Maintain proper documentation for data models, transformations, and workflows Required Skills & Experience Strong hands-on experience...
Guardian
United States
Full Time
