Role Summary
Capgemini is hiring an Azure Databricks Engineer across its major tech hubs in India. In this role, you will design and develop end-to-end cloud data pipelines, optimize ETL workflows, and implement robust data validation frameworks. You will work within an Agile environment to transform structured and semi-structured datasets into high-performance analytical assets across Azure Databricks, PySpark, and Azure Synapse environments.
Key Responsibilities
- Pipeline Engineering: Design and build end-to-end batch and real-time data pipelines using Azure Data Factory, Databricks, PySpark, SQL, Delta Lake, and Azure Data Lake Storage (ADLS).
- ETL & Data Warehousing: Optimize ingestion, transformation, and loading workflows for structured and semi-structured datasets entering Azure Synapse Analytics and ADLS.
- Optimization & Validation: Write efficient PySpark and SQL transformations, aggregations, and query optimizations while enforcing data validation and quality frameworks.
- Security & Compliance: Configure and manage secure access protocols using Azure Key Vault, Role-Based Access Controls (RBAC), and regulatory compliance standards.
- CI/CD & Agile Delivery: Build and maintain continuous integration and continuous deployment (CI/CD) pipelines in data environments while actively participating in Agile/Scrum delivery loops.
Key Qualifications
- Core Experience: 4 to 12 years of hands-on experience in Azure Databricks with strong PySpark development expertise.
- Cloud Stack Depth: Hands-on proficiency with Azure Data Factory (ADF) and Azure Data components (Azure SQL Database, Azure SQL Data Warehouse, Synapse Analytics).
- Programming Proficiency: Strong coding command in Python, PySpark, Scala, Hive, or SQL.
- Data Operations: Practical experience building CI/CD deployment pipelines within big data and cloud environments.
About Capgemini
Capgemini is an AI-powered global business and technology transformation partner, delivering tangible value through AI, technology, and human ingenuity across 50+ countries with nearly 420,000 team members globally.