Role Summary
IBM Consulting is seeking an experienced Data Engineer - Data Platforms (Google Cloud) to join our FutureNow Delivery Center in Gurugram in a hybrid working arrangement. In this role, you will design, implement, and maintain enterprise-grade batch and real-time data pipelines on Google Cloud Platform (GCP). You will architect data lakes and modern data warehouses utilizing BigQuery, Dataflow, Dataproc, and Pub/Sub, automate workflow orchestration using Cloud Composer (Airflow), and engineer scalable data migration strategies to accelerate client hybrid cloud transformations.
Key Responsibilities
- Pipeline Architecture & Ingestion: Design and build streaming and batch ETL/ELT pipelines using Google Cloud Dataflow (Apache Beam), Cloud Dataproc (Spark/Hadoop), and Pub/Sub.
- Cloud Data Warehousing & Modeling: Architect and optimize data layer storage, schema design, and query performance in Google BigQuery, BigTable, and Cloud Spanner.
- Orchestration & Workflow Automation: Schedule, deploy, and monitor complex DAGs and data pipelines across distributed environments using Cloud Composer (Managed Apache Airflow) and Cloud Scheduler.
- Enterprise Data Migration: Engineer end-to-end data migration and synchronization frameworks between legacy on-premises systems and GCP-native data platforms.
- Governance & Performance Tuning: Enforce data quality frameworks, partition and clustering strategies in BigQuery, and optimize compute and memory resource consumption across Spark jobs.
Key Qualifications
- Education: Bachelor's degree (Master's degree preferred) in Computer Science, Information Technology, Software Engineering, or a related discipline.
- Experience: 6 to 8 years of solid hands-on experience in data engineering, data warehousing, and distributed data processing frameworks.
- Core Stack: Strong technical expertise across Google Cloud ecosystem services (BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, BigTable), Python, Apache Spark, and SQL.
- Nice to Have: Hands-on experience with dbt, Apache Beam, Spark Scala, Cloud Spanner, AlloyDB, and valid Google Cloud Professional Data Engineer certification.
- Professional Competencies: Proven consulting skills, structured analytical debugging habits, and clear verbal and written stakeholder communication.
About IBM Consulting
IBM Consulting is the global professional services business of IBM, combining industry expertise with hybrid cloud and AI technologies to accelerate transformation for leading global enterprises.