Role Summary
IBM India Private Limited is hiring a Data Engineer - Data Platforms (AWS) for its IBM Consulting FutureNow Center in Pune. In this role, you will design, build, and operate batch and real-time data pipelines across the AWS Cloud ecosystem. You will architect data layers using AWS RedShift, Aurora, and DynamoDB, manage data streams with AWS Kinesis and Kafka (MSK), and utilize open-source frameworks such as Apache Airflow, dbt, and Spark with Python or Scala.
Key Responsibilities
- AWS Data Pipeline Development: Design, build, and operate batch and real-time data ingestion pipelines using AWS EMR, AWS Glue, Glue Catalog, and Kinesis.
- Data Layer Architecture & Migration: Create optimized data layers across AWS RedShift, Aurora, and DynamoDB, executing seamless migrations via AWS DMS.
- Data Warehouse & Lakehouse Engineering: Develop scalable batch and streaming pipelines for enterprise Data Warehouses and Data Lakes using AWS Kinesis and Managed Streaming for Apache Kafka (MSK).
- Open-Source Tooling & Orchestration: Schedule and orchestrate data services using open-source technologies including Apache Airflow, dbt, and Spark with Python or Scala on AWS.
Key Qualifications
- Education: Bachelor's Degree in Computer Science, Information Technology, or related discipline (Master's Degree preferred).
- Experience: 5 to 8 years of dedicated hands-on experience in AWS data engineering and pipeline development.
- Mandatory Technical Stack: Deep expertise in AWS EMR, Glue, Glue Catalog, Kinesis, RedShift, Aurora, DynamoDB, DMS, Apache Airflow, dbt, and PySpark/Scala-Spark.
- Preferred Tools: Exposure to AWS Glue DataBrew, AWS Lambda functions with Python, and RedShift Spectrum.
- Core Competencies: Strong problem-solving abilities, technical curiosity, and collaborative engagement within hybrid cloud client transformation projects.
About IBM
IBM is a global leader in hybrid cloud, artificial intelligence, and enterprise consulting solutions, enabling digital transformation for Fortune 500 companies worldwide.