Role Summary
American Express is seeking a skilled Data Engineer II to join our Technology organization in Gurugram, Haryana, India. In this hybrid role, you will design, enhance, and optimize large-scale data pipelines and database architectures powering critical financial services and analytics platforms. Leveraging Google Cloud Platform (GCP) technologies—including BigQuery and Dataproc—alongside Apache Spark and distributed database frameworks, you will drive seamless data integration, implement indexing and partitioning strategies, and ensure optimal performance for both batch and real-time processing systems.
Key Responsibilities
- Data Pipeline Engineering: Build, enhance, and maintain scalable, reliable, and secure data pipelines and architectures across batch and streaming workloads.
- GCP & Big Data Implementation: Utilize Google Cloud services—specifically BigQuery and Dataproc—in conjunction with Apache Spark to ingest, process, and analyze massive distributed datasets.
- Database Optimization & Partitioning: Design relational and NoSQL data models; implement partitioning, indexing, and clustering mechanisms to boost query performance across OLAP and OLTP systems.
- Performance Tuning: Profile and tune execution plans for large-scale data workflows, mitigating bottlenecks and optimizing resource allocation across compute engines.
- Architecture Alignment & Design Reviews: Evaluate data requirements and technical stories, maintain comprehensive data models, participate in design reviews, and provide production support.
- Cross-Functional Collaboration: Partner closely with product management, analytics, and business engineering teams within Agile/Scrum delivery frameworks to align data capabilities with business goals.
Key Qualifications
- Education: Bachelor's or Master's degree in Computer Science, Computer Engineering, or equivalent practical experience.
- Experience: 4+ years of hands-on software development and data engineering experience working with distributed systems.
- Core Technical Skills: Strong proficiency in Python or Java, advanced SQL, Google Cloud Platform (GCP) data services (BigQuery, Dataproc), Apache Spark, and NoSQL databases.
- Key Methodologies: Big Data partitioning and indexing, batch and real-time processing, distributed computing, database replication/security, and Agile/Scrum software delivery.
- Preferred Capabilities: Hands-on exposure to Cloud Spanner, BigTable, HBase, Hive, or relevant GCP certifications (e.g., Google Professional Data Engineer).
About American Express
American Express is a globally integrated payments company with a 175-year heritage of innovation, integrity, and exceptional service. Delivering differentiated products, world-class customer support, and advanced technology capabilities, American Express backs its customers, communities, and colleagues with opportunities for career growth, inclusivity, and holistic well-being.