Back to search

Senior PySpark Data Engineer (AWS)

We are seeking a highly skilled PySpark Data Engineer with strong experience in building scalable, high-performance data platforms and ETL pipelines. The ideal candidate should possess deep expertise in PySpark, SQL, Python, and distributed data processing, along with exposure to AWS-based data engineering ecosystems including EMR, Redshift, S3, Glue, and Apache Iceberg.

  • Bangalore, India +1 more
  • Full-time
  • ₹70K–₹130K per month
  • Posted Nov 11, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareData & Analytics

Locations (2)

  • Bangalore, India
  • Hyderabad, Telangana, India

Responsibilities

  • Design, develop, and optimize large-scale batch and data processing pipelines using PySpark and EMR.
  • Build and maintain scalable ETL/ELT frameworks for ingestion, transformation, and consumption layers.
  • Develop data pipelines that process high-volume datasets with a focus on performance, reliability, and scalability.
  • Implement data quality validations, reconciliation frameworks, and monitoring capabilities.
  • Work with structured, semi-structured, and unstructured data sources.
  • Develop and optimize data models for analytics and reporting use cases.
  • Collaborate with cross-functional teams to understand business requirements and transform them into technical solutions.
  • Troubleshoot production issues, perform root cause analysis, and implement preventive measures.
  • Mentor junior engineers and drive engineering best practices across the team.

Requirements

  • Experience in a Cloud modernization
  • Experience in a Data platform transformation
  • Experience in a Medallion Architecture

Nice to have: Amazon Redshift, AWS Glue, Amazon S3, AWS Athena, AWS Lambda, AWS Step Functions, Apache Iceberg, Delta Lake, Lakehouse Architecture, Metadata-driven data pipelines, Kafka, Spark Streaming, Kinesis

Skills

PySparkSQLPythonData EngineeringAmazon EMRPySparkApache SparkAmazon EMRAmazon RedshiftAWS GlueAmazon S3AWS AthenaAWS LambdaAWS Step FunctionsApache IcebergDelta LakeKafkaSpark StreamingKinesis

Full description

Request Information Request: Information Technology_IND - IND_Analyst Qty: 1 Candidate Submission Limit Per Supplier: 5 Candidate Submission Limit Per Request: 45 Desired Start Date: 11/12/2026 End Date: 5/12/2027 Hrs/Wk: 45.00 MSP Owner: G, Reshma GBAMS Requisition ID: 11042942 Estimated Expense: Rs0.00 (Sum of Expense * Quantity of Candidates) General Information Job Description: Role Descriptions: pyspark is the official Python API for Apache Spark| an open-source| distributed computing framework built for large-scale data processing| data engineering| and machine learning. Essential Skills: pyspark is the official Python API for Apache Spark| an open-source| distributed computing framework built for large-scale data processing| data engineering| and machine learning. Desirable Skills: Keyword: Skills: Digital : PySpark Experience Required: 6-8 Senior PySpark Data Engineer (AWS) Experience: 5-10 Years Location: Bangalore / Hyderabad Employment Type: Full-Time Role Summary We are seeking a highly skilled PySpark Data Engineer with strong experience in building scalable, high-performance data platforms and ETL pipelines. The ideal candidate should possess deep expertise in PySpark, SQL, Python, and distributed data processing, along with exposure to AWS-based data engineering ecosystems including EMR, Redshift, S3, Glue, and Apache Iceberg. The candidate will work closely with architects, business stakeholders, and analytics teams to design, build, optimize, and support enterprise-scale data solutions. ________________________________________ Key Responsibilities • Design, develop, and optimize large-scale batch and data processing pipelines using PySpark and EMR. • Build and maintain scalable ETL/ELT frameworks for ingestion, transformation, and consumption layers. • Develop data pipelines that process high-volume datasets with a focus on performance, reliability, and scalability. • Implement data quality validations, reconciliation frameworks, and monitoring capabilities. • Work with structured, semi-structured, and unstructured data sources. • Develop and optimize data models for analytics and reporting use cases. • Collaborate with cross-functional teams to understand business requirements and transform them into technical solutions. • Troubleshoot production issues, perform root cause analysis, and implement preventive measures. • Mentor junior engineers and drive engineering best practices across the team. ________________________________________ Must-Have Skills Data Engineering • 5-10 years of hands-on Data Engineering experience. • Strong expertise in: o PySpark o SQL o Amazon EMR • Experience building enterprise-grade ETL/ELT pipelines. • Strong understanding of: o Incremental loads o CDC patterns o Partitioning strategies o Data quality frameworks o Error handling and restartability Good-to-Have Skills AWS Data Engineering Experience in one or more of the following: • Amazon Redshift • AWS Glue • Amazon S3 • AWS Athena • AWS Lambda • AWS Step Functions Modern Data Lake Technologies • Apache Iceberg • Delta Lake • Lakehouse Architecture • Metadata-driven data pipelines Streaming Technologies • Kafka • Spark Streaming • Kinesis ________________________________________ Preferred Experience • Experience working on cloud modernization or data platform transformation programs. • Exposure to Medallion Architecture (Bronze/Silver/Gold). • Experience designing reusable frameworks and common ingestion patterns. • Experience supporting analytics and reporting workloads. • Knowledge of enterprise data governance and lineage concepts. Comments for Suppliers: Rate Details Rate Card Class: IND_Analyst Category: Category 4 Level: Level II Bill Rate: Rs70,000.00 - Rs130,000.00 Monthly NOOT Markup: $0.00

Senior PySpark Data Engineer (AWS) at ClifyX, INC | Referso Jobs