Back to search

Senior Data Platform Engineer

Design, build and manage complex, high-performance enterprise data services supporting data integration, reconciliation, data controls, data quality and operational intelligence.

  • Chennai, Tamil Nadu, India
  • Full-time
  • Posted Nov 18, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareData & Analytics

Responsibilities

  • Architect, develop and maintain complex, reusable data services for data ingestion, transformation, standardisation, validation, reconciliation and data processing using Scala, Spark, Python and Java.
  • Build scalable integration frameworks supporting files, databases, APIs, Kafka, CDC and enterprise messaging, including batch, near-real-time and streaming ingestion.
  • Develop high-performance reconciliation services supporting complex matching rules, multi-pass matching, tolerance handling, one-to-one and many-to-many matching, balance proofing and exception generation.
  • Engineer reusable control frameworks for completeness, accuracy, timeliness, record counts, control totals, source-to-target validation, auditability, data lineage, replay and recovery.
  • Develop configurable data quality services supporting schema validation, business rules, profiling, anomaly detection, data quality scoring, exception handling and automated remediation.
  • Build and optimise distributed processing workloads using Databricks, Apache Spark, S3, Apache Iceberg, Delta Lake, Unity Catalog and Lakeflow orchestration.
  • Develop Kafka/MSK-based event-processing services, CDC integration, stateful streaming, checkpointing, deduplication and reliable event delivery.
  • Design and develop REST APIs, microservices, metadata-driven processing frameworks, configuration services and reusable libraries using Scala, Java/Spring Boot and Python.
  • Integrate AI/ML capabilities for intelligent matching, anomaly detection, data quality remediation, metadata discovery and AI-assisted data processing.
  • Implement AWS infrastructure, Terraform, CI/CD, automated testing, observability, performance optimisation, FinOps, production support and disaster recovery.

Requirements

  • Bachelor’s degree/University degree or equivalent experience
  • Master’s degree
  • Experience in a Financial services

Nice to have: Enterprise reconciliation platform development, Real-time matching, Balance proofing, Stateful stream processing, AI/ML, MLflow, Vector search, RAG, Agentic AI integration, MongoDB Atlas, PostgreSQL/Aurora, Financial services data governance

Skills

ScalaJavaPythonApache SparkAWSDatabricksKafkaTerraformSQLMicroservicesREST APIsSparkPySparkSpark SQLStructured StreamingS3Apache IcebergDelta LakeUnity CatalogLakeflowMSKSpring BootFastAPIDockerGitCloudWatchOpenTelemetry

Full description

We are seeking a highly skilled, hands-on Senior Data Platform Engineer to design, build and manage complex, high-performance enterprise data services supporting data integration, reconciliation, data controls, data quality and operational intelligence. The role combines strong software engineering expertise with distributed data processing, cloud-native architecture and modern lakehouse technologies. The successful candidate will develop reusable, scalable and resilient data services supporting high-volume batch and real-time processing across enterprise applications. Key Responsibilities:- Enterprise Data Services Engineering: Architect, develop and maintain complex, reusable data services for data ingestion, transformation, standardisation, validation, reconciliation and data processing using Scala, Spark, Python and Java. Data Integration & Processing: Build scalable integration frameworks supporting files, databases, APIs, Kafka, CDC and enterprise messaging, including batch, near-real-time and streaming ingestion. Reconciliation & Matching Engines: Develop high-performance reconciliation services supporting complex matching rules, multi-pass matching, tolerance handling, one-to-one and many-to-many matching, balance proofing and exception generation. Data Controls & Integrity: Engineer reusable control frameworks for completeness, accuracy, timeliness, record counts, control totals, source-to-target validation, auditability, data lineage, replay and recovery. Data Quality Engineering: Develop configurable data quality services supporting schema validation, business rules, profiling, anomaly detection, data quality scoring, exception handling and automated remediation. Databricks & Lakehouse Engineering: Build and optimise distributed processing workloads using Databricks, Apache Spark, S3, Apache Iceberg, Delta Lake, Unity Catalog and Lakeflow orchestration. Streaming & Event-Driven Services: Develop Kafka/MSK-based event-processing services, CDC integration, stateful streaming, checkpointing, deduplication and reliable event delivery. Software & API Development: Design and develop REST APIs, microservices, metadata-driven processing frameworks, configuration services and reusable libraries using Scala, Java/Spring Boot and Python. AI & Intelligent Data Services: Integrate AI/ML capabilities for intelligent matching, anomaly detection, data quality remediation, metadata discovery and AI-assisted data processing. Cloud Engineering, DevOps & Reliability: Implement AWS infrastructure, Terraform, CI/CD, automated testing, observability, performance optimisation, FinOps, production support and disaster recovery. Essential Technical Skills:- Software Engineering: Strong hands-on Scala, Java or Python development, object-oriented design, concurrency, distributed systems and software design patterns. Distributed Processing: Apache Spark, Scala Spark, PySpark, Spark SQL, Structured Streaming and performance tuning. Cloud & Lakehouse: AWS, Databricks, S3, Apache Iceberg, Delta Lake and Unity Catalog. Data Integration: Kafka/MSK, CDC, REST APIs, JDBC, file processing and enterprise messaging. Data Services: Data transformation, reconciliation, rule engines, data validation, data controls, data quality and exception processing. Application Development: Microservices, REST APIs, Spring Boot/FastAPI, metadata-driven frameworks and service integration. Data Engineering: SQL, data modelling, schema evolution, partitioning, incremental processing and data optimisation. DevOps & SRE: Terraform, Docker, Git, CI/CD, automated testing, CloudWatch, OpenTelemetry and production troubleshooting. Desirable Skills:- Experience developing enterprise reconciliation or financial data processing platforms. Real-time matching, balance proofing and stateful stream processing. AI/ML, MLflow, vector search, RAG and agentic AI integration. MongoDB Atlas, PostgreSQL/Aurora and enterprise workflow integration. Financial services data governance, regulatory controls and operational resilience. Experience 8+ years of hands-on software development and data platform engineering. Proven experience building complex, high-throughput distributed data services. Strong development experience with Scala/Spark and cloud-native architectures. Experience designing reusable frameworks rather than only implementing ETL pipelines. Demonstrated ownership of production systems, performance, reliability and scalability. Experience delivering enterprise-scale platforms within regulated environments preferred. Expected Outcomes:- Scalable, reusable enterprise data services. High-performance batch and streaming reconciliation processing. Automated, auditable data quality and control frameworks. Reliable and cost-efficient Databricks/AWS workloads. Production-ready services with strong observability, resilience and automated recovery. Accelerated onboarding of new data sources and business use cases through configurable, self-service capabilities. Education: Bachelor’s degree/University degree or equivalent experience Master’s degree preferred ------------------------------------------------------ Job Family Group: Technology ------------------------------------------------------ Job Family: Applications Development ------------------------------------------------------ Time Type: Full time ------------------------------------------------------ Most Relevant Skills Please see the requirements listed above. ------------------------------------------------------ Other Relevant Skills For complementary skills, please see above and/or contact the recruiter. ------------------------------------------------------ Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster.

Senior Data Platform Engineer at Citi | Referso Jobs