Back to search

Data Engineer

Design and develop data integration solutions using Python/PySpark and AWS services.

  • Charlotte, North Carolina, United States
  • Posted Nov 11, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software

Responsibilities

  • Design and build batch/real-time ETL data pipelines
  • Develop data integration solutions
  • Implement data extraction, transformation, and loading processes
  • Optimize SQL queries and database performance
  • Create data models and perform data profiling
  • Integrate REST APIs
  • Automate workflows with schedulers
  • Maintain CI/CD pipelines

Skills

PythonPySparkAWS GlueAWS LambdaMSK (Kafka)S3Step FunctionsRDSEKSPostgreSQLSQL ServerOracleSybaseSQL performance tuningUNIX scriptingCI/CDBitbucketGitHubJenkinsControl-MAutosysAgile/SCRUMPythonPySparkAWS GlueAWS LambdaMSKS3Step FunctionsRDSEKSPostgreSQLSQL ServerOracleSybaseBitbucketGitHubJenkinsControl-MAutosys

More jobs nearby

Full description

Job Description The primary role of the Data Engineer is to function as a critical member of a data team by designing data integration solutions that deliver business value in line with the company's objectives. They are responsible for the design and development of data/batch processing, data manipulation, data mining, and data extraction/transformation/loading into large data domains using Python/Pyspark and AWS tools. Job Responsibilities •Developer experience specifically focusing on Data Engineering. •Hands-on experience in Development using Python and Pyspark as an ETL tool. •Experience in AWS services like Glue, Lambda, MSK (Kafka), S3, Step functions, RDS, EKS etc. •Experience in Databases like Postgres, SQL Server, Oracle, Sybase etc. •Experience with SQL database programming, SQL performance tuning, relational model analysis, queries, stored procedures, views, functions, and triggers. •Strong technical experience in Design (Mapping specifications, HLD, LLD), Development (Coding, Unit testing). •Knowledge in developing UNIX scripts, Oracle SQL/PL-SQL. •Experience with data models, data mining, data analysis and data profiling. •Experience in working with REST API's. •Experience in workload automation tools like Control-M, Autosys etc. •Good knowledge in CI/CD DevOps process and tools like Bitbucket, GitHub, Jenkins. •Strong experience with Agile/SCRUM methodology. •Experience with other ETL tools (DataStage, Informatica, Pentaho, etc.). •Knowledge in MDM, Data warehouse and Data Analytics. Core Objective: Design and build batch/real-time ETL data pipelines, integrations, and large-scale data solutions using Python/PySpark on AWS. Non-Negotiable Must-Haves (Screening Criteria) • Core Stack: Hands-on development using Python and PySpark for scalable ETL/ELT pipelines. • AWS Data Services: Production experience with AWS Glue, Lambda, MSK (Kafka), S3, Step Functions, and RDS. • Databases & Advanced SQL: Deep experience with relational databases (PostgreSQL, SQL Server, Oracle) and advanced SQL programming (performance tuning, stored procedures, complex queries, data modeling). • System Design & Linux: Solid background in High-Level/Low-Level Design (HLD/LLD), data mapping specs, and shell/UNIX scripting. Nice-to-Haves & Secondary Tools • CI/CD & Schedulers: Git (Bitbucket/GitHub), Jenkins, and enterprise schedulers (Control-M, Autosys). • Legacy/Alternative ETL: Exposure to tools like Informatica, DataStage, or Pentaho. • Concepts: Master Data Management (MDM), REST API integrations, and Data Warehousing best practices.

Data Engineer at Expert In Recruitment Solutions | Referso Jobs