Title: Sr. Data Engineer
Duration: 12 Months
Location: Remote
Job Description:
OTI's Office of Data Strategy (ODS) leverages city agency data to serve New Yorkers more equitably and effectively. Historically, our work has focused on analyzing data to optimize agency operations and making Open Data more accessible to the public. Today, ODS is expanding its impact by driving the strategy for non-public data-sharing infrastructure across city agencies. This includes automating data feeds, implementing robust quality assurance, and improving the overall user experience of the City's data-sharing platforms.
The MWBE resource will support the Office of Data Strategy (ODS) within the Office of Technology and Innovation (OTI) implementing data management strategy. This will include strategic input in delivery, development and integration of the City's Master Data Management (MDM) platform, as well as the building of analytic products and data integration platforms.
Scope of Services:
OTI's Office of Data Strategy (ODS) works with City agencies and their data to help serve New Yorkers more equitably and effectively. This work takes three main forms: analyzing data to improve City agency operations; making Open Data more accessible to all New Yorkers; and advancing citywide data infrastructure, integration, and sharing.
ODS seeks a Senior Data Engineer role to ensure the efficient and successful implementation and support of complex data engineering solutions for City agencies. This resource should demonstrate a solid understanding of industry-standard implementation methodologies using data integration engineering technologies, tools, and processes.
Tasks:
Support ODS implement MDM Person and Business domain that will integrate multiple sources of data and accompanying web services and APIs
Develop and support architectural blueprints that leverage the MDM as a foundational service, among other things, support the creation of unified profiles that can be used for such activities as linking business transactions relevant to a business owner
With guidance from ODS Technical Leads, coordinate the work of senior-level data engineers tasked with building pipelines to the MDM and other data platforms
Support the development of analytics tools that utilize the data pipeline to provide actionable insights into customer engagement and experience, operational efficiency, and other key business performance metrics
Review Extraction, Transformation, Load strategies (ETL) for data from a wide variety of data sources using SQL, cloud, and 'big data' technologies
Create, update, and maintain system documentation
Manage the development of APIs, for data access or landing data as output for further downstream consumption in the appropriate target data store
Coordinate appropriate security scan accreditations
Perform special projects and initiatives as assigned
Develop custom code and services that allow seamless integration between systems
Contribute to new technology projects as assigned and provide operations and maintenance support for existing applications
Unit test newly developed code and support system testing efforts
Troubleshoot errors, defects, and other system issues
Create and maintain excellent documentation (including data dictionaries, source-to-target maps, and database policies and guidelines) and source code repositories
Responsible for working with City's Cyber Command unit to follow and complete the application security accreditation process
Mandatory Skills/Experience
Note: Candidates who do not have the mandatory skills will not be considered
At least 8 years of experience managing the development and implementation of large technology projects, with at least 3+ years involving an MDM implementation
At least 8 years of experience in writing SQL
At least 8 years of experience in copying, transferring, manipulating, and automating data operations that were manual processes
Experience with tools and components of data architecture such as Informatica Power Center, IICS, SSIS, or similar ETL tools
Experience working with Amazon Web Services or Microsoft Azure cloud computing platform and services
In-depth knowledge of SQL and other database solutions
Knowledge of modeling database schemas for large datasets
Experience developing cloud-ready applications
Experience working with programming languages like Python, Java, and Perl
Desirable Skills/Experience:
5+ years hands-on experience in development with the suite of tools from Informatica PowerCenter and B2B Data Transformation
Experience using Oracle 10g/11g, SQL Server and/or a database appliance
Knowledge of metadata-driven enterprise reporting platforms
Experience with application development using Java Spring Boot Framework
Extensive knowledge of service-oriented architecture using SOAP and RESTful web services
Experience with 3-tier architecture
Experience with Oracle, Microsoft SQL Server, MySQL, RDS, PostgreSQL
Experience working hands-on in cloud-based platforms such as AWS, Microsoft Azure, and Snowflake is a plus
Experience with an agile, iterative development process and version control tools such as Git, GitHub, and Subversion