|
数据工程师
上班地点：上海浦东区湖滨路222号企业天地
Responsibilities?
? Design and develop critical data integration projects?
? Develop and maintain ETL (Extract, Transform, and Load) solutions for data-sharing projects
? Integrate data with third party API, rest & SOAP API.
? Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and ‘big data’ technologies.
? Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
? Develop custom data models and algorithms to apply to data sets.
? Work with data and analytics experts to strive for greater functionality in our data systems
? Develop and deliver the data service for internal system and 3rd party system?
Requirements
? Over 3 years of working experience in a data engineer role.
? Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
? Considerable experience in ETL tools and solutions like Spark, AWS glue, Fivetran, Stitch, Singer Tap and target.?
? Familiar with scheduling solution like Airflow, Azkaban etc.?
? Family with the version control solution like Github, GitLab
? Advanced knowledge of relational databases, including proficiency with SQL?
? Experience working with cloud data warehouse(AWS redshift, Neptune, Google Big Query, or other cloud data warehouse)
? Proficient in one or more of the following programming language: Python or Scala
? Familiar with programming language like php, js?
? Consolidated working experience in AWS or GCP platform
? Ability to work in fast paced environment and shift gears quickly.
? A successful history of manipulating, processing and extracting value from large disconnected datasets
? Ability to work in fast paced environment and shift gears quickly.
|