Senior Data Engineer

Job Responsibilities

• Develop, build and maintain data pipeline architecture
• Assemble large and complex data sets that meet various business requirements in a functional manner
• Identify, create, and implement process improvements for internal use
• Key member of the involved with building the infrastructure required for optimal extraction and transformation of data from a wide variety of sources using Python and AWS and big data- technologies such as Glue, Lambda and EMR
• Work closely with various stakeholders including the Product Managers, Executives, Engineers and Design teams to assist with technical issues and support the data infrastructure.
• Work on various data tools for analytics and data scientist team members to help them in the process of building and optimizing the product into an world leader.

Experience Required

• Strong Python coding knowledge
• Previous experience creating/running Python or Pyspark projects working within AWS Glue.
• Experience working with AWS Data Lakes
• Previous work on AWS Data Pipeline and CI/CD processes
• Technology skills with big data tools: Hadoop, Spark, Kafka, etc.
• Working knowledge with data pipeline and workflow management tools: Luigi, Airflow, AWS
• Experience with stream-processing systems: Kinesis, Spark-Streaming, etc
• Familiarity with a variety of datasets, structured, semi structured and unstructured etc.
• Experience building and optimizing and big data- data pipelines, architecture and data sets.
• Strong analytic skills
• Experience and capacity to support and working with cross-functional teams in a highly dynamic and ever changing environment.

Essential and Desirable Skills

• 4+ years of total experience in a similar role
• Must have a degree in Computer Science, Statistics, Informatics or a related degree