Key Responsibilities

Process Structured and Unstructured Data

Gathers and processes raw, structured, semi-structured, and unstructured data using batch and real-time data processing frameworks.

Understands and enforces appropriate data master management techniques.

Data Validation and Quality

Ensures data quality and implements tools and frameworks for automating the identification of data quality issues.

Work with internal and external data providers on data validation providing feedback and making customized changes to data feeds and data mappings.

Agile Planning

Understands the challenges that the analytics organization faces in their day-to-day work and partner with them to design viable data solutions.

Recommends improvements to processes, technology, and interfaces that improve the effectiveness of the team and reduce technical debt.

Product Development

Implements and optimizes data solutions in enterprise data warehouses and big data repositories.

Installs, maintains, monitors, and supports business intelligence, distributed computation, and big data analytics tools.

Application Support

Provides ongoing support, monitoring, and maintenance of deployed products.

Advice and Guidance

Actively works with less experienced data engineers providing technical guidance and oversight.

Emerging Technology Monitoring

Actively participates in the engineering community, staying up to date on new data technologies and best practices and shares insights with others in the organization.

Skills

• Understand and comply with relevant architectural policies and procedures, taking responsibility for assessing and managing risks around the use of information.
• Ensure that information is presented effectively.
• Ensure that effective controls are in place for internal delegation, audit and control.
• Derive an overall strategy of data management, within an established information architecture (including both structured and unstructured data), that supports the development and secure operation of existing and new information and digital services.
• Plan effective data storage, security, sharing and publishing within the organization.
• Plan, establish and manage processes to access and independently validate external information from multiple sources.
• Design, code, test, correct and document programs and scripts from agreed specifications, and subsequent iterations, using agreed standards and tools, to achieve a well-engineered result.
• Take part in reviews of own work and leads reviews of colleagues' work.
• Manage application enhancements to improve business performance.

Qualifications

Essential

• Working experience in design, development, and implementation of highly scalable, high-volume software systems and components, client-facing web applications, and major Internet-oriented applications and systems
• Working experience working with high volume heterogeneous data, preferably with distributed systems such as Hadoop (or similar)
• Working experience with batch and real-time data processing frameworks
• Working experience with data modelling, data access, and data storage techniques
• Working experience working with relational and NoSQL databases
• Working experience with business intelligence tools and platforms
• Working experience with data quality tools
• Working experience with application lifecycle methodologies (e.g. waterfall, agile, iterative)
• Bachelor’s degree in Computer Science

Desirable

• Scrum Developer Certification or equivalent
• 5 – 8 years of working experience