Responsibilities

·         Translate application storyboards and use cases into functional applications.

·         Design, build, and maintain efficient, reusable, and reliable code.

·         Create and maintain optimal data pipeline architecture,

·         Assemble large, complex data sets that meet functional / non-functional business requirements.

·         Ensure the best possible performance, quality, and responsiveness of the applications.

·         Identify bottlenecks and bugs, and devise solutions to these problems.

·         Develop high performance & low latency components to run Spark clusters.

·         Interpreting functional requirements into design approaches those can be served through Big Data platform.

·         Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and big data technologies.

·         Collaborate and partner with Global Teams based across different locations.

·         Be able to propose best practices and standards, handover to the operations.

·         Perform the testing of software prototypes and transfer to the operational team.

·         Processing of data using Hive, Impala & HBASE.

·         Performing analysis of large data sets and derive insight.

Skills required

·         Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.

·         Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.

·         Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.

·         Strong analytic skills related to working with unstructured datasets.

·         Build processes supporting data transformation, data structures, metadata, dependency and workload management.

·         A successful history of manipulating, processing and extracting value from large disconnected datasets.

·         Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.

·         Experience supporting and working with cross-functional teams in a dynamic environment.

We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field. They should also have experience using the following software/tools:

·         Experience with big data tools: Hadoop, Spark, Kafka, etc.

·         Experience with relational SQL and NoSQL databases.

·         Experience with data pipeline and workflow management tools like Airflow, etc.

·         Experience with AWS cloud services: EC2, EMR, RDS, Redshift

·         Experience with stream-processing systems: Storm, Spark-Streaming, etc.

·         Experience with object-oriented/object function scripting languages: Python, Scala, Java, C++ etc.