Big Data Developer
Our client is seeking an experienced Hadoop/Big Data Developer to join our organization in providing hands-on development for the overall design and deployment of a Hadoop based platform. Developing conceptual, logical and physical target-state architecture, engineering and operational specs for Big Data solutions across all lines of businesses by effectively applying software engineering principles, patterns and best practices. The candidate will collaborate with business users and application teams across the organization to build consensus and to promote adoption of Big Data solutions effectively. Additionally, provides technology insight and expertise on industry trends
- Develop code using Big Data Technologies such as Hadoop and Storm/Spark to write MapReduce and In-memory applications using Java, SQL, Pig, Hive, Impala, etc.
- Turn business requirements into conceptual, logical and physical design for various data types and large volumes.
- Hands-on development of code that focuses on data ingestion as well as data analysis
- Architect, design and implement high performance, large volume data integration processes, database, storage, and other back-end services
- Implement security, encryption best practices for big data environments.
- Create and maintain data flow diagrams, entity relationship diagrams, system diagrams, business and logical systems integration as it pertains to Analytics/Business Intelligence and big data systems.
- Find and implement solutions to integrate event-based, semi-structured and 3rd party data sources (e.g. E-commerce transaction data) with enterprise relational models and strategies for transactional and analytical systems to effectively share data.
- Partner with our Enterprise Architecture team to ensure that the defined architectures align with other architecture domains (e.g. infrastructure, EDW, security, etc).
- Bachelor’s Degree in Computer Science, Mathematics, Engineering or related field/equivalent experience
- Ability to perform data related benchmarking, performance analysis and tuning
- Solid experience with NoSQL and MPP/columnar database technologies
- Demonstrated ability to architect and model mission critical BI, OLAP, OLTP, ETL, NoSQL, MPP, In-memory and batch Hadoop solutions leveraging multiple data technologies (Oracle, Teradata, Informatica; Hive, HBase; Impala, Solr, Spark)
- Operational expertise in the use of SQL/HQL/NoSQL
- Demonstrated experience translating business and technical requirements into comprehensive data reporting strategies and analytic solutions.
- Extensive background and expertise in developing and managing data technologies, technical operations, reusable data services, and related tools and technologies.
- Excellent communication and interpersonal skills as well as leadership and management abilities
- 5+ years of IT experience; 2 focused in areas of Big Data development and operation.
- 2+ years developing big data solutions with a major RDBMS platform (Oracle, Teradata)
- 2+ Experience with Hadoop ecosystem frameworks such as: MapReduce, Hive, Yarn, HBase, Pig Latin, Storm, Kafka, Flume HCatalog, Spark, Oozie, Zookeeper, Java and others
- Working knowledge of setting up, configuring and optimization of Hadoop clusters
- Experience with one or more of Hadoop-capable BI tools such as Datameer, Platfora, Splunk, and similar
- Experience with cloud and platform services, including self-provisioning, dynamic resource allocation and usage metering
- Ability to code within large scale, long-term data projects and work effectively in a cross-functional team.