Job Description
Big Data Developer [BDD]
- 4 to 5 years of total experience
- 2 to 3 years hands on experience in Hadoop
- Should have experience in MVC/Programming language/ETL
Scope of work
- Configure Big Data related services on the cluster
- Develop data ingestion & transformation components
- Testing and deployment of the Big Data related interfaces and scripts
- Schedule data pipelines in Oozie to enable jobs to load data in the data lake
- Release note and training manual
- Write scoop scripts for extracting data from R&M systems like (ISOM, Softmar, CSL)
- Use Apache Nifi & Kafka for streaming data and Spark for in-memory data processing
- Write shell / Perl / python scripts to run perform and run data transformations on extracted data
- Build necessary data views for visualizations / analytics tools like Zeppelin and Zoomdata for information delivery
- Data Ingestion pipelines setup and configured as a workflow (Unix + Oozie)
- Kafka and Spark configuration and testing
- Data ingestion use case development (Structured & Semi structured)
- Data ingestion use case testing (Structured & Semi structured)
- Deployment document / Training & Operational Document
- Data Transformation Model & Interfaces
- Generic Data ingestion routines framework
- Data ingestion use case development and testing (Unstructured)