Collection of code for submitting Spark/Hadoop/Hive/Pig tasks to EMR (AWS Elastic MapReduce) | #DE

A Hadoop Map-Reduce job to process the DBLP dataset to produce a graph depicting which professors at the CS department of UIC have co-authored publications.

This Repository contains the small projects related to Hive, Hadoop, and Spark. Its my contribution of learning new technology and provide my concise knowledge on big data different infrastructures.

Hadoop Map Reduce

This project aims at leveraging the capabilities of Apache's Hadoop framework by applying Map-Reduce primitives to analyze publication data. This data distributed as the dblp-dataset is analyzed for no. of publications per author, per year, no. of co-authors per paper etc.

This project implements a Byte Pair Encoding (BPE) tokenization approach along with a Word2Vec model to generate word embeddings from a text corpus. The implementation leverages Apache Hadoop for distributed processing and includes evaluation metrics for optimal dimensionality of embeddings.

Hadoop MapReduce Programs using Scala to process log files.

Perform MapReduce using Scala and Hadoop

Generate Some insights on the publically available Bibliography dataset - DBLP using Hadoop MapReduce, Scala

Distributed computational problem-solving project, which aims to perform large-scale graph matching using cloud computing technologies. The project allows users to import two directed graphs and analyze the differences between them.

Data Algorithms for Apache Spark RDD and Hadoop Mapreduce in Scala and Java

Implementation of a Map-Reduce application for distributed processing of Log files using Apache Hadoop and Scala

A distributed system built with Hadoop File System that employs map-reduce approach to analyze large volumes of data to extract insights

Running map reduce on a DBLP dataset to find the relations between UIC professors and their publications.

Average Temperature - Hadoop - Mapper - Reducer