HDM: A Big Data Processing Framework Composable PROJECT TITLE : HDM: A Composable Framework for Big Data Processing ABSTRACT: Frameworks like MapReduce and Spark have been established in recent years to make constructing Big Data programs and applications easier. The jobs in these frameworks, on the other hand, are only loosely specified and bundled as executable jars, with no functionality exposed or explained. As a result, deployed jobs are not naturally composable and reusable for future development. It also makes it difficult to add optimizations to the data flow of job sequences and pipelines. The Hierarchically Distributed Data Matrix (HDM) is a functional, strongly-typed data representation for creating composable Big Data applications, which we discuss in this work. A runtime framework is included with HDM to help with application execution, integration, and management on distributed infrastructures. Multiple optimizations are implemented to increase the performance of running HDM jobs based on the functional data dependency graph of HDM. When compared to the current state of the art, Apache Spark, the experimental findings demonstrate that our optimizations can reduce Job-Completion-Time by 10 to 40% for various types of applications. Did you like this research project? To get this research project Guidelines, Training and Code... Click Here facebook twitter google+ linkedin stumble pinterest Data Analysis Machine Learning Projects Python Data Mining Projects Python Artificial Intelligence Projects Python Deep Learning Projects Python Data Science Projects Python Big Data Projects Distributed Processing Distributed Systems Big Data Processing Classification Methods for Identifying Soil Texture Classes Using Remote Sensing Images Confidence Measurement in a Face Recognition System Using Composite Features for Eye Detection
