An Architecture for Big Data Analysis in the Context of Social Media: The Case of Twitter

Abstract

Big Data analytics puts lots of pressure on ICT providers for developing new tools and technology to manage complex data. Challenges include storing and processing of huge volume of unstructured data, handling high-velocity data streams, cleansing noise and abnormality in the data, analyzing the data and finding value or meaning full results. Current tools and technologies are incapable to store, process and analyze huge amount of diverse data. In this research, we proposed an architecture that enables an effective storage and analysis of unstructured data, and developed a prototype to evaluate and test it. This research presents the work of investigating and designing a Big Data analysis solution using a MapReduce platform named Hadoop and a data warehouse infrastructure built on top of Hadoop called Hive which enables the analysis of unstructured data. The proposed architecture is validated through the development of a prototype that can analyze unstructured data using Hadoop MapReduce, HDFS (Hadoop File System), and Hive. We also evaluated whether this architecture is achieving its goals and objectives. The evaluation is conducted through streaming Twitter data, storing, processing, finally fetching and performing sentiment analysis. Twitter, one of the largest social media sites, is used as a data source in our experiment. Hence, this study provided an architecture that helps to address problems related to Big Data analysis and besides it is novel due to the use of MapReduce and Hive in a unique way for Twitter data analysis.  The results of this work can be applied by enterprises in sentiment analysis to understand how their customers feel about a particular product or service and to track how those opinions change over time, and also to get information regarding the relative performances of their competitors.