Fluent Essays

Please read the question and attached PDF throughlyAs large volumes of raw and complex data, Big data enables programmers to take better decisions and optimize business processes by understanding customer behavior, latest trends, and changing patterns. An enterprise can collect, store, and analyze these large datasets in a number of ways. An enterprise can even use robust big data tools to store, access, and manage the structured and unstructured data collected from various sources in a faster and more efficient way. But no business can utilize and leverage big data optimally without addressing a number of challenges.Big data challenges are numerous: Big data projects have become a normal part of doing business — but that doesnt mean that big data is easy. The most obvious challenge associated with big data is simply storing and analyzing all that information. In its Digital Universe report, IDC estimates that the amount of information stored in the worlds IT systems is doubling about every two years. By 2020, the total amount will be enough to fill a stack of tablets that reaches from the earth to the moon 6.6 times. And enterprises have responsibility or liability for about 85 percent of that information. Much of that data is unstructured, meaning that it doesnt reside in a database. Documents, photos, audio, videos and other unstructured data can be difficult to search and analyze.In order for organizations to capitalize on the opportunities offered by big data, they are going to have to do some things differently. And that sort of change can be tremendously difficult for large organizations.While this weeks topic highlighted the uncertainty of Big Data, the author identified the following as areas for future research. Pick one of the following for your Research paper.:Additional study must be performed on the interactions between each big data characteristic, as they do not exist separately but naturally interact in the real world.The scalability and efficacy of existing analytics techniques being applied to big data must be empirically examined.New techniques and algorithms must be developed in ML and NLP to handle the real-time needs for decisions made based on enormous amounts of data.More work is necessary on how to efficiently model uncertainty in ML and NLP, as well as how to represent uncertainty resulting from big data analytics.Since the CI algorithms are able to find an approximate solution within a reasonable time, they have been used to tackle ML problems and uncertainty challenges in data analytics and process in recent years.To do:Read the required readingComplete the Research PaperYour paper should meet the following requirements:• Be approximately 5 pages in length, not including the required cover page and reference page.• Follow APA guidelines. Your paper should include an introduction, a body with fully developed content, and a conclusion.• Support your response with the readings from the course and at least five peer-reviewed articles or scholarly journals to support your positions, claims, and observations. The UC Library is a great place to find resources.• Be clear with well-written, concise, using excellent grammar and style techniques. You are being graded in part on the quality of your writing. uncertainty_in_big_data_analytics.pdf Unformatted Attachment Preview (2019) 6:44 Hariri et al. J Big Data https://doi.org/10.1186/s40537-019-0206-3 Open Access SURVEY PAPER Uncertainty in big data analytics: survey, opportunities, and challenges Reihaneh H. Hariri* , Erik M. Fredericks and Kate M. Bowers *Correspondence: rhosseinzadehha@oakland. edu Oakland University, Rochester, MI, USA Abstract Big data analytics has gained wide attention from both academia and industry as the demand for understanding trends in massive datasets increases. Recent developments in sensor networks, cyber-physical systems, and the ubiquity of the Internet of Things (IoT) have increased the collection of data (including health care, social media, smart cities, agriculture, finance, education, and more) to an enormous scale. However, the data collected from sensors, social media, financial records, etc. is inherently uncertain due to noise, incompleteness, and inconsistency. The analysis of such massive amounts of data requires advanced analytical techniques for efficiently reviewing and/ or predicting future courses of action with high precision and advanced decisionmaking strategies. As the amount, variety, and speed of data increases, so too does the uncertainty inherent within, leading to a lack of confidence in the resulting analytics process and decisions made thereof. In comparison to traditional data techniques and platforms, artificial intelligence techniques (including machine learning, natural language processing, and computational intelligence) provide more accurate, faster, and scalable results in big data analytics. Previous research and surveys conducted on big data analytics tend to focus on one or two techniques or specific application domains. However, little work has been done in the field of uncertainty when applied to big data analytics as well as in the artificial intelligence techniques applied to the datasets. This article reviews previous work in big data analytics and presents a discussion of open challenges and future directions for recognizing and mitigating uncertainty in this domain. Keywords: Big data, Uncertainty, Big data analytics, Artificial intelligence Introduction According to the National Security Agency, the Internet processes 1826 petabytes (PB) of data per day [1]. In 2018, the amount of data produced every day was 2.5 quintillion bytes [2]. Previously, the International Data Corporation (IDC) estimated that the amount of generated data will double every 2 years [3], however 90\% of all data in the world was generated over the last 2 years, and moreover Google now processes more than 40,000 searches every second or 3.5 billion searches per day [2]. Facebook users upload 300 million photos, 510,000 comments, and 293,000 status updates per day [2, 4]. Needless to say, the amount of data generated on a daily basis is staggering. As a result, techniques are required to analyze and understand this massive amount of data, as it is a great source from which to derive useful information. © The Author(s) 2019. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creat​iveco​mmons​.org/licen​ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Hariri et al. J Big Data (2019) 6:44 Advanced data analysis techniques can be used to transform big data into smart data for the purposes of obtaining critical information regarding large datasets [5, 6]. As such, smart data provides actionable information and improves decision-making capabilities for organizations and companies. For example, in the field of health care, analytics performed upon big datasets (provided by applications such as Electronic Health Records and Clinical Decision Systems) may enable health care practitioners to deliver effective and affordable solutions for patients by examining trends in the overall history of the patient, in comparison to relying on evidence provided with strictly localized or current data. Big data analysis is difficult to perform using traditional data analytics [7] as they can lose effectiveness due to the five V’s characteristics of big data: high volume, low veracity, high velocity, high variety, and high value [7–9]. Moreover, many other characteristics exist for big data, such as variability, viscosity, validity, and viability [10]. Several artificial intelligence (AI) techniques, such as machine learning (ML), natural language processing (NLP), computational intelligence (CI), and data mining were designed to provide big data analytic solutions as they can be faster, more accurate, and more precise for massive volumes of data [8]. The aim of these advanced analytic techniques is to discover information, hidden patterns, and unknown correlations in massive datasets [7]. For instance, a detailed analysis of historical patient data could lead to the detection of destructive disease at an early stage, thereby enabling either a cure or more optimal treatment plan [11, 12]. Additionally, risky business decisions (e.g., entering a new market or launching a new product) can profit from simulations that have better decisionmaking skills [13]. While big data analytics using AI holds a lot of promise, a wide range of challenges are introduced when such techniques are subjected to uncertainty. For instance, each of the V characteristics introduce numerous sources of uncertainty, such as unstructured, incomplete, or noisy data. Furthermore, uncertainty can be embedded in the entire analytics process (e.g., collecting, organizing, and analyzing big data). For example, dealing with incomplete and imprecise information is a critical challenge for most data mining and ML techniques. In addition, an ML algorithm may not obtain the optimal result if the training data is biased in any way [14, 15]. Wang et al. [16] introduced six main challenges in big data analytics, including uncertainty. They focus mainly on how uncertainty impacts the performance of learning from big data, whereas a separate concern lies in mitigating uncertainty inherent within a massive dataset. These challenges normally present in data mining and ML techniques. Scaling these concerns up to the big data level will effectively compound any errors or shortcomings of the entire analytics process. Therefore, mitigating uncertainty in big data analytics must be at the forefront of any automated technique, as uncertainty can have a significant influence on the accuracy of its results. Based on our examination of existing research, little work has been done in terms of how uncertainty significantly impacts the confluence of big data and the analytics techniques in use. To address this shortcoming, this article presents an overview of the existing AI techniques for big data analytics, including ML, NLP, and CI from the perspective of uncertainty challenges, as well as suitable directions for future research in these domains. The contributions of this work are as follows. First, we consider uncertainty challenges in each of the 5 V’s big data characteristics. Second, we review several Page 2 of 16 Hariri et al. J Big Data (2019) 6:44 techniques on big data analytics with impact of uncertainty for each technique, and also review the impact of uncertainty on several big data analytic techniques. Third, we discuss available strategies to handle each challenge presented by uncertainty. To the best of our knowledge, this is the first article surveying uncertainty in big data analytics. The remainder of the paper is organized as follows. “Background” section presents background information on big data, uncertainty, and big data analytics. “Uncertainty perspective of big data analytics” section considers challenges and opportunities regarding uncertainty in different AI techniques for big data analytics. “Summary of mitigation strategies” section correlates the surveyed works with their respective uncertainties. Lastly, “Discussion” section summarizes this paper and presents future directions of research. Background This section reviews background information on the main characteristics of big data, uncertainty, and the analytics processes that address the uncertainty inherent in big data. Big data In May 2011, big data was announced as the next frontier for productivity, innovation, and competition [11]. In 2018, the number of Internet users grew 7.5\% from 2016 to over 3.7 billion people [2]. In 2010, over 1 zettabyte (ZB) of data was generated worldwide and rose to 7 ZB by 2014 [17]. In 2001, the emerging characteristics of big data were defined with three V’s (Volume, Velocity, and Variety) [18]. Similarly, IDC defined big data using four V’s (Volume, Variety, Velocity, and Value) in 2011 [19]. In 2012, Veracity was introduced as a fifth characteristic of big data [20–22]. While many other V’s exist [10], we focus on the five most common characteristics of big data, as next illustrated in Fig. 1. Volume refers to the massive amount of data generated every second and applies to the size and scale of a dataset. It is impractical to define a universal threshold for big data volume (i.e., what constitutes a ‘big dataset’) because the time and type of data can influence its definition [23]. Currently, datasets that reside in the exabyte (EB) or ZB ranges are generally considered as big data [8, 24], however challenges still exist for datasets in smaller size ranges. For example, Walmart collects 2.5 PB from over a million customers every hour [25]. Such huge volumes of data can introduce scalability and uncertainty problems (e.g., a database tool may not be able to accommodate infinitely large datasets). Many existing data analysis techniques are not designed for large-scale databases and can fall short when trying to scan and understand the data at scale [8, 15]. Variety refers to the different forms of data in a dataset including structured data, semi-structured data, and unstructured data. Structured data (e.g., stored in a relational database) is mostly well-organized and easily sorted, but unstructured data (e.g., text and multimedia content) is random and difficult to analyze. Semi-structured data (e.g., NoSQL databases) contains tags to separate data elements [23, 26], but enforcing this structure is left to the database user. Uncertainty can manifest when converting between different data types (e.g., from unstructured to structured data), in representing data of mixed data types, and in changes to the underlying structure of the dataset at run time. From the point of view of variety, traditional big data Page 3 of 16 Hariri et al. J Big Data (2019) 6:44 Fig. 1 Common big data characteristics analytics algorithms face challenges for handling multi-modal, incomplete and noisy data. Because such techniques (e.g., data mining algorithms) are designed to consider well-formatted input data, they may not be able to deal with incomplete and/or different formats of input data [7]. This paper focuses on uncertainty with regard to big data analytics, however uncertainty can impact the dataset itself as well. Efficiently analysing unstructured and semi-structured data can be challenging, as the data under observation comes from heterogeneous sources with a variety of data types and representations. For example, real-world databases are negatively influenced by inconsistent, incomplete, and noisy data. Therefore, a number of data preprocessing techniques, including data cleaning, data integrating, and data transforming used to remove noise from data [27]. Data cleaning techniques address data quality and uncertainty problems resulting from variety in big data (e.g., noise and inconsistent data). Such techniques for removing noisy objects during the analysis process can significantly enhance the performance of data analysis. For example, data cleaning for error detection and correction is facilitated by identifying and eliminating mislabeled training samples, ideally resulting in an improvement in classification accuracy in ML [28]. Velocity comprises the speed (represented in terms of batch, near-real time, real time, and streaming) of data processing, emphasizing that the speed with which the data is processed must meet the speed with which the data is produced [8]. For example, Internet of Things (IoT) devices continuously produce large amounts of sensor data. If the device monitors medical information, any delays in processing the data and sending the results to clinicians may result in patient injury or death (e.g., a pacemaker that reports emergencies to a doctor or facility) [20]. Similarly, devices in the cyber-physical domain often rely on real-time operating systems enforcing strict timing standards on execution, Page 4 of 16 Hariri et al. J Big Data (2019) 6:44 and as such, may encounter problems when data provided from a big data application fails to be delivered on time. Veracity represents the quality of the data (e.g., uncertain or imprecise data). For example, IBM estimates that poor data quality costs the US economy $3.1 trillion per year [21]. Because data can be inconsistent, noisy, ambiguous, or incomplete, data veracity is categorized as good, bad, and undefined. Due to the increasingly diverse sources and variety of data, accuracy and trust become more difficult to establish in big data analytics. For example, an employee may use Twitter to share official corporate information but at other times use the same account to express personal opinions, causing problems with any techniques designed to work on the Twitter dataset. As another example, when analyzing millions of health care records to determine or detect disease trends, for instance to mitigate an outbreak that could impact many people, any ambiguities or inconsistencies in the dataset can interfere or decrease the precision of the analytics process [21]. Value represents the context and usefulness of data for decision making, whereas the prior V’s focus more on representing challenges in big data. For example, Facebook, Google, and Amazon have leveraged the value of big data via analytics in their respective products. Amazon analyzes large datasets of users and their purchases to provide product recommendations, thereby increasing sales and user participation. Google collects location data from Android users to improve location services in Google Maps. Facebook monitors users’ activities to provide targeted advertising and friend recommendations. These three companies have each become massive by examining large sets of raw data and drawing and retrieving useful insight to make better business decisions [29]. Uncertainty Generally, “uncertainty is a situation which involves unknown or imperfect information” [30]. Uncertainty exists in every phase of big data learning [7] and comes from many different sources, such as data collection (e.g., variance in environmental conditions and issues related to sampling), concept variance (e.g., the aims of analytics do not present similarly) and multimodality (e.g., the complexity and noise introduced with patient health records from multiple sensors include numerical, textual, and image data). For instance, most of the attribute values relating to the timing of big data (e.g., when events occur/have occurred) are missing due to noise and incompleteness. Furthermore, the number of missing links between data points in social networks is approximately 80\% to 90\% and the number of missing attribute values within patient reports transcribed from doctor diagnoses are more than 90\% [31]. Based on IBM research in 2014, industry analysts believe that, by 2015, 80\% of the world’s data will be uncertain [32]. Various forms of uncertainty exist in big data and big data analytics that may negatively impact the effectiveness and accuracy of the results. For example, if training data is biased in any way, incomplete, or obtained through inaccurate sampling, the learning algorithm using corrupted training data will likely output inaccurate results. Therefore, it is critical to augment big data analytic techniques to handle uncertainty. Recently, meta-analysis studies that integrate uncertainty and learning from data have seen a sharp increase [33–35]. The handling of the uncertainty embedded in the entire process of data analytics has a significant effect on the performance of learning Page 5 of 16 Hariri et al. J Big Data (2019) 6:44 from big data [16]. Other research also indicates that two more features for big data, such as multimodality (very complex types of data) and changed-uncertainty (the modeling and measure of uncertainty for big data) is remarkably different from that of small-size data. There is also a positive correlation in increasing the size of a dataset to the uncertainty of data itself and data processing [34]. For example, fuzzy sets may be applied to model uncertainty in big data to combat vague or incorrect information [36]. Moreover, and because the data may contain hidden relationships, the uncertainty is further increased. Therefore, it is not an easy task to evaluate uncertainty in big data, especially when the data may have been collected in a manner that creates bias. To combat the many types of uncertainty that exist, many theories and techniques have been developed to model its various forms. We next describe several common techniques. Bayesian theory assumes a subjective interpretation of the probability based on past event/prior knowledge. In this interpretation the probability is defined as an expression of a rational agent’s degrees of belief about uncertain propositions [37]. Belief function theory is a framework for aggregating imperfect data through an information fusion process when under uncertainty [38]. Probability theory incorporates randomness and generally deals with the statistical characteristics of the input data [34]. Classification entropy measures ambiguity between classes to provide an index of confidence when classifying. Entropy varies on a scale from zero to one, where values closer to zero indicate more complete classification in a single class, while values closer to one indicate membership among several different classes [39]. Fuzziness is used to measure uncertainty in classes, notably in human language (e.g., good and bad) [16, 33, 40]. Fuzzy logic then handles the uncertainty associated with human perception by creating an approximate reasoning mechanism [41, 42]. The methodology was intended to imitate human reasoning to better handle uncertainty in the real world [43]. Shannon’s entropy quantifies the amount of information in a variable to determine the amount of missing information on average in a random source [44, 45]. The concept of entropy in statistics was introduced into the theory of communication and transmission of information by Shannon [46]. Shannon entropy provides a method of information quantification when it is not possible to measure criteria weights using a decision–maker. Rough set theory provides a mathematical tool for reasoning on vague, uncertain or incomplete information. With the rough set approach, concepts are described by two approximations (upper and lower) instead of one precise concept [47], making such methods invaluable to dealing with uncertain information systems [48]. Probabilistic theory and Shannon’s entropy are often used to model imprecise, incomplete, and inaccurate data. Moreover, fuzzy set and rough theory are used for modeling vague or ambiguous data [49], as shown in Fig. 2. Evaluating the level of uncertainty is a critical step in big data analytics. Although a variety of techniques exist to analyze big data, the accuracy of the analysis may be negatively affected if uncertainty in the data o ... Purchase answer to see full attachment