Fluent Essays

instructions 1. Go through the Power Point presentation.2. Write short note on following Use cases (Any one) focusing on how Data Science and Big Data analytics impacts various functionalities of particular domain1-Banking2-Classification of emails –Spam/Ham3-E-commerce4-Retail chains – Starbucks5-Predicting Election Results introduction_to_data_science_week_2.pdf Unformatted Attachment Preview 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 1 INTRODUCTION TO DATA SCIENCE Week-2 Understanding Data Science Dr. Anand Rajavat Professor & Director Shri Vaishnav Institute of Information Technology Shri Vaishnav Vidyapeeth Vishwavidyalaya-Indore, India 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 2 Introduction to Big Data • In this Session, we will discuss a brief introduction to the Data • • • • • Science Various domains where Big Data Analytics and Data Science are used? Discuss aspects of Data Science Vs Analysis Vs Software Delivery Aspects of Data Scientists vs. Data Analysts vs. Data Engineers How Businesses Benefit from Data Scientists Data Science Goals and Deliverables 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 3 Introduction to Big Data • Steps of Data Science Process • Difference between Business Intelligence (BI) and Data Analysis • lifecycle of Data Science with the help of a use case • Role of Big Data in Data Science • Components of Data Science 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore Introduction to Big Data • Data Science Uses • Python for Data Science • Use of Big Data and Data Science Use case  Banking  Classification of emails –Spam/Ham  E-commerce  Retail chains – Starbucks  Predicting Election Results 4 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 5 Outcomes of the Session able to understand • What is Data Science? • Components of Data Science • How Data Science is different • Why Python for Data from Business Intelligence (BI) and Data Analysis? • The lifecycle of Data Science • Concepts of Modelling /Statistics • Role of Big Data in Data Science Science? • How leading companies are utilizing the power of Data Science with Python? • Use of Big Data and Data Science (Examples) 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 6 Recollection • Big Data are data sets so large or so complex that traditional methods of storing, accessing, and analyzing their breakdown are too expensive. • However, there is a lot of potential value hidden in this data, so organizations are eager to harness it to drive innovation and competitive advantage. • Big Data technologies and approaches are used to drive value out of data rich environments in ways that traditional analytics tools and methods cannot. 5/21/2020 Big Data Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 7 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore What To Do With These Data? • Aggregation and Statistics Data warehousing and OLAP • Indexing, Searching, and Querying Keyword based search Pattern matching (XML/RDF) • Knowledge discovery Data Mining Statistical Modeling • Data Driven  Predictive Analytics  Deep Learning 8 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 9 Big Data & Data Science • ―… the exciting job in the next 10 years will be statisticians,‖ Hal Varian, Google Chief Economist • The U.S. will need 140,000-190,000 predictive analysts and 1.5 million managers/analysts by 2020. McKinsey Global Institute’s • New Data Science institutes being created or repurposed – NYU, Columbia, Washington, UCB. • New degree programs, courses, boot-camps  e.g., at Berkeley: Stats, I-School, CS, Astronomy  Data Science Stream at AUST  RDA-CODATA School of Research Data Science 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 10 Overview to Data Science • Today, all sectors have integrated technology into their day-to- day business. • Technology brings along with it growing data sets. Hence, processing and analyzing such data sets is becoming more and more vital. • But there is a key question that needs answering. Who would synthesize and give meaning to such scattered data existing in various forms? Enter data science. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 11 Overview to Data Science • Harvard Business Review classifies Data Scientist to be ―The exciting Job of the 21st Century. • A McKinsey report forecasts the demand for data scientists to multiply so rapidly such that it would outgrow the supply by 50\%. Surely, in the upcoming years, this would be one of the sought-after fields wherein job candidates would opt to build their career in. Did you know? • Big companies such as Amazon, L’Oreal, Viacom 18, British American Tobacco, and many others are hiring Data Scientists!) 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 12 Data- The Modern Age Oil - Example • With its dramatic ups and lows, crude oil prices may witness fluctuations; thus, taking a toll on the oil producing nations. But, data remains to be priceless as it will form the crux of decision–making in future. • At the same time, an analogy can be drawn between data and oil. Certainly, data and crude oil are precious resources. But, we cannot derive value from both unless they are processed and refined. • As miners extract crude oil, in the same way, Data engineers extract data and Data Scientists refine data. However, the supply chain of data is not as complex and cumbersome as that of oil. After the oil is extracted, it needs to be then transported to tankers. Which then is routed through pipelines to be finally stored in storehouses? With the tremendous rise of Cloud Computing services, transportation and storage of data have never been easier! 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 13 What is Data Science? • An area that Manages, Manipulates, Extracts, and Interprets knowledge from tremendous amount of data • Data science (DS) is a multidisciplinary field of study with goal to address the challenges in big data • Data science principles apply to all data – big and small 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 14 What is Data Science? Some definitions link computational, statistical, and substantive expertise. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 15 What is Data Science? • Theories and techniques from many fields and disciplines are used to investigate and analyze a large amount of data to help decision makers in many industries such as Science, Engineering, Economics, Politics, Finance, and Education • Computer Science Pattern recognition, visualization, data warehousing, High performance computing, Databases, AI • Mathematics Mathematical Modeling • Statistics Statistical and Stochastic modeling, Probability. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 16 What is Data Science? Other definitions focus more on technical skills alone. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 17 Data Science? • Data Science is nothing but a science of making sense of data. • It involves the usage of automated methods to analyze data and extract information or insights to find the unknown. • It creates data products which help in decision making. This further aids in driving business value and building confidence. 5/21/2020 Data Science? Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 18 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 19 Definition of Data Science • Data science is a field of study and practice that focused on obtaining insights from data. • Practitioners of data science use programming skills, statistics knowledge, and machine learning techniques to mine large data sets for patterns that can be used to analyze the past or even predict the future. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore What is Data Science? 20 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 21 The ―Science‖ in Data Science The term science is usually synonymous with the scientific method. The process outlined in previous image is very similar to the process characterized by the expression, scientific method. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore Scientific Method as an Ongoing Process 22 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 23 How can we solve real world problems with Data Science? • All the tools and resources will help to resolve problems. But, the main aim should be to identify the right problem and to find the right tool or model which would be used while solving the problems. • Data Science is now impacting myriad areas. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 24 1. NETFLIX Based on the movies viewed previously, Netflix uses collaborative filtering algorithms to recommend movies to its users. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 25 2. SOCIAL MEDIA Many social media sites are also using data science. Whether it is recommending you connections on LinkedIn or new products on Amazon. Whether it is personalizing your Facebook feeds or suggesting to you people to follow on Twitter. Data Science does it all! 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 26 3. OTHER ONLINE SERVICES • Many online apps such as eHarmony (a dating Site), Uber, Spotify, Stitch Fix, Hulu, Bombfell, Pandora, etc. are collecting and using real-time data to customize and improve their users‗ experience. • Personalization not only increases customer engagement and retention but also improves conversion rates; thus, positively impacting the organizational bottom-line. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 27 Data Scientists also work in other domains 4. BIOINFORMATICS Scientists are working on analyzing genome sequence. 5. ASTROPHYSICS Physicists use data science concepts while building and analyzing 100 TBs of astronomical data. (You should check this out- Interview: Kirk Borne,Data Scientist, GMU on Big Data in Astrophysics) 6. SPORTS Toronto Raptors, an NBA team, is installing cameras on basketball courts. They collect huge amounts of data on a player‗s movement and playing styles. The team then analyses game trends (based on the data collected) there by improving coaching decisions and team performance. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore Data Science Vs Analysis Vs Software Delivery 28 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 29 Component Traditional Analysis Traditional Software Delivery Data Science Tools SAS,R, Excel, SQL, in-house tools Java, source control, Linux, continuous integration, unit testing, bug reports and project management R, Java, scientific Python libraries, Excel, SQL, Hadoop, Hive, Pig, Mahout and other machine learning libraries, github for source control and issue management Analytical Methods Regressions, classifications, measuring prediction accuracy and coverage/error, sampling N/A Classification, clustering, similarity detection, recommenders, unsupervised and supervised learning, small-and large-scale computations, measuring prediction accuracy and coverage/error Team Structure Statisticians, Mathematicians, Scientists Developers, Project Managers, Systems Engineers Mathematicians, Statisticians, Scientists, Developers, Systems Engineers Time Frame Either: •Usually on-going research and discovery within a team in the organization Or: •Specific project to determine answers Regular software release cycle, continuous delivery, etc. Either: •Discovery/learning phase leading to product development Or: •On-going research and product invention/improvement 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 30 Big Data Analytics & Data Science – Is there a winner? • Broadly, Big Data Analytics can be called Data Science, but Data Science cannot be called Big Data Analytics. On the surface, they perform the same operation – i.e. mining useful information from data. So, the two fields overlap significantly and often work hand in hand. Big Data Analytics involves mining useful information from raw data. • Data Science uses Machine Learning to train the computer to learn without being explicitly programmed, to make future predictions. Machine Learning is what makes Data Science different from Analytics. • The Machine Learning Algorithms used are – Decision Tree Learning, Artificial Neural Networks, Deep Learning, Clustering, Random Forest Classifiers, Naïve Bayes, Regression and more. • The use of Machine Learning makes computers even smarter – which is one reason why Data Science is so sought after these days. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 31 Are there any stark similarities between the two fields? • There seem to be some significant differences between the two fields. Does this mean that they don‗t coincide at all? No. There are quite a few similarities – Develop useful insights from raw Data Work on Big Data Attempt to use the insights achieved to make smart business decisions Applied in similar fields, like healthcare, finance, social media and sports. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 32 Where are Big Data Analytics and Data Science used? 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 33 Where are Big Data Analytics and Data Science used? 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 34 Where are Big Data Analytics and Data Science used? 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 35 Summary • There is a major overlap between Data Science and Data Analytics and essentially, as they perform the same operation, the two fields have some concrete differences. • They are making our lives easier every day, usually without us even realizing it! • But, if data amazes you and you‗re smitten with this Big Data revolution, then Read more articles on Big Data Analytics, Data Science, Machine Learning and much more! 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 36 What skills are needed to be a Data Scientist? Drew Conway‗s Venn diagram clearly illustrates the three skill sets, viz., I. II. III. Mathematics & Statistics Programming & Database Domain Expertise required to be a data scientist. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore Drew Conway‗s Venn diagram 37 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 38 What skills are needed to be a Data Scientist? You think you don‗t possess extensive knowledge or expertise in all the areas? No problem. Data Science is said to be a team sport. Thus, a data scientist need not necessarily be strong in all the mentioned fields. So, you can still build a career in data science if you either have strong analytical or programming skills. This would serve to make you a valuable player in the data science team! 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 39 I-Mathematics & Statistics • Understanding of basic mathematics and statistics is crucial for a data scientist. • Statistical knowledge helps to interpret and to analyze the data that is collected. • Data Science becomes magical as applying brilliant mathematical concepts to the data yields unexpected insights! The basic concepts are Descriptive and Inferential Statistics, Linear Algebra, Graphing, etc. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 40 II- Programming & Databases • This is the area that separates one from being a statistician or an analyst. For any given data, one needs to write programs to query and retrieve data from databases or apply machine learning algorithms. One should have a good grasp on data science libraries and modules. • Python and R provide some prebuilt libraries. Both can be used by simply importing them into programs. Thus, this makes them good programming languages to start with. • Although Microsoft Excel is a great tool for processing data, it is only suitable when working with small or medium data sets. But when it comes it to Big Data, Python and R are much better. They also provide greater flexibility and control to the user. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 41 II- Programming & Databases • Database Systems act as a central hub to store information. • These can be SQL-based or NoSQL-based. • Relational Database includes PostgreSQL, MySQL, Oracle, etc. with Hadoop, Spark, and Mongo DB being among the others. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 42 III-Domain Expertise • This involves asking the right questions by filtering the relevant data from the entire data universe. • Data Scientists need to interpret the data by understanding its structure. They also need to know the problem(s) they are solving. • For instance, if they are solving an online advertising problem, they should understand the type of customers visiting their website, their interaction with the website and the meaning of such data. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 43 IV. Machine Learning • This is an integral part of Data Science, used to create predictive models. • Machine Learning algorithms are very powerful, thereby eliminating the need to create a new algorithm. • One should know the common ones such as dimensionality reduction, supervised and unsupervised algorithms. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 44 Types of Machine Learning algorithms • You‗ve used machine learning dozens of times in a day, without even realizing that you are doing so! (E.g. searching on Google requires machine learning) • The reason for Google Search to work so well is because the machine learning software knows how to rank pages. • The feature of auto-recognition and tagging of friends on Facebook is also due to machine learning. 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore More skills for a Data Scientist • Data Visualization and Reporting • Risk Analysis • Statistical Analysis • Effective Communication • Software Engineering Skills 45 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore More skills for a Data Scientist • Data Mining and Cleaning • Research • Big Data Platforms • Cloud Tools • Data warehousing and structures 46 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore Data Scientist- At a glance 47 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore Do Big Data Analysts and Data Scientists differ? 48 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 49 Can you become a Data Scientist or an Analyst? Are you fed up with your current job and want to shift to the hottest new profession on the block? Here are some of the requirements the fields demand – • Data Scientist:      Good Statistical and Mathematical skills Good programming skills (Python, R, Java) The ability to ask the right questions (given a data set) Knowledge in Machine Learning A fast learner • Data Analyst:  Good Analytical skills  A keen business mind  The ability to analyze the results after Analytics tools are applied to data  Knowledge in dealing with Analytics tools like Hadoop and Hive  A fast learner If you think that, you have (or can acquire) these abilities, then these fields are for you 5/21/2020 Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore 50 Want to be a Data Scientist? A piece of advice. • • • • • • Do you love data? You consider yourself an eye for identifying trends and patterns? Do you have strong foundations of statistics, code writing, and programming? Are you comfortable with handling unknown facts? Do you think you can deliver to impatient stakeholders? Do you think you can convince them of your findings? If you‗ve answered a yes to these questions, then Data Science is the right career for you! So all you need to do is the master at least one language or tool of ... Purchase answer to see full attachment