DATA ANALYTICS | UNIT-1 UNIT-1 1.1 Introduction to Data Analytics 1.L.1 Overview of Data Analytics Data analytics is the process of examining raw data to draw conclusions, discover patterns, and identify trends, ultimately to support decision-making. Data analytics is a multidisciplinary field that uses statistical and computational methods to discover meaningful pattems and insights from data, The goal is to support effective decision-making. Data analytics often uses tools and techniques like machine learning, simulations, and automated systems, These tools work with unstructured data and turn it into useful information for people. The results from data analytics help organizations understand their customers better, check how well their advertisements are working, personalize and plan their content, and design new products. In this way, data analytics helps organizations work more efficiently in the market and increase their profits. 1.1.2 Importance of Data Analytics Data analytics is crucial for businesses because it allows them to make informed decisions, improve efficiency, and gain a competitive edge. By analysing data, companies can understand customer behaviour, optimize operations, and predict future trends. This leads to better strategies, increased revenue, and reduced costs. Following are the factors which help us to understand why Data Analytics is important, i. Informed Decision Making: Data analytics provides insights into past performance, current trends, and potential future outcomes. This allows businesses to move beyond guesswork and make decisions based on concrete evidence. For example, a retail company can analyze sales data to determine which products are most popular and adjust inventory accordingly. ii, Improved Operational Efficiency: Data analytics helps in spotting delays and inefficiencies in business processes. By studying data related to workflows and operations, companies can simplify their processes and minimize waste. This results in higher productivity, lower costs, and better overall efficiency. fii, Enhanced Customer Understanding: Data analytics helps businesses understand customer behaviour, preferences, and needs. By analyzing customer data, companies can personalize marketing efforts, improve customer service, and develop more effective products and services. This can lead to increased customer satisfaction, loyalty, and retention. FY-MCA 1iv. % DATA ANALYTICS | UNIT-1 Competitive Advantage: Companies that use data analytics can outperform their competitors by making smarter decisions, improving operations, and gaining deeper insights into their customers. This enables them to adapt quickly to market changes, discover new opportunities, and innovate at a faster pace. Risk Management: Data analytics can help organizations identify and manage potential risks by analyzing data on operations, finances, and market trends. This helps businesses avoid financial losses, reputational damage, and other negative outcomes. New Opportunities: By analyzing data on emerging trends and technologies, companies can identify new opportunities for growth and innovation. This can lead to the development of new products and services, new markets, and new business models, 1.1.3 Scope of Data Analytics The data analytics future scope is vast and promising. As technologies evolve and industries recognize the power of data, analytics will be at the forefront, driving innovations, efficiencies, and transformations across the board. Data analytics scope holds exciting potential, with some key trends shaping its future: iv. Growing Importance of AI and Machine Learning: Data analytics and AI are becoming increasingly intertwined. Machine learning algorithms, a subset of AI, can process vast amounts of data and learn from it, making predictions and automations more accurate over time. Expanding Impact in Various Sectors: While industries like finance, healthcare, and retail have been early adopters of data analytics, its footprint is expanding. Sectors like agriculture, energy, and even arts are starting to realize the potential of data-driven insights, Real-time Analytics: The demand for real-time data and insights is growing. Whether it's businesses wanting to make on-the-spot decisions or health professionals monitoring patients in real-time, the future will see a surge in real-time data analytics applications, Enhanced Personalization: With better data analysis capabilities, businesses can offer hyper-personalized experiences, be it in e-commerce product recommendations, content streaming, or even personalized healthcare plans. 1.14 Applications of Data Analytics FY-MCADATA ANALYTICS | UNIT-1 Data analytics has a wide range of applications across various industries, enabling organizations to make informed decisions, optimize processes, and improve overall performance. Here are some sectors where data analytics is used 1. Healthea manage hospital resources, and predict disease outbreaks, The future scope of data science in healthcare is vast, as it enhances diagnosis accuracy and personalises treatments, driving better health management and innovation in medical research, 2. Finance: Data analytics is used for fraud detection, risk management, and investment strategies in finance. The future of data analysts in finance is promising, as they play a critical role in optimising financial operations, improving customer services, and ensuring regulatory compliance. 3. Retail: Data analytics in retail involves understanding customer behaviour, managing inventory, and personalising marketing efforts. The scope of data analytics in retail will be extensive in the future as businesses use data to enhance the shopping experience, increase sales, and streamline supply chains. 4, Marketing: Marketing relies heavily on data analytics to create targeted campaigns, measure effectiveness, and understand market trends. The scope of data science in future marketing efforts includes more sophisticated customer segmentation and predictive analytics, leading to more efficient and impactful marketing strategies. 5, Manufacturing: In manufacturing, data analytics is used for predictive maintenance, quality control, and optimising production processes. The scope of data analytics in this sector is significant, as it helps improve product quality, reduce downtime, and increase operational efficiency. fons: Telecommunications companies use data analytics to manage networks, predict outages, and personalise customer services. The future scope of data analysts in this reliability and providing tailored services to meet customer demands, 7. Education: Data analytics in education helps track student performance, personalise learning experiences, and optimise administrative operations. In India, the future scope of data analytics in education is growing as it supports more effective teaching methods and improved educational outcomes. 8. Transportation: Data analytics in transportation is used for route optimisation, traffic management, and improving safety measures. The future scope of data analytics in this sector includes more competent logistics, better public transport systems, and enhanced travel experiences, Data analytics in healthcare help improve patient outcomes, 6, Telecommuni sector will involve enhancing network FY-MCA1.2 Types of Data Anal DATA ANALYTICS | UNIT-1 9. Energy: The energy sector utilises data analytics for grid management, energy consumption forecasting, and renewable energy integration, The future scope of data analytics for energy involves more efficient resource utilisation and supporting the transition to sustainable energy sources, 10, Sports: Data analytics improves player performance, develops game strategies, and enhances fan engagement in sports. What is the scope of data analytics in sports? It includes advancements in training programs, injury prevention, and personalised fan experiences, driving the evolution of the sports industry. Data analytics can be categorized into four main types: 124 |. Deserip Diagnostic analytics Predictive Prescriptive Descriptive Analytics Descriptive analytics is the foundational level of data analytics. It helps businesses understand what happened in the past by summarizing historical data into meaningful patterns. This type of analytics is often used to create reports, charts, and dashboards that show trends, spikes, or declines in metrics like sales, website visits, or customer engagement. It’s widely used because it provides a clear view of what has already occurred, making it easier to assess past performance, Steps Involved in Descriptive Analytics: Step 1 Step 2 Step 3 Step 4 Step 5 Define Goals: Start by identifying the questions you want answered. For example, “What products were most popular?” Collect Data: Gather relevant data from internal databases (e.g., sales records) or external sources (¢.g., market reports, customer surveys). Data Cleaning: Ensure that the data is accurate, consistent, and free of errors, as clean data is essential for reliable insights. Visualize the Data: Use tools like bar charts, pie charts, and line graphs to visualize the trends. Visualization tools, such as Power BI, Google Data Studio, or Tableau, make it easy to present data in a clear and understandable way. Interpret Insights: Once the data is visualized, analyze the trends and pattems to draw meaningful conclusions. For instance, you may notice FY-MCA 41.2.2 happened. It helps businesses dig into their data to find the real reasons behind certai DATA ANALYTICS | UNIT-1 that sales peak during certain months, indicating the effectiveness of a promotional campaign. Common Uses of Descriptive Analytics: ‘© Sales Reports: Summarizing daily, weekly, or monthly sales figures. Customer Segmentation: Analyzing customer demographics, such as age groups or regions. ‘© Website Traffic: Monitoring the number of visitors and their behavior on a company’s website Example of Descriptive Analytics: An e-commerce platform uses descriptive analytics to track user engagement, By analyzing website visits, time spent on pages, and product views, they can determine which sections of their website are performing well and which need improvement. This helps them enhance user experience and increase sales conversions, Diagnostic Analytics Diagnostic analytics doesn’t just tell you what happened — it explains why it trends or unusual events, For example, instead of only knowing that sales dropped in one region, diagnostic analytics can help figure out why that happened. Or, if website traffic suddenly increased, it can reveal what caused it. To do this, it uses methods like looking deeper into the data, finding relationships between different factors, and analyzing root causes, This way, companies can understand the reasons behind the numbers and take the tight actions to fix problems or make the most of new opportunities. Steps Involved in Diagnostic Analytics: Step 1 Identify the Problem: Define the specific issue or anomaly you want to investigate, such as a decrease in sales or an increase in customer complaints Step 2 Collect and Examine Data: Gather the data related to the problem, including internal data (e.g., transaction records) and external data (¢.g., competitor performance or industry benchmarks), Step3 Perform Root Cause Analysis: Use diagnostic techniques like drill-down analysis to explore the underlying causes. For example, if FY-MCA 51.23 DATA ANALYTICS | UNIT-1 sales dropped in a region, you might drill down into customer feedback to identify issues with product quality. Step 4 Use Statistical Models: Leverage statistical techniques to identify correlations or anomalies in your data. For example, you might find that product returns spike after a marketing campaign, suggesting issues with the promoted product. Common Uses: '* Sales Decline: Investigating a drop in sales and identifying potential causes, such as poor customer service, a competitor’s promotion, or seasonal factors. © Customer Churn Analysis: Identifying why customers are leaving by analyzing factors such as product dissatisfaction, pricing, or competitor influence, © Operational Failur to determine whether it’s caused by equipment failure, operator error, ot supply chain issues. Analyzing machine downtime in a manufacturing plant Example: A telecommunications company notices a sudden spike in customer cancellations in one region. By applying diagnostic analytics, they discover that the issue stems from a service outage that lasted several hours, which led to customer dissatisfaction. The company can now address the technical issue and improve their communication with affected customers. Predictive Analytics Predictive analytics focuses on forecasting future outcomes based on historical data, It answers the question: What is likely to happen? By analyzing past pattems and tren , predictive analytics helps businesses make educated guesses about the future. This type of analytics uses techniques such as statistical models, machine learning algorithms, and data mining to predict future events. Regression Analysis, Time Series Forecasting, and Machine Learning Algorithms are the techniques used in Predi ive Analytics: Steps involved in Predictive Analytics: Step 1 Identify the Goal: Define what you want to predict, such as future sales or customer churn, Step 2 Collect and Clean Data: Gather relevant historical data and ensure it’s cleaned and ready for analysis. Poor data quality can significantly affect predictions, FY-MCA 6Step 3 Step 4 DATA ANALYTICS | UNIT-1 Build Predictive Models: Use statistical tools or machine learning models to forecast outcomes. Tools like Python (with libraries such as Scikit-learn) or R can be used for this Evaluate Model Accuracy: Test your predictive models to ensure accuracy. You can fine-tune them by adjusting parameters or using different datasets. Common Uses: ‘© Sales Forecasting: Predicting future sales based on historical sales data, seasonal trends, and market conditions, © Customer Behaviour Prediction: Using past purchase data to predict customer buying habits and recommend relevant products. © Risk Management: Predicting financial risks such as loan defaults or insurance claims by analysing past risk factors and customer profiles, Example: A healthcare provider uses predictive analytics to forecast the likelihood of patients developing chronic illnesses based on their medical history and lifestyle data, This allows the provider to offer carly interventions and preventative care, improving patient outcomes and reducing healthcare costs. 1.2.4 Prescriptive Analytics Prescriptive analytics goes beyond predicting what might happen — it tells you what actions to take. It answers the question: “What should we do next?” This type of analytics not only forecasts future outcomes but also gives clear recommendations to help make the best decisions. Businesses often use it when they need to choose the most effective way to reach a goal, such as cutting costs, increasing profits, or improving efficiency, Optimization Models, Simulation Models and Decision Trees are techniques used in Preseriptive Analytics. Steps involved in Prescriptive Analytics: Step 1 Step 2 FY-MCA Define Objectives: Clearly identify the goals you want to achieve, such as reducing costs, increasing sales, or improving efficiency. Collect Relevant Data: Gather historical and real-time data that can help inform decisions. For example, a supply chain manager would collect data on traffic pattems, delivery times, and fuel costs,DATA ANALYTICS | UNIT-1 Step3 Build Prescriptive Models: Use tools like optimization algorithms and simulation models to analyze different scenarios and recommend the best course of action. Step4 Implement and Monit After choosing a recommendation, implement it and monitor the results to ensure it leads to the desired ‘outcome. Common Uses of Prescriptive Analytics: © Supply Chain Optimization: Recommending the most efficient routes for shipments to reduce co ind improve delivery times. © Personalized Marketing: Suggesting targeted promotions or product recommendations based on individual customer behavior and preferences. © Inventory Management: Recommending how much inventory to hold based on demand forecasts, minimizing the risk of overstocking or stockouts, Example: ‘A delivery company could use pre riptive analytics to find the best delivery routes. By looking at traffic patterns, fuel prices, and vehicle performance, it can figure out the fastest and most cost-effective way to make deliveries, saving money and time. 1.2.5. Data Science vs Data Analytics vs Business Intelligence (BI) Data science, data analytics, and business intelligence (BI) are all related fields focused on using data to inform decisions, but they differ in scope and methodology. Data science is the broader field, encompassing the entire process from data collection to predictive modeling. Data analytics focuses on understanding past and present data to identify trends and patterns, while business intelligence uses historical data to create reports and dashboards for decision-making. The comparison between these three fields can be given as follows, Data Science Data Analytics Business Intelligence (BD) Discover new pattems, build [Analyze data to find trends, | Monitor and report business predictive and Al-driven | relationships, and causes performance using past and solutions current data Future-focused and | Problem-focused and [Past and present-focused innovation-oriented insight-oriented decision support FY-MCA 8DATA ANALYTICS | UNIT-1 “What will happen?” or “How can we make it happen? “Why did this happen?” or “What does the data tell us?” “What happened?” Structured, unstructured semi-structured, (eg, text, images, audio) Mostly structured and semi-structured Mostly structured Machine learning, AT, deep Statistical analysis, data Dashboards, reports, KPIs, PyTorch, Hadoop, Spark Tableau, Power BI learning, natural language | mining, diagnostic _—_& | scorecards processing predictive analytics Python, R, TensorFlow, [Python, R, SQL, Excel,| Power BI, Tableau, Qik, Excel Predictive models, AT systems, advanced algorithms Insights, pattems, actionable recommendations Dashboards, visualizations, performance reports it happens Predict customer churn before Identify reasons for sales drop ina region ‘Show monthly sales trends by region 1.3 Data Types Data can generally be categorized into three types based on its structure: structured, unstructured and semi-structured, Each type comes with its own characteristics, advantages, and challenges. y004 1010 4140 1008 4010 4000 yo04 4010 ee 1001) 0104 1440/1110 9110 1000 1001) 1004 0040 1010 4440 4440 1004/0104 1040 4140 1900 4000 Structured Data FY-MCADATA ANALYTICS | UNIT-1 Structured data is the easiest type of data to work with because it is highly organized and stored in a fixed format — usually in rows and columns, like a spreadsheet ‘or database table. Its clear structure makes it simple to search, query, and analyze using standard tools. This is why structured data are used to manage large amounts of information and complex operations such as research and statistical analysis. Structured Data Characteristics: = Stored in predefined schemas (a fixed blueprint for how data is organized). = Each value is placed in a specific field — for example, Name, Date, Amount. = Relationships between different data points are clearly defined, Examples: = SQL Databases — like a customer database with fields such as Name, Email, and Phone Number. = Spreadsheets — Excel files or CSV files with well-organized tables. ‘Advantages: = Works smoothly with Business Intelligence (BI) systems for reporting and dashboards. = Easy to store, query and analyse — Can be easily used by applications and systems = Are ideal for creating reports and dashboards. a FY-MCA 10DATA ANALYTICS | UNIT-1 Challenges: — Rigid structure — not suitable for storing diverse or unpredictable data types. = Requires upfront schema design, which can limit flexibility if changes are needed later. = Can be expensive to collect and process = May be limited in the amount of information they can contain. = Can be difficult to update and maintain. 1.3.2. Unstructured Data Unstructured data is the most flexible form of data, but also the most challenging to manage. It has no predefined format or schema, which makes storing it easy but analyzing it more complex. Despite the difficulty, unstructured data makes up most of the information we deal with every day — from social media content to videos and emails. m w Handorittord Customer Reviews images wane Tweets Emails Videos ‘Aucios Unstructured Data Key Characteristics — No fixed structure or schema, = Can include text, images, videos, audio, or mixed media, = Often stored in raw formats without organization. Examples: — Text documents — Word files, PDFs. = Media files - Images (JPG, PNG) and Videos (MP4, AVI). = Online content — Social media posts, emails, chat logs. Advantages: = Extremely versatile — can store any type of data. = Vital for AI applications like Natural Language Processing (NLP) and Computer Vision, FY-MCA "DATA ANALYTICS | UNIT-1 = Contain a large amount of information — Can be easily collected from a variety of sources = Can be used to identify trends and patterns. Challenges: = Requires advanced techniques (e.g, machine learning, AI) to extract meaningful insights. = Often demands large storage and high processing power. Difficult to archive, interrogate and analyse = Difficult to use with applications and systems — Not suitable for creating reports and dashboards. 1.3.3. Semi-Structured Data Semi-structured data sits between structured and unstructured data, It’s more flexible than structured data but still contains some level of organization. It doesn’t follow a sttict, fixed schema, but it uses tags, keys, or markers to give the data a recognizable structure. This makes it great for handling diverse, evolving datasets. Configuration Files ee Semistructured Data FY-MCA 2DATA ANALYTICS | UNIT-1 Characteris Partially organized — not as rigid as structured data. = Uses tags, keys, or markers to identify and organize data (e-g., JSON, XML). ‘ommonly used for data exchange between systems and in APIs. Examples: — ISON files— { "name": "John", "age": 30 } — XML files - John30 = Log files —e.g., PowerShell commands like: Advantages: = Balances structure and flexibility. = Well-suited for complex formats like APIs, logs, and configuration files can be stored, queried and analysed more efficiently than unstructured data can be used by applications and systems more easily than unstructured data = can be used to create reports and dashboards, Challenges: = Requires specialized tools for parsing (e.g., JSON parsers, XPath for XML). = Querying is more complex than with fully structured data, = more expensive to collect and process than unstructured data limited in the amount of information they can contain = difficult to update and maintain, 1.3.4 Overview of the Data Analytics Process Data analytics is a field that involves the examination, cleaning, transformation, and modeling of data to discover useful insights, draw conclusions, and support decision-making. While the specific steps can vary depending on the project and the organization, a general, iterative process is commonly followed. This process ensures that the analysis is well-defined, the data is reliable, and the results are meaningful and actionable. FY-MCA 1BStep 1 Step2 Data Collection: Once the obje Step 3 FY-MCA DATA ANALYTICS | UNIT-1 <>} Data Analytics e Life Cycle 6 Define the Objective: This is the foundational step of any data analytics Project. Before you can begin to analyse data, you must have a clear understanding of the business problem or research question you're trying to solve. This involves collaborating with stakeholders to define specific, measurable goals and key performance indicators (KPIs). For example, instead of a vague goal like "improve sales," a more specific objective would be “identify the factors that led to a 10% decrease in sales in the last quarter." This clarity will guide the entire process and prevent wasted effort is defined, the next step is to gather the relevant data, Data can come from a multitude of sources, both internal and external to an organization. = Internal Soure Databases, customer relationship management (CRM) systems, financial records, and website logs. = External Sources: Publicly available datasets, social media feeds, and third-party APIs. ‘The data collected can be either quantitative (numerical, like sales figures or ‘website clicks) or qualitative (descriptive, like customer reviews or interview transcripts). It's crucial to ensure that the data is relevant and sufficient to answer the question posed in the first step. Data Cleaning and Preparz messy, incomplete, and contains errors. This step, often the most time-consuming, is vital for ensuring the accuracy and reliability of your analysis. Key tasks in data cleaning include: ion: Raw data is almost never perfect. It's often 14Step 4 Step 5 DATA ANALYTICS | UNIT-1 = Handling missing values: Deciding whether to remove records with missing data, of to impute (fill in) the missing values using statistical ‘methods. — Removing duplicates: Identifying and eliminating redundant entries, = Correcting errors and inconsistencies: Fixing typos, standardizing formatting, and correcting illogical values (e.g., an age of 200). = Outlier detection: Identifying and deciding how to handle data points that are significantly different from the rest of the dataset, Exploratory data analysis (EDA): Once data is cleaned and prepared, exploratory data analysis (EDA) pro what the data reveals. This critical phase helps analysts: Understand the distribution and characteristics of key variables = Identify relationships and correlations between variables = Discover patterns, trends, and anomalies = Generate initial hypotheses for deeper investigation Validate assumptions about the data EDA combines visual and statistical techniques to develop a comprehensive understanding of the dataset. Simple summary statistics such as means, medians, and standard deviations provide a starting point, while data visualizations reveal patterns that numbers alone might miss, Data Analysis and Modelling: The advanced analysis and modelling phase includes selecting the appropriate analytical techniques, developing and validating (or choosing) the model, and then interpreting and evaluating the model's results in the context of the original question to be answered or the problem to be solved. The four main types of data analytics are: = Descriptive Analytics: Summarizes what has happened. Techniques include calculating averages, frequencies, and creating simple charts to describe the data's characteristies. = Diagnostic Analytics: Explores why something happened. This involves drilling down into the data, using techniques like correlation and data mining to identify the root causes of a particular outcome. = Predictive Analytics: Uses historical data to forecast what will likely happen in the future. This involves statistical modeling and machine learning techniques like regression analysis. = Prescriptive Analytics: Recommends a course of action. It goes beyond prediction to suggest the best possible outcome by analyzing s the first opportunity to understand FY-MCA 15Step 6 Step 7 DATA ANALYTICS | UNIT-1 the potential implications of different choices. Techniques include optimization and simulation. When a model is developed, its results must be interpreted in the context of the original business question. This involves: Assessing statistical significance and confidence levels Evaluating practical significance and business impact = Understanding model limitations and constraints = Identifying potential biases or ethical considerations = Considering altemative explanations for observed patterns ‘The goal is to produce accurate models that extract meaningful insights that inform decision-making. Technical performance metrics (accuracy, precision, recall) matter, but business relevance remains the ultimate criterion for success. Data Visualization and Communication: The final and crucial step is to communicate your findings in a clear, compelling, and understandable manner. This is where you transform numbers and charts into a narrative. = Data Visualization: Creating charts, graphs, dashboards, and other visual representations to make complex data easy to interpret. A well-designed visualization can highlight trends and insights that might be missed in a table of numbers, = Storytelling: Presenting your findings as a story that addresses the initial business problem. This involves explaining what you found, why it’s important, and what actions should be taken based on the insights, The goal is to bridge the gap between data and decision-making, ‘enabling stakeholders to act on your recommendations. Implementation and monitoring: The implementation phase often requires connecting analytical teams with operational units that will apply the insights in practice. The ultimate value of analytics comes from the actions they enable that produce valuable results. Implementing analytical results involves: — Translating insights into specific action plans = Integrating findings into business processes and decision-making. = Developing implementation timelines and responsibility assignments Creating procedures to track the impact of data-driven changes = Fstablishing feedback mechanisms to refine approaches based on results Implementing an actionable insight is rarely a one-time event. When insights have been implemented, ongoing monitoring helps: = Track the impact of changes made based on analytical findings FY-MCA 16DATA ANALYTICS | UNIT-1 = Identify when models or insights need to be updated due to changing conditions = Discover new questions that emerge from initial results = Refine methodologies based on observed outcomes This monitoring creates a feedback loop, potentially initiating new iterations of the data analysis life cycle. Changes will cause business questions to evolve and new data to become available. The data analytics process is a systematic approach to transforming raw data into actionable insights. A typical process includes the following steps: 1, Problem Definition: Clearly defining the business problem or question to be answered. 2, Data Collection: Gathering relevant data from various sources. 3. Data Clea to prepare the data for analysis. 4, Exploratory Data Analysis (EDA): Exploring the data to identify patterns and understand its characteristis. 5, Modeling and Analysis: Applying statistical or machine learning models to the data 6, Interpretation and Reporting: Communicating the findings and insights to stakeholders g and Preprocessing: Handling missing values, errors, and inconsistencies in an understandable format. 7, Deployment: Implementing the solution or insights to drive business value. 1.4 Data Collection Methods Data means information like facts, numbers, objects, symbols, or events gathered from different places. Organizations collect this information in many ways so they can make smart decisions. Without data, it’s hard to make the right choi and at different times. s, so they gather it from different people For example, before launching a new product, a company needs to know things like how much people want it, what customers like, and what competitors are offering. If they don’t collect this information first, the product might fail because there’s not enough demand or it doesn’t meet customer needs. Data is valuable for every organization, but it’s only useful if it’s studied and processed to get the results the organization wants, There are various methods for collecting data in data science, cach suited to different types of projects and data needs. Here are the most common methods 1, Surveys and Questionnaires FY-MCA 7DATA ANALYTICS | UNIT-1 Surveys and questionnaires are some of the most widely used tools for collecting data directly from individuals or groups. They ate especially effective for gathering qualitative information such as opinions, preferences, satisfaction levels, and personal feedback. These tools can also collect quantitative data (e.g., rating scales, multiple-choice answers), making them versatile for research and business decision-making, ‘A good survey should be carefully planned to ensure clarity, relevance, and objectivity. Questions must be straightforward and free from leading or biased wording. Depending on the target audience, surveys can be conducted in person, over the phone, via email, or using online platforms such as Google Forms, SurveyMonkey, or Typeform, When deployed effectively, surveys can provide deep insights from specific demographics ‘or customer segments. The reliability of survey results depends heavily on two factors: © Question quality — Poorly worded questions can confuse respondents and skew results, ‘© Response honesty and rate - Low participation or dishonest answers can reduce accuracy. Sampling bias is another common challenge, where the people responding are not truly representative of the overall population. 2. Web Scraping Web scraping is the automated process of extracting data from websites. It is especially useful when the information you need is not available in a structured format (such as a downloadable file or API) but is instead displayed on web pages. Popular tools for web scraping include: Beautiful Soup - for parsing HTML and XML content, Selenium — for interacting with websites, especially those requiring logins or handling JavaScript. cal Serapy — for large automated scraping projects. Scraped data can be stored, cleaned, and analyzed for various purposes such as price tracking, trend monitoring, or competitor analysis. ‘When employing web scraping techniques, users may encounter the following challenges: ‘© Legal and ethical considerations — Several websites explicitly prohibit web scraping within their terms of service. ‘* Technical challenges — Frequent changes in web page structures can disrupt or invalidate existing scraping scripts. ‘* Data quality issues — The raw data obtained through scraping often requires significant cleaning and structuring to ensure usability a FY-MCA 18DATA ANALYTICS | UNIT-1 3. APIs (Application Programming Interfaces) ‘An API isa structured communication channel that allows software applications to ‘exchange data. Instead of manually collecting information from a website, APIs provide a direct, standardized way to access data from various services and platforms Utilizing Application Programming Interfaces (APIs) enables seamless access to external data and services. Prominent examples include the Twitter API, which provides access to social media activity and engagement metrics; the Google Maps API, which facilitates the retrieval of location-based data, mapping services, and route optimization; and the OpenWeather API, which delivers both real-time and forecasted weather information. APIs return data in formats like JSON or XML, which makes integration with data analysis tools much easier than scraping, Challenges: ‘© Rate limits — Many APIs restrict how much data you can request in a given time frame. Access restrictions ~ Some require paid subscriptions or special authentication. © Data processing — Large datasets still need to be cleaned and organized before analysis. [Link] and Sensor Data The Intemet of Things (IoT) refers to physical devices connected to the internet that can collect and share data. IoT devices use sensors to measure and transmit information in real-time, making them essential for applications that require continuous ‘monitoring, The Internet of Things (IoT) generates highly valuable data with diverse applications across multiple domains. For instance, pre machine sensors to detect wear and prevent unexpected breakdowns, health monitoring leverages wearable devices to track parameters such as heart rate, physical activity, and sleep patterns, while environmental monitoring employs sensors to measure air quality, ive maintenance utilizes temperature, and water levels. Despite its potential, IoT faces several challenges, including the massive volume of data produced by connected devices that necessitates advanced storage and processing systems, critical concems regarding data security to safeguard devices from unauthorized access or hacking, and the need to ensure data accuracy through proper sensor calibration to minimize false or misleading readings, 5, Database a FY-MCA 19DATA ANALYTICS | UNIT-1 Accessing structured data from existing organizational databases is a widely adopted and efficient method of data collection, particularly when the required information is already stored and managed within an organization's systems. Such data often originates from operational databases, including customer relationship management (CRM) systems, enterprise resource planning (ERP) systems, financial and human resources databases, or from analytical databases such as data warehouses and data marts ‘These databases inherently store information in a structured format, typically using predefined schemas, tables, columns, and relationships in relational databases, or alternative structured forms such as key-value pairs, document-oriented structures, or graph models in NoSQL databases, thereby facilitating efficient querying and retrieval. Data access is generally achieved through query languages like SQL, application programming interfaces (APIs) offered by database systems, or specialized data integration tools. This approach offers several advantages, including efficiency, as the data is readily available without requiring primary collection methods such as surveys or interviews; accuracy, as organizational data often undergoes validation and cleansing; access to historical records, which supports trend analysis and longitudinal studies; and cost-effectiveness, as it reduces both time and financial resources associated with data collection, However, certain considerations must be addressed, such as ensuring data relevance to the research or analytical objectives, addressing potential quality issues like missing values or outdated information, adhering to access permissions and security protocols for sensitive data, and managing integration challenges when data is distributed across multiple systems. FY-MCA 20