Scribd
Upload
16 views7 pages

Data Analytics Course Project Guide

The Data Analytics and Visualization Course Project involves a substantial project where students can conduct either an exploratory analysis of a dataset or implement a scalable data analytics solution. Projects will be evaluated based on significance, technical quality, and organization, with deliverables including a midterm report, presentations, and a final report. Students are encouraged to use various online datasets and software tools for data analysis and visualization.

Uploaded by

zindazed2
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
16 views7 pages

Data Analytics Course Project Guide

The Data Analytics and Visualization Course Project involves a substantial project where students can conduct either an exploratory analysis of a dataset or implement a scalable data analytics solution. Projects will be evaluated based on significance, technical quality, and organization, with deliverables including a midterm report, presentations, and a final report. Students are encouraged to use various online datasets and software tools for data analysis and visualization.

Uploaded by

zindazed2
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Data Analytics and Visualization Course Project

Project Description
The most important learning component of the class consists of a substantial course project. The
project will offer you an opportunity to develop a quantitative and qualitative intuition of data
analysis and visualization methods and algorithms, and to obtain practical experience working
with software and tools for large-scale data analysis and visualization. This can prepare you for
applying state-of-the-art approaches in real-world settings and data. There can be two types of
projects that you can engage in as part of this course.
• Type I: An in-depth exploratory analysis of an interesting dataset that provides insights
of the data that were otherwise hidden or non-obvious.
• Type II: An efficient implementation of a data analytics and visualization solution that
can scale to massive datasets.
Ideally, projects will be a combination of the three types of projects outlined above. All projects
should contain some experimentation on real data. Projects will be evaluated based on:
• Significance/Novelty. Is the analysis “real” and “interesting”, or just a “toy” analysis?
How original, important and well defined are the questions posed? How novel is the
approach? Is the analysis likely to be useful and/or have impact? Are there any
novel/interesting applications of analysis?
• Technical Quality. Is the data analysis approach and methods appropriate and well
described? Are sufficient details provided? Is the technical material correct? Are the data
analytics methods and algorithms reproducible? Are the data visualization methods and
tools creative and interesting? Is the interpretation (discussion and conclusion) well
balanced and supported by the data?
• Organization: Is the report well organized? Is the write-up clear and the language
adequate? Are results presented in the most appropriate manner? Are figures and tables
used appropriately?
Project Deliverables
The table below presents the breakdown of the project deliverables, weights and due dates.

Project Deliverables Weight Due Date


Midterm progress report (~5 pages) 30% Wed, March 18
Midterm in-class presentation 10% Thursday, March 20
Final report (~7 pages) 40% Thursday, April 30
Final in-class presentation 20% Thursday, 17 April

Datasets

Online resources of data


• Kaggle Datasets
• GapMinder Datasets
• Big Data and AI Datasets

1
Online resources of network data
• Stanford Network Analysis Project @ Stanford
• Social and Information Network Analysis @ Stanford
• The Koblenz Network Collection
• The Twitter Project Page at MPI-SWS
• Network Data by Mark Newman
• ICWSM datasets

SOFTWARE TOOLS AND LIBRARIES


Data cleansing/wrangling
• OpenRefine (formerly Google Refine) Standalone open source desktop application for data
cleanup and transformation to other formats
• Google Cloud DataPrep Data preparation and data cleansing
Graph/network analysis
• SNAP Library for working with massive network datasets (C++, Python)
• NetworkX Library for studying graphs and networks (Python)
• JUNG Library for modeling, analysis, and visualization of graphs (Java)
• Metis Family of programs for partitioning graphs
Graph/network exploration and visualization
• Pajek Program for large network analysis and visualization
• Gephi Program for graph visualization and exploration

Data Visualization
• The data visualisation catalogue Reference library for different types of data visualisations
• Highcharts Libraries for aesthetically pleasing standard charts
• Google Charts Rich gallery of interactive charts and data tools by Google.
• D3 Software for interactive visualizations
• Information is Beautiful
• Public Tableau Galleries exploratory data analysis
• PyGWalker - Turn your pandas dataframe into a Tableau-style User Interface for visual
analysis - [Link]
• Plotly Open Source Graphing Library for Python - [Link]

2
Deliverable 1: Mid-term Report (10%)
The project midterm progress report should represent a first (incomplete) draft of your final
report. The expectation is that almost 50% of the work has been completed. The report should
build on one or more large-scale open dataset(s) and/or APIs that are good candidates for
analysis and visualization. The idea is to survey the data sets freely available online and identify
what are their strengths and weaknesses for the needs of the analysis. The report should then
focus on what are some promising intuitions or questions that can be answered through rigorous
analysis of the data. Emphasis should be given on how the solution scales or what methods can
be used to alleviate the challenges of big data analytics

The outline of the report should be structured as follows:


The header of the project should include the course title and an indication that this is a project
proposal, the title of the project, your name(s) and contact information. The content of the project
should have the following parts:
• Introduction/Motivation: What is the project about? What is the data domain? What is
the goal of your project? What is the motivation for rigorous data analytics in this
domain? What are the questions you want to answer? Why is analysis important? Are
you testing any specific hypothesis? What are a few potential applications?
• Problem Definition: Is there any algorithmic component in your project (e.g., a
prediction model, classification model, clustering model, network model)? If yes,
introduce notation, provide formal definitions as needed, define any constraints or
restrictions, define what you try to optimize (e.g., maximize or minimize an optimization
function, or an accuracy/error function). Describe the problem in a formal way. Describe
the hardness of the problem in a formal way.
• Related Work: Position the problem among the body of existing research or technical
studies. How does your project relate to previous work? How is your project replicating/
different/ complementary to previous work? References to papers or other studies you
cite should be explicit.
• Methodology: What are the data analytics methods you will employ? What are the steps
you need to perform? What type of data analysis will you perform? How is this type of
analysis adequate for the data, problem and questions posed? Describe your overall data
analytics architecture. Describe the data collection/ingestion process, data storage
process, data analysis, data serving, and data visualization. Are there any algorithmic
components in your project? What is your approach to solve the problem? Provide any
mathematical background necessary for the methods. Describe any algorithms or
variations of the methods. Describe limitations or difficulties with your approach.
Formally describe any important algorithms used from literature. Try to be as specific as
possible.
• Evaluation/Results: How will you evaluate your work? What experiments do you plan
to do? What datasets have been used? Provide summary statistics of your dataset. Show
representative results of your analysis, discuss important findings, discuss implications
of your analysis to applications.
• Conclusions: What are the conclusions of your work? Are there any highlights? Is there
a need to discuss or further interpret the results? What are some ideas for future work?

3
• References: The proposal should include the full reference of the papers that you want to
base your project ideas on, your approach to solve the problems, and the tools and
datasets that you have employed. Full citation is required. References should be specific
and found inside the text, as appropriate.
Parts of the above outline will (probably) be filled at a later stage, especially the ones that
relate to the details of the methods, the evaluation and the conclusions. At this phase, you
should try to fill in as many parts of the outline as possible so that it is clear what you plan to
do for the final version.

The suggested length of the project proposal is 5 pages, and it must be in PDF format. All reports
should be formatted according to the ACM SIG conference proceedings template in LaTex and
prepared using Overleaf, a free collaborative authoring tool. The template can be accessed here:

[Link]
sigproceedingstemplate/bmvfhcdnxfty
How to Submit?
Submit your mid term progress report on Muele.

Deliverable 2: Midterm In-class Presentation (10%)


The midterm in-class presentation should be seen as your opportunity to present your work in
class, challenge your ideas, get feedback from peers and discuss methods and algorithms. The
expectation is that almost 50% of the work has been completed and some interesting analysis
results are available to share with an audience. At this stage you should be able to tell a story
about your project, even if parts of it have not yet been completely implemented or solved. The
title of the presentation should include the course title and an indication that this is the project’s
midterm in-class presentation, the title of the project, your name and contact information. The
following are some guideline, tips and advice for preparing your presentation.
• You (or your group) will have 10 minutes to present your work in the classroom.
Another 5 minutes will be allocated for questions and discussion.
• You should prepare a set of ~10 slides, given that a slide should take a minute to talk
about on average.
• Presentations should be organized into thematic units. A typical outline includes:
• Motivation of the project, its importance and potential applications.
• Definition of problem (if any), including input, constraints, desirable output and
hardness
• Main project approach ideas, data, methods and the type of analysis proposed
• Highlights of the results (visual, experimental, theoretical)
• Interesting variations and limitations of the analysis
• Concluding remarks
• The talk should be self-sufficient, meaning that you should not make any assumption
about prior knowledge of the audience or previous well-known results. All concepts
should be introduced and appropriate notation should be used consistently throughout
the presentation.
• Focus on the essential parts of the project and avoid too-many technical details. The goal
is to give a summary of the project and convey the contribution of your work to other

4
people. At the same time, you should make sure that important content is adequately
covered.
• Prepare the slides carefully. Text should be easily readable and slides should not be
overloaded with content. Avoid full text sentences and use of math symbols, unless
necessary.
• Practice the talk several times, and time yourself to make sure you are within the time
bounds.
More Advice & Tips
Some interesting advice on how to give a bad talk by David A. Patterson (UC Berkeley):
[Link]

Some design tips for beautiful presentations:


[Link]
How to Submit?
Submit your mid-term class presentation on Muele.

Deliverable 3: Project Final Report (40%)


The project final report should represent all the completed work. The expectation is that most of
the work has been completed and any major results are available. At this stage, you should be
able to provide a complete description of the project, even if a few minor parts have not yet been
implemented or solved. The header of the report should include the course title and an indication
that this is the project’s final report, the title of the project, your name and contact information.
The report should be structured as follows:
• Abstract: The abstract (limited to 150-200 words) should be a comprehensive but concise
description of your project that aims to attract potential readers. It should briefly discuss
the motivation, problem of interest, approach to solve it and main results of your work.
• Introduction/Motivation: What is the project about? What is the data domain? What is
the goal of your project? What is the motivation for rigorous data analytics in this
domain? What are the questions you want to answer? Why the analysis is important? Are
you testing any specific hypothesis? What are a few potential applications?
• Problem Definition: Is there any algorithmic component in your project (e.g., a
prediction model, classification model, clustering model, network model)? If yes,
introduce notation, provide formal definitions as needed, define any constraints or
restrictions, define what you try to optimize (e.g., maximize or minimize an optimization
function, or an accuracy/error function). Describe the problem in a formal way. Describe
the hardness of the problem in a formal way.
• Related Work: Position the problem among the body of existing research or technical
studies. How does your project relate to previous work? How is your project replicating/
different/ complementary to previous work? References to papers or other studies you
cite should be explicit.
• Methodology: What are the data analytics methods you have employed? What are the
steps you need to perform? What type of data analysis you have performed? How this
type of analysis is adequate for the data, problem and questions posed? Describe your
overall data analytics architecture. Describe the data collection/ingestion process, data
storage process, data analysis, data serving, and data visualization. Are there any

5
algorithmic components in your project? What is your approach to solve the problem?
Provide any mathematical background necessary for the methods. Describe any
algorithms or variations of the methods. Describe limitations or difficulties with your
approach. Formally describe any important algorithms used from the literature. Try to
be as specific as possible.
• Evaluation/Results: How did you evaluate your work? What experiments did you
perform? What datasets have been used? Provide summary statistics of your dataset.
How your evaluation provides support (or not) of your methods. Show results of your
analysis, discuss important findings, discuss implications of your analysis to
applications.
• Conclusions: What are the conclusions of your work? Are there any highlights? Is there
need to discuss or further interpret the results? What are some ideas for future work?
• References: The final report should include the full reference of the papers that you have
based your project ideas on, your approach to solve the problems, and the tools and
datasets that you have employed. Full citation is required. References should be specific
and found inside the text, as appropriate.
At this phase, you should try to fill in as many parts of the report as possible so that it is clear
what your term work has been.
Formatting and Style
The suggested length of the final report is 7-8 pages and it must be in PDF format. All reports
should be formatted according to the ACM conference proceedings template in LaTex.

Final Report Evaluation


The final report will be evaluated based on the following mark breakdown.

Final Report Component Weight Due Date


Introduction/Problem 20% Clear motivation that encourages the reader to
Definition read on; clear notation and problem/analysis
definition (if any) including input and
desirable output

Related Work 10% Important references are not


missing; explicit citations
Model/Methodology/Algorithms 30% Clear and well written so that we can fully
understand what
you did
Evaluation/Results 30% Comprehensive evaluation
plan; clear and conclusive set of experiments

Style and Language 10% Overall organization, language,


and style

6
How to Submit?
Submit your project final report on Muele.

Deliverable 4: Final In-class Presentation (20%)


The final in-class presentation should be seen as your opportunity to present your hard work in
class, your ideas, your methods and algorithms, your results, and discuss further implications
for future work or research. The expectation is that most of the work has been completed and
any major results are available to share with an audience. At this stage you should be able to tell
a story about your project, even if parts of it have not been completely implemented or solved.
The title of the presentation should include the course title and an indication that this is the
project’s final in-class presentation, the title of the project, your name and contact information.
The following are tips and advice for preparing your presentation.
• You (or your group) will have 15 minutes to present your work in the classroom.
Another minute will be allocated for question answering.
• You should prepare a set of ~15 slides, given that a slide should take a minute to talk
about on average.
• Presentations should be organized into thematic units. A typical outline includes:
• Motivation of the project, its importance and potential applications.
• Definition of the problem (if any), including input, constraints, output and
hardness
• Main project approach ideas, data, methods and the type of analysis proposed
• Highlights of the results (visual, experimental, theoretical)
• Interesting variations and limitations of the analysis
• Concluding remarks
• The talk should be self-sufficient, meaning that you should not make any assumption
about prior knowledge of the audience or previous well-known results. All concepts
should be introduced and appropriate notation should be used consistently throughout
the presentation.
• Focus on the essential parts of the project and avoid too-many technical details. The goal
is to give a summary of the project and convey the contribution of your work to other
people. At the same time, you should make sure that important content is adequately
covered.
• Prepare the slides carefully. Text should be easily readable and slides should not be
overloaded with content. Avoid full text sentences and use of math symbols, unless
necessary.
• Practice the talk several times, and time yourself to make sure you are within the time
bounds.

How to Submit?
Submit your final presentation slides on Muele.

Common questions

Powered by AI

Large-scale data analytics projects face several challenges including data volume, variety, and velocity, which can overwhelm traditional data processing tools and infrastructure . To address these, scalable storage solutions like distributed databases and cloud services are essential . Data processing frameworks like Hadoop or Spark can help in handling large datasets efficiently by distributing computations across multiple nodes . Data cleansing and normalization become critical to ensure consistency and accuracy across diverse data sources, requiring robust ETL processes . Furthermore, implementing predictive models may demand significant computational resources and optimized algorithms to maintain feasibility and performance . Strategic planning, adopting a modular analytics architecture, and leveraging cloud-based solutions can mitigate these challenges, ensuring that the analytics project remains scalable and manageable .

The ACM SIG conference proceedings template provides guidelines that ensure consistency and professionalism in the structure and presentation of academic reports. By defining sections such as Abstract, Introduction, Methodology, and Results, it dictates a framework that helps authors organize their content logically and enables readers to follow the narrative more efficiently . The formatting requirements encourage clarity and adherence to academic standards, which is particularly important for peer review and ensuring the report is accessible and comprehensible to a wider audience . Additionally, following such templates allows the report to fit seamlessly into conference publications or journals, facilitating academic discourse and dissemination .

Incorporating references into a project report is fundamental to academic integrity and credibility, as it acknowledges the work of others and situates the project within existing research . References should be specific and cited contextually within the text to support claims, provide evidence, and acknowledge the derivation of ideas . Proper citation formats as guided by the ACM template should be followed consistently throughout the report . This not only strengthens the report's authority by demonstrating a thorough engagement with relevant literature but also allows readers to trace the source material for further investigation or verification, thereby upholding intellectual honesty and contributing to scholarly dialogue .

Clearly defining constraints and desirable outputs in a data analysis project's problem definition is crucial to narrowing the scope and guiding the methodological approach effectively . Constraints help identify the boundaries within which the analysis must operate, impacting data collection, processing techniques, and algorithm selection . Articulating desirable outputs aids in setting realistic objectives and establishing metrics for success, allowing for targeted analysis and accurate evaluation of results . This clarity ensures that the project remains focused and aligned with stakeholder goals while providing a foundation for comparisons with existing approaches in the literature . This focus is essential for efficient resource utilization and maximizing the analytical outcomes' relevancy and impact .

Incorporating algorithmic components like prediction or classification models significantly influences a project's data analytics methodology by requiring a systematic approach to model building and evaluation . This includes preprocessing the data to ensure it is suitable for modeling, selecting relevant features, and utilizing algorithms that align with the underlying data characteristics and project objectives . The choice of model impacts the infrastructure, as more complex models may require specialized computational resources or optimizations for efficiency and accuracy . Additionally, it necessitates a rigorous validation framework to assess model performance, including cross-validation strategies and performance metric evaluations such as accuracy, precision, or recall . These algorithmic components often define the core innovative value of the analytics solution, distinguishing it from more straightforward descriptive analyses .

Discussing the results of data analysis with a focus on practical applications involves highlighting key findings and interpreting them in contexts relevant to stakeholders or industry practitioners . Presenting implications requires tying the results back to the original problem statement, demonstrating how the insights address the initial hypotheses or business objectives . This involves detailing the actionable outcomes such as process improvements, strategic recommendations, or policy formulations . Moreover, discussing limitations and potential biases enriches the conversation by setting realistic expectations and proposing areas for future research or application . Practical examples and case studies can further elucidate the applicability of findings, offering concrete scenarios where outcomes might bring tangible benefits or innovations .

The abstract of a project report serves as a concise summary that aims to engage potential readers by providing an overview of the project's motivation, key problems, methodological approach, and main findings in a limited word count . It should clearly articulate the project's significance and potential impact to provoke the reader's interest and encourage further exploration of the full report . By including these elements, the abstract acts as an initial hook that facilitates readers' understanding of the project's scope and value, aiding them in determining the relevance and importance of the report to their own research or interests .

To ensure the scalability of a network analysis project with tools like Gephi or Metis, it is crucial to adopt a data management strategy that accommodates large datasets without compromising performance . This involves partitioning the data to keep it within manageable limits, which Metis excels in by efficiently dividing graphs into smaller, less complex components . Gephi's real-time visualization capabilities can be leveraged for exploratory analyses on reduced-size datasets to minimize performance overhead . Implementing parallel processing methods to distribute load and using memory-efficient data structures also contribute to scalability . Moreover, performing thorough benchmarking and stress testing on the tool's handling of large-scale data is essential to identify bottlenecks and optimize resource allocation, ensuring the architecture can support anticipated data growth .

NetworkX, a Python library, provides an intuitive interface for network analysis and is highly suitable for quick prototyping and algorithmic exploration due to Python's versatility . In contrast, JUNG, designed for Java, is often preferred for applications requiring integration with Java systems and offers broad algorithm support and visualization . On the other hand, Pajek is specialized for handling large-scale networks due to its ability to deal with networks over a million vertices but lacks the modern visualization features present in NetworkX and JUNG . While NetworkX is excellent for moderate-scale networks and educational purposes, it is not as efficient as Pajek for very large networks, given that NetworkX operates purely in-memory and can face performance issues with scalability .

Highcharts is known for its aesthetically pleasing standard charts and is often used when quick and appealing results are needed with minimal overhead . Google Charts offers a rich set of interactive features, making it ideal for web-based data presentations that require user engagement . D3, however, provides a more extensive and customizable framework for building bespoke, dynamic data visualizations, although it demands a deeper level of coding expertise and understanding of web technologies . The choice among these tools can affect the ease of implementation, level of interactivity, and the flexibility of the analysis, ultimately impacting the insights that can be derived from the visualization process and the audience's ability to interact with and interpret the data .

You might also like