Group Members
Muhammad Murtaza F2023266701
Mohsin Khan F2023266704
Abdullah Abid F2023266735
[Link]-Ul-Abideen F202326675
Course title: Datascience Technologies
Section: V-4
Mid-Term Project
Resource person: Shahbaz Qadeer
PROJECT REPORT: Comparative Analysis
of Institutional Faculty Data Extraction
Systems
Target Institutions: University of Pisa (Dipartimento di Informatica) &
Aarhus University (Department of Computer Science)
1. Executive Summary
The objective of this initiative was to develop automated systems to harvest, process, and
visualize comprehensive data regarding faculty and staff from two major European universities:
the University of Pisa and Aarhus University.
The project was executed through the development of two distinct Python-based architectures
tailored to the specific technical constraints of each institution's web infrastructure. The
University of Pisa system focused on a "deep-dive" extraction, utilizing external APIs and
hidden internal endpoints to gather rich profile data, including publications and courses.
Conversely, the Aarhus University system was designed as a "lightweight" high-performance
scraper, specifically engineered to bypass complex server-side pagination security without
relying on heavy browser automation tools like Selenium.
Both systems successfully transformed static web directories into structured datasets, generating
strategic dashboards that highlight key demographic metrics, gender distribution, and physical
occupancy across campus buildings.
2. System Architecture and Methodology
Both systems operate on sequential pipeline architectures utilizing Python 3.x as the core
programming language due to its robust ecosystem for data science.
2.1 Core Technology Stack
● Request Handling: The requests library was used in both systems to handle HTTP
communication.
● Parsing: BeautifulSoup4 (bs4) was utilized to parse HTML/XML and navigate the
Document Object Model (DOM).
● Data Management: Pandas and the csv module handled data cleaning, feature engineering,
and serialization.
● Visualization: Matplotlib and Seaborn were employed to generate publication-quality
statistical charts.
2.2 Workflow Variations
● University of Pisa Data Flow: The system follows a logic of Fetch -> Parse -> Enrich. It
identifies faculty rows and enters a "Data Enrichment Loop" for each person11. This loop
checks a gender cache, reconstructs obfuscated emails, and calls hidden internal APIs to
fetch detailed JSON fragments regarding research output .
● Aarhus University Data Flow: The system follows a logic of Initialize -> Traversal ->
Deduplication. It initializes headers to mimic a legitimate browser and enters a while loop
to handle pagination. Unique to this architecture is a deduplication step using a hash set
(seen_ids) to prevent redundant entries caused by staff appearing across multiple category
pages.
3. Technical Implementation Details & Code Logic
3.1 University of Pisa: Handling Obfuscation and Hidden APIs
The University of Pisa's directory presented specific challenges regarding data privacy and
nested information.
A. Email Reconstruction Logic
To prevent spam, the website hides emails using a cryptml class. The scraper programmatically
reconstructs the email by concatenating separate data attributes found in the HTML
Code Snippet (Python):
Python
# The script extracts parts from attributes like data-name="[Link]"
name_part = email_tag.get('data-name', '')
domain_part = email_tag.get('data-domain', '')
tld_part = email_tag.get('data-tld', '')
# Logic to combine them into a valid email string
email = f"{name_part}@{domain_part}.{tld_part}"
Explanation: This method bypasses the visual obfuscation by accessing the raw DOM attributes
directly, rendering the anti-spam measure ineffective against this scraper.
B. Deep-Dive Extraction (Hidden API)
The system identifies a "hidden" URL within the "ID Card" icon (data-url). It performs
asynchronous GET requests to this endpoint for every profile to extract lists of publications18.
Code Snippet (Python):
Python
# Identifying the hidden API endpoint in the anchor tag
api_url = link_tag.get('data-url')
# Making a separate request for detailed data
response = [Link](api_url)
[Link](0.2) # Polite delay to avoid server blocking
Explanation: This logic transforms the scraper from a simple list-fetcher into a "deep" extractor
that mines data not immediately visible on the main page.
3.2 Aarhus University: Security Tokens and Lightweight Logic
The Aarhus University scraper was designed to handle server-side security features efficiently.
A. Secure Pagination (cHash)
The website uses TYPO3 CMS with cryptographic hashes for pagination. The script dynamically
extracts the full href of the "Next" button to acquire the valid cHash token.
Code Snippet (Python):
Python
# Locating the 'Next' button to get the valid security token
candidate = [Link]('a', string=lambda t: t and "Next" in t)
if candidate:
next_link = candidate['href']
# Joining relative link with base URL
next_link = urljoin("[Link] next_link)
Explanation: Instead of guessing URL parameters (e.g., page=2), which the server rejects, this
code "clicks" the actual button programmatically to ensure the session remains valid.
B. Data Sanitization & Gender Logic
Names are formatted as "Surname, Firstname". The system utilizes string splitting
logic to isolate the token after the comma to correctly identify the first name for
gender analysis.
Code Snippet (Python):
Python
def get_gender(full_name, detector):
# Handling "Surname, Firstname" format
if "," in full_name:
# Split by comma and take the second part (Firstname)
name_part = full_name.split(",")[1].strip()
first_name = name_part.split()[0].capitalize()
return detector.get_gender(first_name)
Explanation: This ensures the gender detector analyzes "Anasuya" (First Name) rather than
"Acharya" (Surname), significantly improving demographic accuracy.
4. Data Analysis and Visualization Results
Upon successful extraction, both systems utilized visualization scripts to generate analytical
dashboards.
4.1 University of Pisa Dashboard
● Gender Distribution: The analysis revealed a significant imbalance with a Male-to-Female
ratio of approximately 3:1.
● Qualification Levels: Using role titles as proxies, the largest demographic was identified as
"Master's Holders" (PhD students), followed by "PhD Holders" (Professors/Researchers) .
● Building Occupancy: By extracting room codes (e.g., "331 O"), the system identified
"Building O" as the primary hub, housing the highest number of personnel.
4.2 Aarhus University Dashboard
● Staff Categories: The visual analytics compared the workforce size of Faculty versus PhD
Students and Administrative Staff.
● Job Titles: A horizontal bar chart highlighted the most common roles, such as "Associate
Professor" and "Postdoc".
● Building Heatmap: Similar to the Pisa analysis, this chart identified which buildings house
the most staff to assist in logistical planning.
5. Challenges and Limitations
Challenge Context Solution/Mitigation
Deliberate [Link](0.2)
API Rate Limiting
Pisa: Extracting detailed info delays were introduced to
required 100+ separate avoid firewall blocks.
requests.
Implemented dynamic link
Pagination Security
Aarhus: Server rejected extraction to scrape the valid
manual URL parameter cHash token.
guessing.
Implemented string splitting to
Data Inaccuracy
Aarhus: Names formatted as isolate the First Name token.
"Surname, Firstname"
confused the gender detector.
This feature was descoped to
AJAX Content
Aarhus: "Selected maintain the "lightweight"
Publications" are loaded via requirement and avoid
JavaScript/AJAX. Selenium overhead.
Accuracy improved by forcing
Inference Accuracy
Pisa: Gender is inferred, not country_id=IT in the API call.
self-reported.
6. Conclusion
The project successfully demonstrated the capability to automate the extraction of complex,
nested data from institutional websites. By combining standard HTML parsing with creative
techniques for handling obfuscated data (Pisa) and cryptographic security tokens (Aarhus), the
systems transformed static web directories into dynamic, analyzable datasets.
The resulting visualizations provide immediate, actionable insights into the demographic
structure and physical footprint of both the Dipartimento di Informatica at the University of Pisa
and the Department of Computer Science at Aarhus University.