Scribd
Upload
2 views8 pages

Data Analysis Assignment

The document outlines a system design for a Community-Verified Directory aimed at collecting and verifying listings from various unstructured sources. It details a four-layer architecture for data collection, cleaning, verification, and publication, along with tools and processes for maintaining data quality and auditability. Additionally, it includes guidelines for verification, handling red flags, duplicate prevention, and scalability to ensure a reliable and efficient directory system.

Uploaded by

thakareee
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
2 views8 pages

Data Analysis Assignment

The document outlines a system design for a Community-Verified Directory aimed at collecting and verifying listings from various unstructured sources. It details a four-layer architecture for data collection, cleaning, verification, and publication, along with tools and processes for maintaining data quality and auditability. Additionally, it includes guidelines for verification, handling red flags, duplicate prevention, and scalability to ensure a reliable and efficient directory system.

Uploaded by

thakareee
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

NAME- Rupesh Suresh Thakare

DATE-16-02-26

Data Analyst Assignment


Community-Verified Directory – System Design & Verification Framework

Q1) Tools + System Design for Community-Verified Directory

Objective

The goal is to design a system that can collect listings from messy, unstructured sources
(intern research, scraping, WhatsApp, community submissions, etc.) and convert them into a
clean, verified, and publish-ready directory.

The system should:

 Scale across cities

 Be auditable

 Reduce founder dependency

 Maintain data quality

1. Overall System Architecture

I would approach this using a 4-layer structure so that raw data and verified data are clearly
separated.

Layer 1 – Raw Input Collection (Messy Layer)

This is where all incoming data first lands.

Sources:

 Google Maps scraping

 Instagram bio scraping

 WhatsApp submissions

 Google Forms
 Intern research sheets

 Pawzz network inputs

Tools Used:

 Google Forms (structured submissions)

 WhatsApp → Zapier/Make → Google Sheets automation

 Python scraping scripts (BeautifulSoup / Selenium)

 Centralized “Raw Intake” Google Sheet

Reasoning:
Google Sheets is simple for interns. Python handles scraping. Automations reduce repetitive
manual work.
This layer ensures no data is lost.

Layer 2 – Standardization & Cleaning Layer

Once collected, data moves to a cleaning stage before verification.

Tool Stack:

 Google Sheets (initial staging)

 Python scripts (cleaning + automation)

 SQL Database (PostgreSQL/MySQL) for scalable storage

Cleaning Tasks:

 Standardize phone numbers (+91 format)

 Normalize addresses

 Remove emojis/special characters

 Standardize category labels

 Split address into Area, City, State, Pincode

Core Data Fields


Field Required?

Business Name Yes

Category Yes

Phone Yes

City Yes

Address Yes

Source Yes

Submission Date Yes

Proof Link Yes

Status Yes

Verified By No

Verified Date No

This ensures structured storage and easy filtering.

2. Proof Standards Per Category

Different categories require different proof levels.

Vet Clinic

 Clinic board photo

 Google Maps listing or website

 Phone confirmation

 License number (if available)

NGO / Shelter

 Registration proof (if applicable)

 Active social media page

 At least 3 recent posts

 Contact number verified

This reduces fake or inactive listings.


3. Workflow Status Pipeline

Submitted → Needs Cleaning → Needs Verification → Verified → Published → Rejected

Definitions:

 Submitted: Raw entry

 Needs Cleaning: Missing/inconsistent data

 Needs Verification: Ready for validation

 Verified: Proof confirmed

 Published: Live listing

 Rejected: Fake or unverifiable

4. Dashboard & KPIs

Tool: Power BI / Google Data Studio

Throughput Metrics

 Listings submitted per week

 Listings verified per week

 Average verification time

Quality Metrics

 Rejection rate

 Duplicate rate

 % incomplete submissions

Backlog Metrics

 Listings pending > 7 days

 Verification queue size

Intern Performance

 Listings processed per intern

 Error rate

This ensures visibility and reduces founder dependency.


5. Handling Unstructured Inputs

Many submissions will come via WhatsApp or screenshots.

Approach:

Create a structured WhatsApp template:

Name:
Category:
Phone:
Address:
Proof Link:

Use:

 Zapier for parsing

 Python regex for phone extraction

 OCR for screenshots

 Basic NLP cleaning

 Auto-flag incomplete entries

6. Auditability System

Maintain a Change Log table:

| Listing ID | Field Changed | Old Value | New Value | Changed By | Date |

Rules:

 No permanent deletion

 Mark inactive instead

 Log all updates

This ensures accountability and transparency.

7. Automation Strategy

 Form → Sheet automation


 Slack alerts for verification

 Python cron jobs for duplicate detection

 Automated missing data checks

Goal: Reduce manual intervention.

8. Scalability Plan

Phase 1: Google Sheets + Python


Phase 2: SQL Database + Dashboard
Phase 3: Web Admin Panel + Role-Based Access

Final Impact

This system:

 Reduces founder dependency

 Enables structured growth

 Prevents duplicates

 Maintains audit trail

 Ensures quality control

 Supports analytics-driven decisions

Q2) Red Flags + Verification + Duplicate Prevention

1. Common Red Flags

 No digital presence

 Phone unreachable

 Same phone across multiple businesses

 Different names at same address

 Fake or inactive social accounts

 Repeated submissions

 Vague address
2. Vet Clinic Verification Checklist

Step 1: Basic Check

 Phone formatted correctly

 Address complete

Step 2: Online Presence

 Google Maps listing exists

 Reviews present

 Phone consistent across platforms

Step 3: Direct Confirmation

 Call clinic

 Confirm services and timings

Step 4: Documentation

 Store screenshot

 Log call notes

Only then mark as Verified.

3. Handling Conflicting Information

If two phone numbers are found:

1. Cross-check Google Maps

2. Check Instagram bio

3. Call both

4. Confirm primary contact

If unresolved → Mark as “Under Confirmation”

Never overwrite data without logging.

4. Duplicate Prevention
Level 1: Exact phone match
Level 2: Fuzzy name match + pincode similarity
Level 3: Manual review queue

5. Audit Trail

| Listing ID | Action | Done By | Date | Proof |

No permanent deletions. Mark inactive instead.

6. Escalation Rules

Reject if:

 Fake address

 Invalid phone

 No proof after follow-ups

 Fraud suspected

Re-check if:

 Minor mismatch

 Recently relocated

Publish with warning if:

 Contact verified but limited documentation

 NGO active but unregistered

Transparency builds trust.

You might also like