AI Based Legacy Data Extraction and Processing Tool

Details (Sample)

Particulars Details

Ministry of Statistics and Programme

Name of Ministry/Department
Implementation

10, East Block, Rama Krishna Puram, New

Address
Delhi, Delhi 110066

Name of the Nodal Officer

Phone Number (Nodal Officer)

Email ID

Domain Area of the Problem Statement

Data extraction from reports/documents and
(Health stats, Price stats, industry
storing in standard database
statistics etc.)

☐ Data Collection
☐ Data Quality
☐ Statistical Methodology
☐ Data Accessibility
☐ Data Integration and Interoperability
Category of problem statement
☐ Data Timeliness

(Select all that are applicable)  Data Standardization

 Data Utilisation and Analysis
☐ Data Visualization
☐ Data Transparency
☐ Technological Infrastructure for Data
Management
 Data Extraction and Pipeline Creation
☐ Others (please specify)

14
 ☐ Research and development of new
What kind of support do you expect
methodology
from the solution giver?  Development or modification of
tools/process

 Subject Matter Experts (SMEs)

 Existing Datasets

☐ Technical Resources
What kind of support/resources will you
☐ Collaborative Networks
be able to share with MoSPI?
☐ IT Infrastructure
 Physical Infrastructure (space for
partners to work from)
☐ Others (please specify)

Problem Statement

A. Problem statement Identified (Max 100 words) *

(Write a crisp and specific problem statement identified by the Ministry/Department. Include
aspects such as what the problem is, whom it impacts, and scale of impact. Try to give
data figures wherever possible. Make sure the problem statement is related to official statistics
areas such as health stats, price indices, etc.)

Legacy data on surveys and macroeconomic indicators, stored in reports, CSVs,

Excel files (with issues such as merged cells, text in Hindi etc.), and images
requires detailed coding for extraction. Without this knowledge/resource for
coding, the data's usability is significantly impacted, hindering efficient analysis
and decision-making.

Contains 46 words

15
B. Methodology used for identifying the problem statement. *
(Detail how the problem was identified, including any studies conducted, resources referred
to, or methodologies applied.)

The issue was identified through an assessment of existing data management

practices within MoSPI. The review involved analyzing the efficiency of using
relational databases for data storage in tables versus if it remains as
document/image in file format. During the review, using modern large language
models (LLMs) for data extraction and analytics was also one of the areas
identified.

C. Challenges imposed and need for solving them. *

(Explanation of the current situation, including relevant data and statistics that highlights the
need for addressing this problem. List all key stakeholders affected by this problem, including
internal teams, external partners, or end-users. Highlight the potential long-term impacts if
the problem remains unsolved.)

The format in which data is often stored gives rise to several challenges:

1. Incompatibility with Modern Tools:

The legacy data is generally stored in non-readable formats, hindering
advanced analysis.

2. Operational Inefficiencies:
Extracting and processing data from these reports, CSVs, Excel formats
etc. is time-consuming and requires code-intensive skills, which slows
down data retrieval and decision-making.

3. Data Fragmentation:
Data in various non-usable formats is often siloed, making it challenging

16
 to integrate datasets for comprehensive analysis.

4. Limited Scalability:
As data volume increases, extraction and processing methods using
intensive coding struggle to keep up, limiting the ability to perform large-
scale analytics.

5. Lack of Standardization:
The diverse formats and languages used in legacy data complicate its
integration with modern systems, leading to inconsistencies and potential
errors.

Key stakeholders affected include data analysts, policymakers, and internal

teams responsible for economic and statistical analysis. If left unresolved, this
issue could reduce productivity, and slow down innovation.

D. Existing processes/systems in place to deal with the challenges (Max 150 words).
*
(How is the Ministry/Department currently addressing the problem statement? In case no way
has been found to manage it, kindly mention that as well.)

Currently, data extraction from these formats is done manually by staff with
coding expertise or through IT support, which is not sustainable in the longer
run.

Contains 26 words

E. Expected outcome(s) for stakeholders post resolution. *

(Clearly outline the benefits and improvements the impacted stakeholders will experience
once the problem is resolved. Also mention the essential features of the solution.)

17
 Once resolved, stakeholders will be able to leverage advanced analytics. This
will result in:

1. Improved Data Accessibility: AI-based low-code/no-code tools for extracting

and converting data from unusable formats to usable ones will enable quicker
and easier data retrieval, reducing dependencies on specialized coding
knowledge.

2. Improved Data Utilization: More users will be able to use the data effectively
without needing specialized technical skills.

3. Enhanced Productivity: Reduced time and effort in processing data, allowing

staff to focus on higher-value tasks.

F. How urgent do you consider it to solve this problem? *

(What is the expected timeline for resolution of the problem statement.)

 High Priority: The problem significantly impacts daily operations; needs

immediate attention.

☐ Medium Priority: The problem affects productivity or efficiency but does

not halt operations. It should be addressed within a reasonable
timeframe.

☐ Low Priority: The problem has minimal impact on overall operations and
can be resolved at a later time without major consequences.

G. Expected timeline for resolution? *

(What is the expected timeline for resolution of the problem statement.)

Short-term: 3-6 months

18
H. Share any global best practices you'd like to highlight?
(Mention any global best practices you know of that could address your issue or be
implemented to solve it. Include links where possible.)

Automated Data Extraction Tools:

Many organizations globally are leveraging AI-powered tools like ChatGPT and
Microsoft Copilot to automatically extract data from PDFs, images, and other
non-usable formats. These tools quickly convert data quickly into usable formats,
enhancing accessibility and reducing reliance on manual processing.

(Complete this section only if the Ministry/Department submitting the proposal has potential solutions in mind)

I. Proposed solutions (Max 200 Words)

(Provide an overview of the proposed solutions, including the key milestones and tentative
timelines for each phase of implementation. Try to post your idea in points /diagrams /
Infographics /pictures.)

N/A

J. Analysis of the feasibility of the solution

(Evaluate the viability of the proposed solution, considering its technical, financial, and
operational aspects, along with identifying potential challenges and risks.)

N/A

19