Project Report: Web Vulnerability
Scanner using HTTP headers
Name: Rajat Bhat
Reg No : 251100690027
Application Title: Web Vulnerability Scanner using headers
Technology Used: Bash Scripting (Linux Environment)
1. Objective
The goal of this project is to create a simple tool that checks a website for basic security
issues. It looks for missing security settings in HTTP headers (like HSTS and X-Frame-
Options) and checks if sensitive files (like .env or [Link]) are exposed. The scanner
helps beginners understand how to spot web vulnerabilities using Linux commands.
2. Problems Addressed
Websites can have security weaknesses that attackers might exploit, such as:
Missing security headers, which can allow attacks like clickjacking or data theft.
Exposed sensitive files (e.g., .env with passwords) that should be hidden.
Using HTTP instead of HTTPS, which risks data being intercepted. This project
builds a tool to automatically detect these issues and report them.
3. Description
The Web Vulnerability Scanner is a Bash script that checks a website for common
security problems. It sends HTTP requests to the target website, analyzes the server’s
response, and creates a report listing any vulnerabilities found. It uses simple Linux
tools to make the process easy to understand and is designed for students learning
cybersecurity basics.
4. Approach
To build the scanner, I used the following steps:
1. Input: Ask the user for a website URL (e.g., [Link]
2. Check Reachability: Use curl to ensure the website is online.
3. Analyze Headers: Use grep to check for security headers like HSTS and X-
Frame-Options.
4. Check Files: Use curl and awk to see if sensitive files are accessible.
5. Check HTTPS: Use grep to verify if the site uses secure HTTPS.
6. Generate Report: Use echo and sed to create a clear report summarizing
findings.
7. Clean Up: Use rm to delete temporary files.
This approach keeps the script simple, using basic tools to perform security checks and
produce a readable output.
5. Commands and Tools Used
Here’s how each tool is used in the script, explained simply:
1. echo
a. What it does: Prints text to the screen or writes to files.
b. Use in script: Shows scan progress (e.g., “Checking if website is up...”)
and writes report headers (e.g., “Website Scan Report”).
c. Cybersecurity role: Displays results and builds the report for easy
reading.
2. curl
a. What it does: Sends HTTP requests to websites and gets responses.
b. Use in script: Checks if the website is reachable, fetches headers (e.g.,
Server: Apache), and tests for sensitive files (e.g., /.env).
c. Cybersecurity role: Gathers data needed to check for vulnerabilities like
exposed files or missing headers.
3. grep
a. What it does: Searches for specific text in files or input.
b. Use in script:
i. Finds the Server header (grep -i "^Server:" [Link]).
ii. Checks for security headers like HSTS (grep -qi "Strict-Transport-
Security" [Link]).
iii. Verifies HTTPS usage (grep -q "^https").
c. Flags:
i. -i: Ignores case (e.g., matches “Server” or “server”).
ii. -q: Silently checks without printing output.
d. Cybersecurity role: Identifies missing security headers or protocols,
flagging potential risks.
4. awk
a. What it does: Processes text by splitting it into parts and applying logic.
b. Use in script: Checks HTTP status codes for sensitive files (awk '{ if ($1
== "200") print "✗ Found: " $2 " (Risk)"; else print "✓ Not found: " $2 }') to
see if files like .env are accessible.
c. Cybersecurity role: Detects exposed files that could leak sensitive data,
like passwords or API keys.
5. sed
a. What it does: Edits or formats text, like removing or adding parts.
b. Use in script:
i. Cleans the Server header (sed 's/Server: //' removes “Server: ” to
show just “Apache”).
ii. Adds a footer to the report (sed -i '$a\-------------------').
c. Cybersecurity role: Makes output and reports clear and professional.
6. rm
a. What it does: Deletes temporary files.
b. Use in script: Removes [Link] after scanning (rm -f [Link]).
c. Cybersecurity role: Ensures no sensitive data (like headers) is left
behind.
6. Code Explanation
Code Breakdown
1. Input Check:
a. Checks if a URL is provided. If not, shows how to use the script and exits.
b. Stores the URL in TARGET and prints a header.
c. Why: Ensures the script has a valid website to scan.
2. Website Reachability :
a. Uses curl to check if the website is online.
b. Prints “Website is up” or “Website is down” and stops if down.
c. Why: Avoids scanning unreachable sites.
3. Server Information :
a. Uses curl to get HTTP headers and save them to [Link].
b. Uses grep to find the Server header and sed to clean it (e.g., removes
“Server: ”).
c. Shows the server type (e.g., “Apache”) or “Not shown” if hidden.
d. Why: Exposed server details can reveal vulnerabilities.
4. Security Headers :
a. Uses grep -qi to check for HSTS and X-Frame-Options headers.
b. Prints if they are present (“Good”) or missing (“Risk”).
c. Why: Missing headers increase risks like data theft or clickjacking.
5. Sensitive Files:
a. Checks files like /.env using curl to get HTTP status codes.
b. Uses awk to print if files are found (200 status) or not.
c. Why: Exposed files can leak sensitive data like passwords.
6. HTTPS Check :
a. Uses grep -q to check if the URL starts with https.
b. Prints if HTTPS is used (“Secure”) or not (“Not secure”).
c. Why: HTTPS ensures data is encrypted.
7. Report Generation :
a. Creates scan_report.txt with echo.
b. Adds server info, header checks, file results, and HTTPS status using
grep and echo.
c. Uses sed to add a footer (divider and “Scan done!”).
d. Why: Summarizes findings in a clear file for review.
8. Final Output and Cleanup :
a. Shows the report filename and how to view it.
b. Uses rm to delete [Link].
c. Why: Ensures no temporary files are left behind.
7. Tools Familiar Before Project
echo: Printing text.
grep: Searching for text patterns.
sed: Editing text.
rm: Deleting files.
if/else: Making decisions in scripts.
8. Tools Learned During Project
curl: Fetching web data like headers and file responses.
awk: Processing text with logic, like checking HTTP status codes.
9. Sample Vulnerabilities Checked
Missing HSTS: Risks unencrypted connections (data theft).
Missing X-Frame-Options: Risks clickjacking attacks.
Exposed Files (e.g., .env, [Link]): Could leak sensitive data.
No HTTPS: Risks data being intercepted.
10. Output
On-Screen: Shows scan progress and results (e.g., “✓ HSTS found”, “✗ Found:
/.env”).
Report File (scan_report.txt): Lists:
o Server type (e.g., “Apache”).
o Security headers (OK or Missing).
o Sensitive files (Found or Not found).
o HTTPS status (OK or Missing).
o
Example Report:
text
Website Scan Report
Target: [Link]
Date: Thu Oct 16 12:15:00 IST 2025
-------------------
Server Info
Apache
Security Headers
HSTS: Missing
X-Frame-Options: OK
Sensitive Files
/.env: Not found
/[Link]: Found (Risk)
/admin: Not found
HTTPS Check
HTTPS: Missing
-------------------
Scan done!
11. Screenshots
[Link] Website
Note : The website used for exploitation is DVWA vulnerable website
Output of Vulnerable website
1. The output shows the header do not contain https heads which refers that the
website is not safe
2. Legitimate Website
Note: The website i have used here is [Link]
1. [Link] output shows the header contains https heads which refers that the
website is safe
12. Learning Outcome
Learned how to automate web security checks using Bash scripting.
Understood how curl and awk help analyze web data for vulnerabilities.
Improved skills in Linux commands and creating reports.
Gained knowledge about HTTP security headers (HSTS, X-Frame-Options) and
why they protect websites.
13. Conclusion
The Web Vulnerability Scanner is a simple, effective tool for finding basic web security
issues. It uses Linux commands like grep, awk, and sed to check for missing headers,
exposed files, and HTTPS usage. This project shows how Bash scripting can help
beginners learn cybersecurity by automating vulnerability detection and reporting results
clearly