0% found this document useful (0 votes)
19 views2 pages

Shell Script for PostgreSQL Data Extraction

This bash script automates the extraction of metadata, checksums, and full data from a PostgreSQL database. It generates SQL files from specified generator files, checks for their existence, and executes them if they are created successfully. The script also includes user prompts and waits for user input before exiting.
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
19 views2 pages

Shell Script for PostgreSQL Data Extraction

This bash script automates the extraction of metadata, checksums, and full data from a PostgreSQL database. It generates SQL files from specified generator files, checks for their existence, and executes them if they are created successfully. The script also includes user prompts and waits for user input before exiting.
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

#!

/bin/bash

# Define parameters
dbname="$1"
schema_name=$2
srvname="$3"
dbuser="$4"
dbpass="$5"
basepath=$6

export PGPASSWORD="$dbpass"
# Metadata - Rowcount extraction
echo -e "\033[1;42mCREATING METADATA SQL FILE FROM GENERATOR FILE & INITIATING
METADATA EXTRACTION\033[0m"
sleep 1
psql -h $srvname -d $dbname -U $dbuser -v schema_name="$schema_name" -v
base_path="$basepath" -f $basepath/Postgres/Input/metadata_sql_gen.sql -t -q -X -o
$basepath/Postgres/Input/metadata_sql.sql
sleep 1
echo "Checking whether the input files are created or not!"
sleep 1
# File path to check
metadata_PATH="$basepath/Postgres/Input/metadata_sql.sql"
# Check if the file exists and is not empty
if [ -f "$metadata_PATH" ] && [ -s "$metadata_PATH" ]; then
echo "Files are created in input folder. Executing metadata extract SQL
Statements..."
sleep 2
psql -h $srvname -d $dbname -U $dbuser -f
$basepath/Postgres/Input/metadata_sql.sql
else
echo "Error: $metadata_PATH is not created, check the script!"
fi

# Generate SQL statements to calculate checksums


echo -e "\033[1;42mCREATING CHECKSUM SQL FILE FROM GENERATOR FILE & INITIATING
CHECKSUM EXTRACTION\033[0m"
sleep 1
psql -h $srvname -d $dbname -U $dbuser -v schema_name="$schema_name" -v
base_path="$basepath" -f $basepath/Postgres/Input/cksum_sql_gen.sql -t -q -X -o
$basepath/Postgres/Input/cksum_sql.sql
sleep 1
echo "Checking whether the input files are created or not!"
sleep 1
# File path to check
cksum_PATH="$basepath/Postgres/Input/cksum_sql.sql"

# Check if the file exists and is not empty


if [ -f "$cksum_PATH" ] && [ -s "$cksum_PATH" ]; then
echo "Files are created in input folder. Executing checksum SQL Statements..."
sleep 2
psql -h $srvname -d $dbname -U $dbuser -f $basepath/Postgres/Input/cksum_sql.sql
else
echo "Error: $cksum_PATH is not created, check the script!"
fi
# Full Extract file gen and execution
echo -e "\033[1;42mCREATING FULL EXT SQL FILE FROM GENERATOR FILE & INITIATING FULL
EXT\033[0m"
sleep 1
psql -h $srvname -d $dbname -U $dbuser -v schema_name="$schema_name" -v
base_path="$basepath" -f $basepath/Postgres/Input/fullext_sql_gen.sql -t -q -X -o
$basepath/Postgres/Input/fullext_sql.sql
sleep 1
echo "Checking whether the input files are created or not!"
sleep 1
# File path to check
fullext_PATH="$basepath/Postgres/Input/fullext_sql.sql"

# Check if the file exists and is not empty


if [ -f "$fullext_PATH" ] && [ -s "$fullext_PATH" ]; then
echo "File are created in the input folder. Executing Full extract SQL
Statements..."
sleep 2
psql -h $srvname -d $dbname -U $dbuser -f
$basepath/Postgres/Input/fullext_sql.sql
else
echo "Error: $fullext_PATH is not created, check the script!"
fi
unset PGPASSWORD

# Print text with different colors and styles


echo "Script execution complete. Press any key to exit."

# Wait for user to press any key


read -n 1 -s

echo "Exiting..."

Common questions

Powered by AI

The bash script performs multiple tasks related to PostgreSQL database management. It generates SQL files for metadata extraction, checksum calculation, and full data extraction from specified generator files and executes these SQL statements if the files are successfully created. Specifically, it first generates a metadata SQL file and executes the SQL for extracting metadata rowcount, then it creates and executes a checksum SQL file for calculating checksums, and finally, it generates and executes a SQL file for a complete data extraction. The script checks whether these generated files exist and contain data before execution. Any issues with file generation are reported with error messages. Additionally, it manages credentials and outputs process completion messages before awaiting further user interaction to terminate the execution .

To enhance security during PostgreSQL operations, the script uses the 'PGPASSWORD' environment variable to temporarily manage the password required for database authentication, avoiding the need for interactive input. By setting this variable at the script's beginning and unsetting it at the end, the script minimizes password exposure to the runtime environment. This method prevents passwords from being hardcoded or stored in script files, which could be insecure. Once the script completes, clearing 'PGPASSWORD' further reduces the risk of unauthorized access .

The bash script interacts with PostgreSQL through shell commands by using the 'psql' command-line utility. It executes SQL files that are generated based on input parameters to perform metadata extraction, checksum calculation, and full data extraction. Each psql command is prefaced with parameters defining the host, database, user, and password, indicating the database connection details and the SQL file to be executed. This integration allows the script to manage database tasks directly from the command line environment, providing a seamless interface for executing complex database management tasks .

The script uses a conditional check mechanism to verify that each SQL file is created and contains data before attempting execution. It checks if files, such as 'metadata_sql.sql', 'cksum_sql.sql', and 'fullext_sql.sql', exist and are not empty using '[ -f "$variable_PATH" ] && [ -s "$variable_PATH" ]' for each respective task. If these conditions are satisfied, it proceeds to execute the SQL statements contained within them. If the files are missing or empty, the script outputs an error message indicating the failure, thus ensuring only successfully generated and populated files are executed .

The script implements error handling by checking the existence and size of the files that should be generated, using conditional statements. For each type of SQL extraction (metadata, checksum, full extract), it verifies that the related SQL file exists and is not empty before attempting execution. If these checks fail, implying that the file was not properly created or populated, the script issues an error message specifically identifying the problematic file, thus alerting the user to the failure. This approach allows for early detection of issues in the file generation stage, preventing attempts to execute incomplete or non-existent SQL instructions .

The script uses a series of input parameters to define variables like 'dbname', 'schema_name', 'srvname', 'dbuser', 'dbpass', and 'basepath', which allows it to be executed for different PostgreSQL databases and environments without modification. This flexibility means that by passing different parameters when invoking the script, users can adapt it to varied use cases, including different databases, schemas, server instances, user credentials, and file paths. Consequently, the script's reliance on input parameters enhances its reusability and reduces the need for hardcoded database configuration details, thus enabling broad applicability .

The script temporarily sets the PostgreSQL password by exporting the environment variable 'PGPASSWORD' to manage database authentication. This variable is initialized with the database password provided as an input parameter '$5' when the script is executed. By setting 'PGPASSWORD', the script facilitates the execution of PostgreSQL command-line operations without requiring interactive password input. The password is effectively managed within the script's runtime and is unset upon completion to maintain security .

The script automates repetitive and error-prone database maintenance tasks, such as metadata extraction, checksum calculation, and full data extraction, by executing a sequence of predefined SQL statements. For database administrators, automation offers several benefits including increased efficiency, reduced human error, and consistent operation across database instances. By leveraging scripts, administrators can ensure that complex tasks are executed in a timely and reliable manner without ongoing manual intervention, thereby freeing up resources for more strategic activities .

Creating SQL files in separate phases for metadata, checksum, and full extract facilitates organized and modular database management. Each phase targets specific operational goals, ensuring precision and focus. By dividing tasks into distinct segments, the script minimizes cross-phase dependencies, reducing complexity and improving fault isolation. Should an error occur in one phase, it won't necessarily impact the others, allowing for more straightforward debugging and corrective actions. Furthermore, this design supports scalability and efficiency, as each phase can be enhanced individually without affecting the entire script, benefiting maintenance and evolution of database management practices .

If the script did not implement checks for file creation and content, numerous risks could emerge. Without verifying whether a SQL file is created and populated, the script might attempt to execute incomplete or incorrect SQL statements, leading to incomplete data operations or database errors. This could result in unreliable metadata, inaccurate checksums, and incomplete data extractions. Additionally, in a production environment, such oversight might lead to data integrity issues, prolonged debugging efforts, and potential downtime while troubleshooting malformed scripts. Therefore, the pre-execution checks serve as a crucial safeguard against these vulnerabilities by ensuring that only valid, complete files trigger database actions .

You might also like