Scribd
Upload
17 views3 pages

Containerize ETL Script with Docker

This document provides a guide on how to containerize an ETL validation script using Docker to ensure consistent execution across different environments. It includes instructions for creating a Dockerfile, specifying dependencies in a requirements.txt file, and running the container with necessary environment variables for database connection. The advantages of this approach include isolation, portability, and scalability for ETL testing and deployment.

Uploaded by

maheshtester1919
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
17 views3 pages

Containerize ETL Script with Docker

This document provides a guide on how to containerize an ETL validation script using Docker to ensure consistent execution across different environments. It includes instructions for creating a Dockerfile, specifying dependencies in a requirements.txt file, and running the container with necessary environment variables for database connection. The advantages of this approach include isolation, portability, and scalability for ETL testing and deployment.

Uploaded by

maheshtester1919
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Would you like me to show you how to containerize this script using Docker so it can run

consistently across any environment?


Containerizing your ETL validation script with Docker ensures it runs exactly the same way on
your local machine, a server, or in the cloud, without worrying about whether the right Python
version or SQL drivers are installed.
1. The Blueprint: Dockerfile

Since your script uses pyodbc to talk to SQL Server, the Docker image needs the official
Microsoft ODBC driver installed. Create a file named Dockerfile (no extension) in your project
folder:

Dockerfile

# Use an official Python runtime as a parent image


FROM python:3.10-slim

# Install system dependencies and Microsoft ODBC Driver for SQL Server
RUN apt-get update && apt-get install -y \
curl gnupg unixodbc-dev \
&& curl [Link] | apt-key add - \
&& curl [Link] >
/etc/apt/[Link].d/[Link] \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql18 \
&& apt-get clean

# Set the working directory


WORKDIR /app

# Copy requirements and install Python libraries


COPY [Link] .
RUN pip install --no-cache-dir -r [Link]

# Copy the script and the rest of the application


COPY . .

# Run the script when the container starts


CMD ["python", "etl_validation.py"]
2. The Dependencies: [Link]

Create a [Link] file to list the libraries your script needs:

Plaintext

pyodbc

3. Running the Container

To build and run your validation tool, use these commands in your terminal:
Build the image:
docker build -t etl-validator .

Run the container:


Since the script needs to connect to your database, you should pass your credentials as
Environment Variables for security:

docker run -e DB_PASSWORD='your_password' etl-validator

4. Important Shift: SQL Authentication

When running inside a Docker container (which is usually Linux-based), Windows


Authentication (Trusted_Connection=yes) is difficult to set up. It is much easier to switch
your connection string to use SQL Authentication:

# Update this in your Python script for Docker compatibility


conn_str = (
f"Driver={{ODBC Driver 18 for SQL Server}};"
f"Server={[Link]('DB_SERVER')};"
f"Database={[Link]('DB_NAME')};"
f"UID={[Link]('DB_USER')};"
f"PWD={[Link]('DB_PASSWORD')};"
"TrustServerCertificate=yes;"
)

Why this is better for ETL Testing

● Isolation: Your testing environment is completely separate from your dev environment.
● Portability: You can hand this Docker image to a QA teammate, and it will work
immediately.
● Scalability: You can trigger this container to run as part of a CI/CD pipeline every time
new data is loaded.

You might also like