Chapter 3: Containerization and Continuous Integration in AWS
This chapter delves into two crucial technologies in modern software
development and deployment, specifically within the Amazon Web Services
(AWS) ecosystem, containerization with Docker and continuous integration
with Jenkins streamline the development process, improve application
portability, and enhance scalability.
Jenkins, as a Continuous Integration (CI) server, needs access to the source
code of your application. This is fundamental for automating the build, test,
and deployment phases.
Git is the industry-standard version control system, allowing developers
to track changes to their code, collaborate effectively, and manage different
versions of the software. Integration is essential because Jenkins pulls the
latest code from the Git repository to work with.
When you create a project (or "job") in Jenkins, you provide the URL of
your Git repository (e.g., GitHub, GitLab, Bitbucket) so Jenkins knows where
the code is located. Git repositories contain multiple branches, and you need to
specify which branch Jenkins should use, such as main for production-ready
code or develop for ongoing development.
Jenkins needs permission to access the Git repository, which is done
securely using either SSH keys or tokens. SSH keys involve generating a pair of
keys (private and public), where the public key is added to the Git repository,
and the private key is used by Jenkins. Tokens, like Personal Access Tokens in
GitHub, are strings that grant specific permissions. The most common trigger
for builds is when a developer pushes new code to the Git repository, notifying
Jenkins to automatically start a new build, a core part of CI.
In a collaborative workflow, developers create "pull requests" to propose
changes, and Jenkins can be configured to automatically build and test the
code in a pull request before it's merged into the main branch, helping to catch
errors early. You can also schedule Jenkins to run builds at specific times (e.g.,
every night) for tasks like running automated tests or generating reports. To set
up your first automation task in Jenkins, you start by creating a "Freestyle
Project," giving it a descriptive name (e.g., "MyWebApp-Build"), and
configuring source code management with Git by providing the Git repository
URL, branch, and authentication details.
For the very first build, manual triggering is typically used to ensure
everything is set up correctly. Jenkins allows you to define what actions to take
during a build, and a simple way to start is by executing shell commands, such
as echo "Hello from Jenkins!", to verify that Jenkins can execute commands.
After configuring the build, you click "Build Now" in the Jenkins job interface to
start the build process and check the "Console Output," a detailed log of
everything that happens during the build, crucial for monitoring progress,
identifying errors, and troubleshooting.
Docker is a platform for developing, shipping, and running applications. It
solves the problem of "it works on my machine" by packaging applications and
their dependencies into isolated units called containers. These containers
include everything an application needs to run: the code itself, libraries, system
tools, and the runtime environment, ensuring that the application behaves
consistently across different environments (developer's laptop, testing server,
production cloud).
The Docker architecture includes the Docker Client, the command-line
interface (CLI) that developers use to interact with Docker with commands like
docker run and docker build; the Docker Daemon, a background service on the
host machine responsible for building, running, and managing containers; and
the Docker Registry, a service for storing and sharing Docker images, such as
the public Docker Hub or private registries like AWS ECR (Elastic Container
Registry).
Key features of Docker include portability and consistency, as a container
packages everything an application needs to run, ensuring it runs the same way
everywhere and reduces deployment issues across development, testing, and
production. Isolation is another important feature, as containers are isolated
from each other and the host system, providing security and preventing
conflicts between applications with different software requirements.
Containers are also resource-efficient compared to Virtual Machines (VMs),
sharing the host OS kernel for faster startup times and lower resource
consumption, allowing more applications to run on the same hardware.
Docker simplifies the development lifecycle, enabling developers to
quickly set up consistent environments, easily package and deploy applications
rapidly, and facilitate Continuous Integration and Continuous Delivery (CI/CD)
pipelines. Furthermore, Docker makes it easier to scale applications
horizontally and is the foundation for container orchestration tools like
Kubernetes and Docker Swarm, which automate the management and scaling
of large numbers of containers in complex deployments.
Containerization is analogous to physical shipping containers,
standardizing how goods are transported to ensure consistency and isolation.
Software containers are lightweight, portable packages that bundle an
application with its dependencies, creating an isolated environment for the
application. Before containers, deploying applications was often complex,
involving manual configuration of server environments, which was time-
consuming and error-prone.
Virtual Machines (VMs) provide isolation but are resource-intensive, with
each VM including a full operating system. Containers are more efficient
because they share the host OS kernel and have their own isolated user space,
making them much lighter and faster than VMs. The core idea is to package an
application and its dependencies into a self-contained unit that can be easily
moved and run consistently across different environments, solving the "it
works on my machine!" problem and simplifying deployment.
Virtual Machines (VMs) emulate an entire computer system, with each
VM including its own operating system, kernel, libraries, and applications. This
provides strong isolation, as one VM's crash generally doesn't affect others.
However, because each VM has its own OS, they tend to be resource-intensive
and have significant boot times. VMs are ideal for running different operating
systems on the same hardware, legacy applications, or when strong isolation is
paramount.
Containers, on the other hand, are lightweight and share the host
operating system's kernel. Each container packages only the application and its
necessary dependencies, offering good isolation at the application level while
sharing the underlying OS. Containers are much more resource-efficient,
allowing for higher density, and start almost instantly, making them ideal for
rapid deployment and scaling. They are excellent for microservices, web
applications, CI/CD pipelines, and situations requiring speed and efficiency.
Docker must be installed on the machine where containers will run, with
installation steps varying across Linux, Windows, and macOS. It's best to refer
students to the official Docker documentation for the most up-to-date and
accurate installation instructions, as this session emphasizes understanding
Docker concepts and how to use Docker commands, rather than spending time
troubleshooting individual installation issues.
Interacting with Docker involves using basic commands. The docker run
command creates and starts a container from an image. For example, docker
run hello-world runs a simple test container by checking if the hello-world
image is on your system, downloading it from Docker Hub if necessary, and
then creating and starting a container from that image.
The hello-world image is a small image that prints a message and exits,
used to verify that Docker is installed correctly. The docker ps command lists
running containers, showing details like the Container ID, status, and port
mappings, similar to the ps command in Linux. The docker ps -a command lists
all containers, both running and stopped. The docker stop command gracefully
stops a running container, requiring the Container ID, and sends a signal to the
container to shut down after finishing its current operations.
The docker rm command removes a stopped container permanently, also
requiring the Container ID, and frees up the resources it was using. It's
important to stop a container before removing it. The docker images command
lists all downloaded or built Docker images on your system, which are like the
"templates" for creating containers. The docker pull command downloads an
image from a registry like Docker Hub. For example, docker pull ubuntu
downloads the official Ubuntu image, which can then be used with docker run
to create Ubuntu containers.
Creating custom Docker images involves using a Dockerfile, a text file
containing instructions to build a Docker image, like a "recipe" for creating your
container. Key instructions include FROM, which specifies the base image to
start from, often a standard OS image or another existing image; COPY, which
copies files and directories from your local machine into the image; RUN, which
executes commands inside the image during the build process, such as
installing software or setting up the environment; WORKDIR, which sets the
working directory inside the image for subsequent commands; and CMD, which
specifies the default command to run when a container is started from the
image, such as starting a web server.
To build an image, you use docker build -t <name> . in the directory
containing the Dockerfile. The docker build command builds the image, the -t
<name> flag lets you name the image, and the specifies the current directory
as the build context.
After building an image, you run containers using docker run with your
custom image's name. Configuration options include mapping ports with -p
<host_port>:<container_port> to make your application accessible from the
outside, setting environment variables with -e <VARIABLE_NAME>=<value> to
configure application behavior, and configuring container settings for
consistent, isolated application environments.
Jenkins is an open-source automation server, a central hub that helps
software development teams automate repetitive tasks like building, testing,
and deploying code, freeing up developers to focus on writing code instead of
manual processes. Continuous Integration (CI) involves developers writing
code, followed by manual compilation, testing, and deployment, which is time-
consuming and error-prone.
Jenkins automates this entire flow, called a "pipeline". You configure
Jenkins with a series of steps to be executed, and when a trigger occurs (e.g., a
developer pushing code to a repository), Jenkins automatically starts this
pipeline. Jenkins can pull the latest code, build the application, run various
automated tests, generate reports, and, if everything is successful, deploy the
application to the target environment.
Jenkins has a vast plugin ecosystem, integrating with almost any tool in
the software development landscape, including version control systems (Git),
testing frameworks (JUnit), and deployment platforms (Kubernetes), making it
very versatile and adaptable to different project needs. Jenkins' core value
proposition is automation, taking manual, error-prone tasks and making them
automatic and repeatable, leading to faster development cycles, reduced
errors, and more consistent releases.
Jenkins is a cornerstone of CI/CD practices. Continuous Integration
involves frequently merging code changes into a central repository, followed by
automated building and testing. Continuous Delivery automates the release
process, ensuring that software can be deployed to production at any time.
Jenkins makes implementing these practices much easier. The thousands of
available plugins extend Jenkins' functionality to support various tools,
technologies, and workflows, such as plugins for specific programming
languages, testing tools, or cloud platforms.
Jenkins pipelines allow you to define your entire CI/CD workflow as code
(usually in a Jenkinsfile), which can be version-controlled alongside the
application code, providing transparency, reproducibility, and allowing for
treating the deployment process like any other part of the software. Jenkins
provides a central dashboard where teams can monitor the status of their
builds, tests, and deployments, making it easier to identify and resolve issues
quickly, and improving team collaboration and overall project health.
Terraform is an open-source Infrastructure-as-Code (IaC) tool that allows
you to define and provision infrastructure resources, such as virtual machines,
storage, and networks, using code in a declarative configuration language.
Launching Jenkins with Terraform means using Terraform to automatically
create and configure the infrastructure for Jenkins, instead of manually setting
up a server. You write Terraform code to describe the infrastructure you want,
and when you run Terraform, it reads your code and communicates with cloud
providers (like AWS, Azure, GCP) to create those resources.
For launching Jenkins, this typically involves creating a virtual machine
(e.g., an EC2 instance in AWS), configuring its settings, configuring network
settings, and optionally using "provisioners" (scripts that run after the VM is
created) to install Jenkins, set up users, and configure basic Jenkins settings.
The benefits of using Terraform include ensuring consistency, as every time you
run the Terraform code, you get the same infrastructure without manual setup
errors; repeatability, as you can easily create multiple Jenkins servers with the
same configuration; scalability, as you can easily scale your Jenkins
infrastructure by modifying your Terraform code; version control, as you can
store your Terraform code in Git; and automation, as the entire process of
setting up Jenkins infrastructure is automated.
Terraform defines and manages infrastructure using code for consistent
setups and automates Jenkins server setup, avoiding manual configuration. The
high-level steps involve defining EC2 instance details, configuring the instance
type, OS, and network, and optionally running scripts to install Jenkins. This
ensures consistency, scalability, and version control for Jenkins infrastructure.
Infrastructure as Code (IaC) provides repeatability and consistency, automation
of infrastructure provisioning, version control and collaboration, and
integration of infrastructure and application deployment.
Getting started with Jenkins configuration typically involves accessing the
Jenkins UI via a web browser at a specific port (e.g., [Link]
ip:8080). Key configuration areas include System Configuration for global
settings, Plugin Management for installing and updating plugins, User &
Security Management, and Node Management for connecting and managing
"nodes" or "agents" to distribute build workloads. Plugins greatly enhance
Jenkins' capabilities and are like "apps" for the Jenkins server, such as the Git
plugin for integrating with Git, the Maven plugin for building Java projects, the
Docker plugin for working with Docker, and AWS plugins for deploying to AWS.
Jenkins configuration refers to the process of setting up and customizing
Jenkins to meet the specific needs of your software development workflows.
This involves defining build jobs, configuring plugins, managing users and
security, setting up nodes for distributed builds, and tailoring the overall
behavior of the Jenkins server.
The base installation provides the framework, but you need to configure
the machines, assembly lines, and quality control processes to build your
specific product efficiently. Jenkins configuration is about setting up these
"machines" and "processes" for your software projects, involving different
types of jobs or projects, like building a Java application, running automated
tests, or deploying a Docker container. For each job, you define the source code
repository, build steps, test commands, and deployment procedures.
Furthermore, you configure Jenkins itself, including installing and
managing plugins, managing user access and security settings, and, for larger
projects, configuring "nodes" or "agents" – separate machines that Jenkins can
distribute build workloads to, improving performance and scalability.
Essentially, configuration tailors Jenkins from a generic automation server into
a bespoke CI/CD engine for your team.