Table of Contents
Scaffolding...................................................................................................................... 2
Purpose and benefits of CI and CD...............................................................................2
CI (Continuous Integration):............................................................................. 2
CD (Continuous Delivery/Continuous Deployment):........................................ 2
Where to use DevOps?.................................................................................................. 3
Use DevOps in the following scenarios:........................................................... 3
Branch commands......................................................................................................... 4
Github Actions................................................................................................................ 4
Key Elements in Workflows:............................................................................. 4
YAML in GitHub Actions.................................................................................................4
Docker..............................................................................................................................5
Basic Docker Commands:................................................................................ 5
Port Mapping.................................................................................................... 5
DockerFile........................................................................................................ 5
From Activity 4.................................................................................................. 7
CMD in DockerFile........................................................................................... 7
Maintaining Docker Tags............................................................................................. 8
Best Practices for Tagging:............................................................................... 8
Tag by Semantic Version:............................................................................... 11
What is SHA?................................................................................................. 12
Timestamps for tagging............................................................................................. 12
Advantages of Using Timestamps:...................................................................... 12
Disadvantages of Using Timestamps:..................................................................12
Scaffolding
Scaffolding refers to setting up the foundational structure for machine learning projects. It
involves organizing project components, establishing coding practices, defining pipelines, and
configuring the infrastructure required for continuous integration, continuous deployment
(CI/CD), and model lifecycle management. The goal is to streamline development by providing a
consistent framework for building, testing, deploying, and monitoring machine learning models.
Key aspects of scaffolding in MLOps include:
1. Project Structure: Defining directories, data, models, and code organization.
2. Automation Pipelines: Setting up data pipelines, model training pipelines, and
deployment workflows.
3. Version Control: Implementing code, data, and model versioning.
4. Environment Setup: Configuring environments for local, staging, and production
deployments.
5. Testing Frameworks: Establishing unit tests, integration tests, and model validation
processes.
By creating this structure early, teams can reduce technical debt and improve collaboration
across data science, engineering, and operations teams.
Selenium -> tool for web application testing especially for applications which have a GUI
Purpose and benefits of CI and CD
CI (Continuous Integration):
● Purpose: Continuous Integration is a development practice where developers frequently
integrate their code into a shared repository, typically several times a day. Automated
tools are used to build, test, and validate new code before it is merged into the main
branch.
● Benefits:
○ Early Error Detection: Developers can identify and fix issues early by regularly
integrating their work.
○ Improved Code Quality: Automated testing ensures that new changes do not
break the codebase.
○ Faster Development: Frequent integrations enable developers to work in
parallel and reduce integration challenges.
CD (Continuous Delivery/Continuous Deployment):
● Purpose:
○ Continuous Delivery: Ensures that the codebase is always in a deployable
state. After code changes pass all stages of testing, the software is automatically
prepared for deployment but requires manual approval to be pushed to
production.
○ Continuous Deployment: Extends Continuous Delivery by automatically
deploying every change that passes all tests directly into production without
manual intervention.
● Benefits:
○ Faster Releases: Automation of the release process allows for rapid and reliable
deployment.
○ Reduced Risk: Small, incremental changes reduce the risk of introducing large,
problematic bugs.
○ Increased Productivity: Engineers spend less time on manual deployments and
can focus on writing code
Where to use DevOps?
● DevOps is a methodology and set of practices that integrates development and
operations teams to improve collaboration, automate processes, and streamline the
software development lifecycle (SDLC).
Use DevOps in the following scenarios:
1. Frequent Software Releases: For teams that need to deploy code regularly and rapidly
(e.g., SaaS companies), DevOps ensures faster and more reliable releases.
2. Cloud-native Applications: DevOps practices like infrastructure as code (IaC),
containerization, and automated scaling work well with cloud environments (e.g., AWS,
Azure, GCP).
3. Microservices Architecture: DevOps simplifies the management and orchestration of
microservices through automation and containerization tools like Kubernetes and
Docker.
4. Agile Development: Teams working with Agile methodologies benefit from the
increased automation and integration that DevOps provides, aligning closely with Agile’s
focus on iterative development.
5. Continuous Improvement/Optimization: DevOps enables ongoing optimization and
monitoring of software performance, infrastructure, and user feedback.
6. Startups & Growing Businesses: For small and medium-sized companies looking for
rapid scaling, DevOps provides the agility and automation necessary to keep up with
growing user demands.
7. Large-scale Enterprises: Large organizations can adopt DevOps to streamline
collaboration between traditionally siloed teams (development, QA, operations) and
manage complex software ecosystems.
Branch commands
Create a branch: git checkout -b branch_name
Switch branches: git checkout branch_name
Delete a branch: git branch -d branch_name
Github Actions
Key Elements in Workflows:
● Jobs: Define what to run (e.g., build, test, deploy).
● Steps: Commands or actions that execute in a job.
● Triggers: Define when a workflow should run (e.g., on: push).
YAML in GitHub Actions
Common elements:
● on: Specifies events to trigger the workflow (e.g., on: push).
● jobs: Define the steps for each stage (e.g., testing, building).
actions/checkout@v3, which checks out the repository code so that subsequent steps can
access it.
Docker
Basic Docker Commands:
● Build an image: docker build -t image_name .
● Run a container: docker run -d -p 80:80 image_name
● Stop a container: docker stop container_id
Port Mapping
Port mapping allows you to expose a Docker container's internal port to the host machine,
enabling external access to services running inside the container (e.g., a web server). By
default, containers are isolated from the host and don’t expose any ports.
Syntax: When running a container, you can specify the host and container port mapping using
the -p option.
docker run -d -p <host_port>:<container_port> <image_name>
Example: If a container runs a web server on port 80, and you want to access that server from
your host machine on port 8080, you would run:
docker run -d -p 8080:80 nginx
● This maps the container's port 80 (where the web server is running) to port 8080 on the
host, allowing you to access the web server at [Link]
DockerFile
A Dockerfile is a text file that contains all the instructions to build a Docker image
docker build -t my-nginx-image . //build docker image
docker run -d -p 8080:80 my-nginx-image //run docker container
From Activity 4
CMD in DockerFile
In a Dockerfile, the CMD instruction is used to specify the default command that will be executed
when a container is started from the image. It defines what should happen when the container
runs, such as starting an application.
Key Points about CMD:
1. Default Command: CMD sets the default command to run when the container starts. You
can override it by providing a different command when running the container.
2. Format: There are two formats for CMD:
Exec form (preferred):
CMD ["executable", "param1", "param2"]
○ This format does not invoke a shell and runs the executable directly.
Shell form:
CMD command param1 param2
○ This format runs the command in a shell, which can affect how commands and
parameters are interpreted (e.g., variable expansion).
3. Multiple CMD Instructions: If you specify multiple CMD instructions in a Dockerfile, only
the last one will take effect.
4. Usage Context: The CMD instruction is often used in combination with the ENTRYPOINT
instruction to define the main application that runs in the container.
Example Usage
Here’s a simple example of using CMD in a Dockerfile for a Python Flask application:
# Use the Python base image
FROM python:3.9-slim
# Set the working directory
WORKDIR /app
# Copy requirements and install
COPY [Link] .
RUN pip install -r [Link]
# Copy the application code
COPY . .
# Expose the port the app runs on
EXPOSE 5000
# Set the default command to run the Flask app
CMD ["flask", "run", "--host=[Link]", "--port=5000"]
Maintaining Docker Tags
Docker tags are used to label versions of Docker images. Tags are a simple way to organize
and retrieve images.
Best Practices for Tagging:
1. Use Descriptive Tags: For example, instead of using generic tags like latest, use
more specific tags such as version numbers (v1.0, v2.0), or environment-specific tags
(dev, prod).
Example:
docker build -t my-nginx-image:v1.0 .
docker build -t my-nginx-image:latest .
○
2. Semantic Versioning: For production-ready images, use semantic versioning (e.g.,
v1.0.0).
○ This helps track updates and changes systematically.
3. Use Latest with Caution: The latest tag refers to the most recent image build, which
can change unexpectedly. For production environments, it's safer to use version-specific
tags.
Push Images to DockerHub:
Tag your image:
docker tag my-nginx-image:latest username/my-nginx-image:latest
Push it to DockerHub:
docker push username/my-nginx-image:latest
Here’s a sample workflow that pushes your Flask app to Docker Hub with a version tag (you can
set the tag dynamically based on the branch or commit SHA):
—----------------------------------------------------------------------------------------------------------------------------
name: Build and Push to Docker Hub
on:
push:
branches:
- main # Trigger this workflow when pushing to the main branch
workflow_dispatch: # Allows manual triggering of the workflow
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Log in to Docker Hub
run: echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{
secrets.DOCKER_USERNAME }}" --password-stdin
- name: Build and Push Docker Image
run: |
# Define the image tag based on the current branch or commit SHA
IMAGE_TAG=${{ [Link] }}
# Build and tag the image
docker build -t ${{ secrets.DOCKER_USERNAME }}/flask-app:${IMAGE_TAG} .
# Push the tagged image to Docker Hub
docker push ${{ secrets.DOCKER_USERNAME }}/flask-app:${IMAGE_TAG}
# Optionally, tag this image as "latest" and push it
docker tag ${{ secrets.DOCKER_USERNAME }}/flask-app:${IMAGE_TAG} ${{
secrets.DOCKER_USERNAME }}/flask-app:latest
docker push ${{ secrets.DOCKER_USERNAME }}/flask-app:latest
—----------------------------------------------------------------------------------------------------------------------------
Triggers (on: push):
● This workflow triggers automatically when you push changes to the main branch, but
you can change it to any branch you prefer.
● You can also manually trigger the workflow with the workflow_dispatch event.
Steps:
● Checkout Code: Uses the actions/checkout action to pull the latest code from the
repository.
● Set up Docker Buildx: This action ensures you can build Docker images.
● Log in to Docker Hub: The workflow logs into Docker Hub using the credentials stored
in GitHub secrets.
● Build and Push Docker Image:
○ The image is tagged using [Link] (the commit SHA), which uniquely
identifies each build.
○ After the image is built and tagged, it is pushed to your Docker Hub account.
○ Optionally, the image is also tagged as latest and pushed.
Tag by Branch Name:
IMAGE_TAG=${{ github.ref_name }}
Tag by Semantic Version:
You can set the IMAGE_TAG to a version from a file like [Link] or a release tag:
- name: Set image tag
run: |
IMAGE_TAG=$(cat [Link])
echo "IMAGE_TAG=${IMAGE_TAG}" >> $GITHUB_ENV
What is SHA?
SHA stands for Secure Hash Algorithm. In the context of Git, SHA refers specifically to the
SHA-1 hash that Git uses to uniquely identify commits in the repository. Here’s what you need to
know:
● Unique Identifier: Each commit in a Git repository is identified by a SHA-1 hash, which
is a 40-character string that serves as a unique identifier. For example, a commit SHA
might look like e8a6c2e3fb68d09b68c7ef6d5931eabc9de14567.
● Deterministic: The SHA is calculated based on the contents of the commit, including the
files and the commit message. If you change anything, the SHA will also change,
ensuring that every commit is unique.
● Referencing Commits: You can use the SHA to reference specific commits in
commands, issues, and pull requests, allowing for precise tracking and retrieval of code
states.
Using SHA for tagging your Docker images helps you maintain a clear and traceable history of
what code was deployed at any given time. Let me know if you have more questions about this!
Timestamps for tagging
Advantages of Using Timestamps:
1. Simplicity:
○ Timestamps are straightforward and easy to implement. You can generate them
automatically based on the time of the commit or build.
2. Chronological Order:
○ Timestamps inherently provide a chronological order, making it easy to see when
changes were made and in what sequence.
3. Unique Identifiers:
○ If used correctly (e.g., including seconds or milliseconds), timestamps can be
unique for each build, reducing the chances of collision.
4. Easy to Read:
○ Timestamps can be human-readable, making it simple for developers to
understand when a particular version was created.
5. Automated Versioning:
○ Integrating timestamps into automated CI/CD pipelines can simplify the process
of tagging and tracking versions without manual intervention.
Disadvantages of Using Timestamps:
1. Lack of Context:
○ Timestamps don’t provide information about the content or significance of
changes, making it harder to determine the purpose of a specific version without
additional context (like commit messages).
2. Potential for Ambiguity:
○ If multiple builds happen in quick succession, timestamps can become
ambiguous. For example, if two commits are made within the same second, they
could end up with the same timestamp.
3. Not Semantic:
○ Timestamps don’t follow semantic versioning principles, making it harder to
understand the nature of changes (e.g., major, minor, or patch).
4. Difficulties in Rollback:
○ Rolling back to a specific version based on timestamps can be challenging if the
tagging doesn’t clearly represent the state of the code at that time.
5. Time Zone Issues:
○ Depending on how timestamps are generated, discrepancies can occur due to
different time zones, potentially causing confusion in distributed teams.
SHORT QUESTIONS
Code Sharing using GitHub
1. What is GitHub used for?
A platform for hosting and sharing code, enabling collaboration and version control.
2. How do you share code using GitHub?
By pushing commits to a repository, others can clone or fork the repository to access the
code.
PR and Merging
3. What is a pull request (PR)?
A request to merge code changes from one branch into another, typically for review.
4. What does merging do in GitHub?
It integrates changes from one branch into another (usually from feature branch to main
branch).
Code Branching and Features/Bug Fixes
5. What is a branch in GitHub?
A separate workspace in a project for making changes without affecting the main
codebase.
6. Why create a new branch for bug fixes or features?
It allows isolated development of new features or bug fixes without disturbing the stable
codebase.
GitHub Actions for Code Integration and Deployment
7. What is GitHub Actions?
A CI/CD tool for automating workflows like building, testing, and deploying code in
GitHub.
8. How can GitHub Actions be used for code deployment?
By setting up workflows in YAML files that automate the process of deploying code to
production environments.
Docker for CD
9. What is Docker used for in CD (Continuous Deployment)?
It packages applications into containers, ensuring consistency across different
environments.
10. How does Docker simplify deployment?
By creating containerized environments that work the same locally and in production.
YAML Scripts for Dockerfile and GitHub Actions
11. What is YAML used for in GitHub Actions?
YAML defines workflows for CI/CD pipelines like building, testing, and deploying code.
12. How is a Dockerfile integrated into GitHub Actions?
YAML scripts in GitHub Actions can build Docker images from Dockerfiles and deploy
them as part of the workflow.