advdevops journal
advdevops journal
2. Problem analysis: Identify, formulate, review research literature, and analyze complex
engineering problems reaching substantiated conclusions using first principles of mathematics,
natural sciences, and engineering sciences.
5. Modern tool usage: Create, select, and apply appropriate techniques, resources, and
modern engineering and IT tools including prediction and modeling to complex engineering
activities with an understanding of the limitations.
6. The engineer and society: Apply reasoning informed by the contextual knowledge to
assess societal, health, safety, legal and cultural issues and the consequent responsibilities
relevant to the professional engineering practice.
8. Ethics: Apply ethical principles and commit to professional ethics and responsibilities
and norms of the engineering practice.
1
11. Project management and finance: Demonstrate knowledge and understanding of the
engineering and management principles and apply these to one’s own work, as a member and
leader in a team, to manage projects and in multidisciplinary environments.
12. Life-long learning: Recognize the need for, and have the preparation and ability to
engage in independent and life-long learning in the broadest context of technological change.
Datta Meghe College of Engineering, Airoli, Navi Mumbai
Department Vision : To develop and foster students for successful careers in the dynamic
field of Information Technology.
Department Mission :
M1: To create and disseminate knowledge through research, teaching & learning and to
enhance society in meaningful and sustainable ways.
To impart a suitable environment for students and staff to showcase innovative ideas
in the field of IT.
M2:
2
Program Educational Objectives (PEO)
PEO 2
Possess the ability to function ethically and responsibly with good cultural
values and integrity to apply the best principles and practices of Information
Technology towards the society.
PEO 4
PSO3
3
Datta Meghe College of Engineering, Airoli
Department of Information Technology
Course Name: Advanced Devops Lab (R-19)
Course Code: ITL504 Year of
Study:2021 Semester: V
Course Outcomes
ITL504.1 To understand the fundamentals of Cloud Computing and be fully proficient
with Cloud based DevOps solution deployment options to meet your business
requirements
ITL504.2 To deploy single and multiple container applications and manage application
deployments with rollouts in Kubernetes
ITL504.3 To apply best practices for managing infrastructure as code environments and
use terraform to define and deploy cloud infrastructure
ITL504.4 To identify and remediate application vulnerabilities earlier and help integrate
security in the development process using SAST Technique
ITL504.5 To use Continuous Monitoring Tools to resolve any system errors (low memory,
unreachable server etc.) before they have any negative impact on the business
productivity
ITL504.6 To engineer a composition of nano services using AWS Lambda and Step
Functions with the Serverless Framework
4
DATTA MEGHE COLLEGE OF ENGINEERING
AIROLI, NAVI MUMBAI - 400708
CERTIFICATE
This is to certify that Mr. Rishi Kamal Tank of IT Branch Class TE-IT Roll No. 58
Date 21.10.21
Examined on
5
Datta Meghe DEPARTMENT OF INFORMATION TECHNOLOGY
6
To perform Port, Service monitoring, 83 16.09.21
Windows/Linux server monitoring using Nagios
10 LO1,LO5
Assignment No1 LO1-LO6 122 27.7.21
7
DATTA MEGHE COLLEGE OF ENGINEERING,
AIROLI
Advance Devops
EXPERIMENT 01
Date: 15/07/21
8
Experiment 01
Aim
To understand the benefits of Cloud Infrastructure and Setup AWS Cloud9 IDE, Launch AWS
Cloud9 IDE and Perform Collaboration Demonstration.
Theory
AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write,
run, and debug your code with just a browser. It includes a code editor, debugger, and
terminal. Cloud9 comes prepackaged with essential tools for popular programming languages,
including JavaScript, Python, PHP, and more, so you don’t need to install files or configure your
development machine to start new projects. Since your Cloud9 IDE is cloud-based, you can
work on your projects from your office, home, or anywhere using an internet-connected
machine. Cloud9 also provides a seamless experience for developing serverless applications
enabling you to easily define resources, debug, and switch between local and remote
execution of serverless applications. With Cloud9, you can quickly share your development
environment with your team, enabling you to pair programs and track each other's inputs in
real time.
Benefits:
CODE WITH JUST A BROWSER
AWS Cloud9 gives you the flexibility to run your development environment on a managed
Amazon EC2 instance or any existing Linux server that supports SSH. This means that you can
write, run, and debug applications with just a browser, without needing to install or maintain
a local IDE. The Cloud9 code editor and integrated debugger include helpful, time-saving
features such as code hinting, code completion, and step-through debugging. The Cloud9
terminal provides a browser- based shell experience enabling you to install additional
software, do a git push, or enter commands.
9
AWS Cloud9 makes collaborating on code easy. You can share your development environment
with your team in just a few clicks and pair programs together. While collaborating, your team
members can see each other in real time, and instantly chat with one another from within the
IDE.
AWS Cloud9 makes it easy to write, run, and debug serverless applications. It preconfigures
the development environment with all the SDKs, libraries, and plug-ins needed for serverless
development. Cloud9 also provides an environment for locally testing and debugging AWS
Lambda functions. This allows you to iterate on your code directly, saving you time and
improving the quality of your code.
AWS Cloud9 comes with a terminal that includes sudo privileges to the managed Amazon EC2
instance that is hosting your development environment and a preauthenticated AWS
Command Line Interface. This makes it easy for you to quickly run commands and directly
access AWS services
AWS Cloud9 makes it easy for you to start new projects. Cloud9’s development environment
comes prepackaged with tooling for over 40 programming languages, including [Link],
JavaScript, Python, PHP, Ruby, Go, and C++. This enables you to start writing code for popular
application stacks within minutes by eliminating the need to install or configure files, SDKs,
and plug-ins for your development machine. Because Cloud9 is cloud-based, you can easily
maintain multiple development environments to isolate your project’s resources.
Steps:
10
3. Click on Create Environment :
11
4. Provide the name for the Environment (WebAppIDE) and click on next.
12
13
6. Review the Environment name and Settings and click on Create Environment
14
It will take a few minutes to create an aws instance for your Cloud 9 Environment.
7. Till that time open IAM Identity and Access Management in order to Add user In another
tab.
15
8. Add the user provided manual password if you want and click on Next permission tab.
16
9. Click on Create group
11. After that group is created click on next if u want to provide tag else click on Review for
10. Provide group name and click on create group. user settings and click on create user as
shown in fig.
12. Now close that window and Navigate to user Groups from the left pane in IAM.
17
13. click on your group name which you have created and navigate to permission tab as shown
15. Now we move towards our cloud9 IDE Environment tab it shows as shown
14. Now click on Add permission and select Attach Policy after that search for Cloud9 related
policy and select Aws cloud9 Environment Member policy
16. If you check at bottom side Cloud9 IDE also giving you and aws CLI for command
operations:
as we here checked git version, iam user details
18
17. Now we will setup collaborative environment Click on File you can create new file or choose
from template, here m opting html file to collaborate.
19
19. Now in order to share this file to collaborate with other members of your team click on
Share option on Right Pane and username which you created in IAM before into
Invite members and enable permission as RW (Read and Write) and click on Done.
Click OK for Security warning.
[Link] Open your Browsers Incognito Window and login with IAM user which you configured
before.
20
21. After Successful login with IAM user open Cloud9 service from dashboard services and click
on shared with you environment to collaborate.
[Link] on Open IDE you will same interface as your other member have to collaborate in real
time, also you all within team can do group chats as shown below:
21
[Link] can also explore settings where you can update permissions of your teammates as from
RW to R only or you can remove users too.
22
Conclusion
Hence, we understood the benefits of Cloud Infrastructure and Setup AWS Cloud9 IDE, Launch
AWS Cloud9 IDE and Performed Collaboration Demonstration.
23
DATTA MEGHE COLLEGE OF ENGINEERING,
AIROLI
Advance Devops
EXPERIMENT 02
Date: 22/07/21
1
Experiment 02
AIM
To Build Your Application using AWS CodeBuildandDeploy on S3 / SEBS using AWS CodePipeline,
deploy SampleApplication on EC2 instance using AWS CodeDeploy.
THEORY
Continuous deployment allows you to deploy revisions to production environment
automatically without explicit approval from a developer, making the entire software release
process automated. You will create the pipeline using AWSCodePipeline, a service that builds,
tests, and deploys your code every time there is a code change. You will use your GitHub
account, an Amazon Simple Storage Service (S3) bucket, or an AWS CodeCommit repository as
the source location for the sample app’s code. You will also use AWS Elastic Beanstalk As the
deployment target for the sample app. Your completed pipeline will be able to detect changes
made to the source repository containing the sample app and then automatically update your
live sample app.
Step1: Create a deployment environment Your continuous deployment pipeline will need a
target environment containing virtual servers, or Amazon EC2 instances, where it will deploy
sample code. You will prepare this environment before creating the pipeline. 1) To simplify the
process of setting up and configuring EC2 instances for this tutorial, you will spin up a sample
environment using AWS Elastic Beanstalk. Elastic Beanstalk Lets you easily host web applications
without needing to launch, configure, or operate virtual servers on your own.
It automatically provisions and operates the infrastructure(e.g. virtual servers, load balancers,
etc.) and provides the application stack (e.g. OS, language and framework, web and application
server, etc.) for you.
2) Name your web app and choose PHP from the drop-down menu(or any other language you
are interested in) and then click Create Application.
2
anAmazon S3 bucket, Amazon CloudWatch alarms, and a domain name for your application.
Note: This will take several minutes to complete.
Step2: Get a copy of the sample code In this step, you will retrieve a copy of the sample app’s
code and choose a source to host the code.
3
The pipeline takes code from the source and then performs actions on it. You can use one of
three options as your source: aGitHub repository, an Amazon S3 bucket, or an AWS CodeCommit
repository. Select your preference and follow the steps below:
a. If you plan to use Amazon S3 as your source, you will retrieve the sample code from the
AWS GitHub repository, saveit toyour computer, and upload it to an Amazon S3 bucket.
• Visit our GitHub repository containing the sample codeat
[Link] codedeploy-linux-2.0 • Click
the dist folder.
b. Save the source files to your computer:
• Click the file named aws-codepipeline-s3-aws- codedeploy_linux.zip
• Click View Raw.
• Save the sample file to your local computer. c. open the Amazon S3 console and create
your AmazonS3bucket:
• Click Create Bucket
• Bucket Name: type a unique name for your bucket, such as aws codepipeline-demo
bucket- variables. All bucket names in Amazon S3 must be unique, souse one of your own,
not one with the name shown in example.
• Region: In the drop-down, select the region where you will create your pipeline, such as
apSouth-1
• Click Create.
[Link] console displays the newly created bucket, whichis empty.
• Click Properties.
• Expand Versioning and select Enable Versioning. Whenversioning is enabled, Amazon S3
saves every version of every object in the bucket. e. You will now upload the sample code
to the AmazonS3bucket:
• Click Upload.
• Follow the on-screen directions to upload the .zip file containing the sample code you
4
downloaded from GitHub
S te p 3 : C re a te y o u r P ip e lin e
In this step, you will create and configure a simple pipeline with two actions: source and deploy.
5
You will provide CodePipelinewith the locations of your source repository and deployment
environment. A true continuous deployment pipeline requires build stage, where code is
compiled and unit tested. CodePipeline lets you plug your preferred build provider intoyour
pipeline. However, in this we will skip the build stage. Goto Pipeline again and create it
S te p 4 : D e p lo y S ta g e
• Deployment provider:Click AWS Elastic Beanstalk. • Application name: MYEBS.
• Environment name: Click Myebs-env.
6
pipeline perform each action. To verify your pipeline ran successfully, monitor the progress of
the pipeline as it moves through each stage. The status of each stage will change fromNo
execution yet to In Progress, and then to either Succeeded or
Failed. The Pipeline should complete the first run within a few minutes.
Now go to your EBS environment and click on the URL to view the sample website you deployed.
7
You have successfully created an automated software releasepipeline using AWS CodePipeline!
Using CodePipeline, you created a pipeline that uses GitHub, Amazon S3, or AWS CodeCommit
as the source location for application code and then deploys the code to an AmazonEC2instance
managed by AWS Elastic Beanstalk.
Step 5: Commit a change and then update your app In this step, you will revise the sample
code and commit the change to your repository. CodePipeline will detect your updated sample
code and then automatically initiate deploying it to your EC2 instance via Elastic Beanstalk.
Note that the sample web page you deployed refers toAWSCodeDeploy, a service that
automates code deployments. InCodePipeline, CodeDeploy is an alternative to using
ElasticBeanstalk for deployment actions. Let’s update the sample code so that it correctly states
that you deployed the sample using Elastic Beanstalk. a. Visit your own copy of the repository
that you forked GitHub.
• Open [Link] •
Select the Edit icon
b. Update the webpage by copying and pasting the following text on line 30:
c. Commit the change to your repository.
d. Return to your pipeline in the CodePipeline console. In Afew Minutes, you should see the
Source change to blue, indicating that the pipeline has detected the changes you made
toyour source repository. Once this occurs, it will automatically move the updated code
to Elastic Beanstalk.
• After the pipeline status displays Succeeded, in the status area for the Beta stage, click AWS
Elastic Beanstalk. e. The AWS Elastic Beanstalk console opens with the details of the deployment.
Select the environment you created earlier. And click the URL again from EBS environment again.
8
Step 6: Clean up your resources
To avoid future charges, you will delete all the resources you launched throughout this tutorial,
which includes the pipeline, the Elastic Beanstalk application, and the source you set up host the
code.
a. First, you will delete your pipeline:
• In the pipeline view, click Edit.
• Click Delete.
• Type in the name of your pipeline and click Delete. b. Second, delete your Elastic Beanstalk
application: • Visit the Elastic Beanstalk console.
• Click Actions.
• Then click Terminate Environment.
You have successfully created an automated software release pipeline using AWS CodePipeline!
Using CodePipeline, you created a pipeline that uses GitHub, Amazon S3, or AWS CodeCommit
as the source location for application code and then deploys the code to an AmazonEC2instance
managed by AWS Elastic Beanstalk. Your pipeline will automatically deploy your code every time
there is a code change
CONCLUSION
Hence , building Application using AWS CodeBuild and deploying on S3 / SEBS using AWS
CodePipeline, deploy Sample Application on EC2 instance using AWS CodeDeploy is successfully
implemented.
9
DATTA MEGHE COLLEGE OF ENGINEERING AIROLI
Advance Devops
EXPERIMENT 03
Date: 22/07/21
Experiment 03
Theory:
Kubernotes:
Kubernetes is a portable, extensible, open-source platform for managing containerized workloads
and services, that facilitates both declarative configuration and automation.
Nodes:
A Pod always runs on a Node.
A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine,
depending on the cluster.
A Node can have multiple pods, and the Kubernetes control plane automatically handles scheduling
the pods across the Nodes in the cluster.
The control plane's automatic scheduling takes into account the available resources on each Node.
Controller:
A controller tracks at least one Kubernetes resource type.
These objects have a spec field that represents the desired state.
The controllers for that resource are responsible for making the current state come closer to that
desired state.
Cloud Controller Merge:
Cloud infrastructure technologies let you run Kubernetes on public, private, and hybrid clouds.
Kubernetes believes in automated, API-driven infrastructure without tight coupling between
components.
The cloud-controller-manager is a Kubernetes control plane component that embeds cloudspecific
control logic.
The cloud controller manager lets you link your cluster into your cloud provider's API, and
separates out the components that interact with that cloud platform from components that only
interact with your cluster.
Design:
Outputs:
Installing the Kubernetes-cli using chocolatey.
Installing minikube:
Date: 29/07/21
Experiment 04
Aim: To install Kubectl and execute Kubectl commands to manage the
Kubernetes cluster and deploy Your First Kubernetes Application.
Theory:
Kubectl:
The Kubernetes command-line tool, kubectl, allows you to run commands
against Kubernetes clusters. You can use kubectl to deploy applications,
inspect and manage cluster resources, and view logs.
When you create a Deployment, you'll need to specify the container image
for your application and the number of replicas that you want to run.
Outputs:
Installing kubectl using chocolatey:
Deploying first application using Kubernetes:
Checking the deployed application:
Date: 12/08/21
EXPERIMENT NO : 05
Date: 12/08/2021
EXPERIMENT NO : 06
Date: 26/08/2021
EXPERIEMENT NO : 07
THEORY :
SonarQube is a Code Quality Assurance tool that
collects and analyzes source code, and provides
reports for the code quality of your project. It
combines static and dynamic analysis tools and
enables quality to be measured continually over
time. Everything from minor styling choices, to
design errors are inspected and evaluated by
SonarQube. This provides users with a rich
searchable history of the code to analyze where the
code is messing up and determine whether or not it
is styling issues, code defeats, code duplication, lack
of test coverage, or excessively complex code. The
software will analyze source code from different
aspects and drills down the code layer by layer,
moving module level down to the class level, with
each level producing metric values and statistics
that should reveal problematic areas in the source
code that needs improvement.
Sonarqube also ensures code reliability,
Application security, and reduces technical debt
by making your code base clean and maintainable.
Sonarqube also provides support for 27 different
languages, including C, C++, Java, Javascript, PHP,
GO, Python, and much [Link] also
provides Ci/CD integration, and gives feedback
during code review with branch analysis and pull
request decoration.
SonarQube Benefits:
• CI tools do not have a plugin which would make all of
these tools work easily together
• CI tools do not have plugins to provide nice drill-
down features that SonarQube has
• CI Plugins does not talk about overall compliance
value
• CI plugins do not provide managerial perspective
• There is no CI plugin for Design or Architectural
issues
• CI plugins do not provide a dashboard for overall
project quality
SonarQube Setup Step
1: SonarQube Instance
Date: 09/09/2021
EXPERIMENT NO . 8
THEORY :-
SonarQube is a universal tool for static code analysis that has
become more or less the industry standard. Keeping code clean,
simple, and easy to read is also a lot easier with SonarQube.
SonarQube is an open-source platform developed by SonarSource
for continuous inspection of code quality. Sonar does static code
analysis, which provides a detailed report of bugs, code smells,
vulnerabilities, code duplications.
Benefits of SonarQube
1. Sustainability - Reduces complexity, possible vulnerabilities,
and code duplications, optimising the life of applications. 2.
Increase productivity - Reduces the scale, cost of
maintenance, and risk of the application; as such, it removes
the need to spend more time changing the code
3. Quality code - Code quality control is an inseparable part of
the process of software development.
4. Detect Errors - Detects errors in the code and alerts
developers to fix them automatically before submitting them
for output.
5. Increase consistency - Determines where the code criteria
are breached and enhances the quality
Business scaling - No restriction on the number of projects to be
evaluated.
6. Enhance developer skills - Regular feedback on quality
problems helps developers to improve their coding skills.
SonarQube Setup
SonarQube Instance
Before proceeding with the integration, we will setup
SonarQube Instance. Choice of the platform is yours.
}
stage ("Unit Test"){ steps{ sh
"python3 test_basic.py"
}
}
stage ("Python Bandit Security Scan"){
steps{ sh "cat
report/[Link]" sh "sh
run_bandit.sh || true"
sh "ls"
}
}
stage ("Dependency Check with Python Safety"){
steps{ sh "docker run --rm --volume
\$(pwd)
pyupio/safety:latest safety check" sh "docker run --rm --
volume \$(pwd)
pyupio/safety:latest safety check --json > [Link]"
}
}
stage ("Static Analysis with python-taint"){ steps{
sh "docker run --rm --volume \$(pwd)
vickyrajagopal/python-taint-docker pyt ."
}
}
stage ("sonar-publish"){ steps { echo
"===========Performing Sonar
Scan============"
sh "${tool("sonarqube")}/bin/sonar- scanner"
}
}
}
}
Where it will just execute the SonarQube Scanner and collect
the SAST information and Python bandit report in the format of
JSON. Then, it will publish the same in the SonarQube Server. If
you login to the SonarQube and visit the Dashboard, you will
see the Analysis of the project there.
Date: 09/09/2021
EXPERIMENT NO. : 09
1
Advance Devops EXPERIMENT 10
2
Name: Rishi Tank
3
Year: TE
4
Branch: Information Technology
Div.: B
Roll no. : 58
Date: 16/09/2021
Experiment No-10
Aim : To perform Port, Service monitoring, Windows/Linux server monitoring using Nagios.
Theory :
5
To monitor Windows Machines you will need to follow several steps and they are:
6
Install NSClient++ addon on the Windows Machine.
Configure Nagios Server for monitoring Windows Machine.
Add new host and service definitions for Windows machine monitoring.
7
Restart the Nagios Service.
8
To make this guide simple and easier, a few of configuration already done for you in the Nagios
installation.
A check_nt command definition already added to the [Link] file. This definition command is
used by check_nt plugin to monitor Windows services. A windows-server host template already
created in the [Link] file. This template allows you to add new Windows host definitions.
9
$ mkdir /etc/nagios/objects/monitorhosts
10
Create two directories, say linuxhosts and windowshosts in/etc/nagios/objects/monitorhosts/
$ mkdir /etc/nagios/objects/monitorhosts/windowshosts
$ mkdir /etc/nagios/objects/monitorhosts/linuxhosts
11
12
$ cp /etc/nagios/objects/[Link]
/etc/nagios/objects/monitorhosts/windowshosts/[Link]
13
14
$ cp /etc/nagios/objects/[Link]
/etc/nagios/objects/monitorhosts/linuxhosts/[Link]
15
$ nano /etc/nagios/objects/monitorhosts/windowshosts/[Link]
16
$ nano /etc/nagios/objects/monitorhosts/linuxhosts/[Link]
17
18
$ nano /etc/nagios/[Link]
19
Check the Nagios Configuration
20
$ /usr/sbin/nagios -v /etc/nagios/[Link]
21
22
Restart Nagios Service
$ service nagios restart
23
$ sudo yum install nrpe -y
24
25
26
Open nrpe config file
$ nano /etc/nagios/[Link]
27
Log in to your Windows Server and download nsclient++ and install it. You can use the following
28
link to download the nsclient++ for windows.
[Link]
29
Select Generic
30
Select Typical
Enter Nagios Server IP Address in Allowed Hosts and tick mark the modules as
31
mentioned in the below screenshot.
32
Click on Finish
33
[Link] settings
34
Now open the following file as run as administrator in your Windows Server
35
C:\Program Files\NSClient++\[Link]
36
CheckExternalScripts = enabled
37
CheckHelpers = enabled
CheckEventLog = enabled
CheckNSCP = enabled
CheckDisk = enabled
CheckSystem = enabled
38
We are now all done in our Windows Server.
Open Security Group for Windows Server and allow port 5666 and 12489 and
ICMP for Nagios Server IP.
39
Open Security Group for Linux Server and allow port 5666 and ICMP port for
Nagios Server IP.
Note: If your servers are not in the AWS environment, you can allow these ports in
the local firewall of both the servers.
Now your both Linux and Windows Servers are ready to Monitor. You can monitor
your servers using the following URL.
[Link]
Default Username: nagiosadmin
40
Default Password : nagiosadmin
Conclusion:
41
Hence, We successfully performed Port, Service monitoring, Windows/Linux server monitoring
42
using Nagios.
43
DATTA MEGHE COLLEGE OF ENGINEERING, AIROLI
Assignment 1
Assignment 2
Roll no. : 58
Date: 07/09/2021
Q. Define input variable, query data with output and store remote state of
terraform.
Terraform supports a few different variable formats. Depending on the usage, the
variables are generally divided into inputs and outputs.
• The input variables are used to define values that configure your
infrastructure. These values can be used again and again without having to
remember their every occurrence in the event it needs to be updated.
Input variables
Input variables are usually defined by stating a name, type and a default value.
However, the type and default values are not strictly necessary. Terraform can
deduct the type of the variable from the default or input value.
Output variables
Output variables provide a convenient way to get useful information about your
infrastructure. As you might have noticed, much of the server details are calculated
at deployment and only become available afterwards. Using output variables you
can extract any server-specific values including the calculated details.
Apply your configuration
Query Data
Cloud infrastructure, applications, and services emit data, which Terraform can
query and act on using data sources. Terraform uses data sources to fetch
information from cloud provider APIs, such as disk image IDs, or information about
the rest of your infrastructure through the outputs of other Terraform configurations.
Data sources allow you to load data from APIs or other Terraform workspaces. You
can use this data to make your project's configuration more flexible, and to connect
workspaces that manage different parts of your infrastructure. You can also use data
sources to connect and share data between workspaces in Terraform Cloud and
Terraform Enterprise.
Initial configuration
Destroy infrastructure
Store Remote State
Remote backends allow Terraform to use a shared storage space for state data. The
Terraform Cloud remote backend also allows teams to easily version, audit, and
collaborate on infrastructure changes. Terraform Cloud also securely stores
variables, including API tokens and access keys. It provides a safe, stable
environment for long-running Terraform processes.
Login to terraform
Initialize Terraform