Dynatrace Autonomous Cloud

Management

GKE, Jenkins, Ansible Tower

Key blue print – unbreakable continuous delivery model

• Automate operations (self-healing) – auto-mitigate bad

SELF
deployments in production
HEALING

SHIFT RIGHT / • Automate deployment (shift-right) – push “monitoring-as-code”

DEPLOYMENT for auto-validation and auto-alerting
SPEED

• Automate quality (shift-left) – automate the pipeline and stop

SHIFT LEFT /
QUALITY GATES bad code changes before they reach prod

MULTI-PLATFORM MONITORING • Automate monitoring – use monitoring strategically as feature

of the end-to-end pipeline

Confidential 2
What we will accomplish

• Deploying the SockShop application to our GKE cluster

• Setting up Jenkins on K8s in GKE

• Import a Jenkins Pipeline

• Setup Dynatrace Integration with Jenkins

• Using Jenkins Plugin and Monspec

• Deploy different code builds and test their performance

• Fail pipeline if code is not performant based on monspec

• Create a self-healing application using Dynatrace and Ansible Tower

Confidential 3
 Workshop Setup

SampleApp
GitHub Repository
po
Red
loa
wn
Do

#1 Create Dynatrace Tenant

#2

#3 Deploy Jenkins
#4 Deploy Ansible Tower

K8s
Every Team
On a laptop (or via CloudShell)

Jenkins

GKE Instances Ansible Tower

Confidential 4
Sock Shop Application

• Based on: [Link]

• Opensource microservice sample app

• Polyglot
• Java Spring Boot
• Go
• [Link]
• RabbitMQ (order queuing)
• MongoDB, MySQL (data stores)

Confidential 5
Sock Shop architecture

Confidential 6
Jenkins

• The Worlds most advanced BASH terminal

• In most cases it’s fine to deploy to K8s via simple kubectl

commands in k8s
• Most of deployments probably done this way
• Spoiler Alert: We’re going to be doing this today

• No true “easy” way to determine a failure or if app is up and

running
$kubectl wait –n dev deploy/myapp –-for condition=available

This won’t catch pods that didn’t actually come online

Confidential 7
What our Pipeline will look like (hopefully)

Confidential 8
Let’s Start the fun

Confidential 9
Setup our GKE Cluster

• Login to your GCP Account

Confidential 10
Setup our GKE Cluster

• In the search box, type “Kubernetes” and select

“Kubernetes Engine” from the search results

• This may take a few minutes if this is the first time you

are attempting to launch Kubernetes in this account.

Confidential 11
Setup our GKE Cluster

• After the initial APIs are setup, your Kubernetes Engine screen should look like this (unless you
already have an existing cluster setup)

Confidential 12
Setup our GKE Cluster – Launching Cloudshell

• To launch and interact with our Kubernetes cluster we use Cloudshell

• Cloudshell provides command-line access to your cloud resources directly from your browser
• No need for a separate bastion host
• Required tools pre-installed (git, docker, kubectl, etc…)
• Persistent storage

Confidential 13
Setup our GKE Cluster – Cloning our repository

• In Cloudshell, clone/download the following GitHub Repository:

[Link]

• This will create a folder called dtacmworkshop

• This will contain the scripts we will use for the remainder of the workshop

Confidential 14
Dynatrace Credentials – Tenant ID & Environment ID

• In order to proceed further we need to make note of our Dynatrace Tenant/Environment and
API/PaaS Tokens
• For the Tenant ID, you can find it in the first part of your URL
• [Link] – SaaS Deployments
• For example, for [Link] , Tenant ID=jwx05250
• [Link]
• For example, for [Link] Tenant ID=abc123

• The Environment ID is only applicable for Dynatrace Managed Deployments. You can find this
value in the second half of your URL
• [Link]
• For example, for [Link] Environment ID=1234-5678
Confidential 15
Dynatrace Credentials – API Token

• Go in Settings -> Integration -> Dynatrace API

1. Click on Generate Token
2. Enter a name for your token (e.g. GKE)
3. Don't forget to click on the Save button

Confidential 16
Dynatrace Credentials – PaaS Token

• Go in Settings -> Integration -> Platform as a Service

1. Either copy the existing InstallerDownload token or click on Generate Token
2. Enter a name for your token (e.g. GKE), click Save

Confidential 17
Creating our Credentials

• We now need to store a local copy of our Dynatrace Tenant and API info for use when creating our
cluster

• Navigate to the directory 1-Credentials and execute the script [Link]

• This will ask for your Dynatrace Tenant/Environment information as well as your API and PaaS tokens
• These values can be obtained from your Dynatrace tenant (see previous slides)
• Once you have entered the values and confirmed they are correct, we can move on to creating our cluster

Confidential 18
Setup our GKE Cluster

• Back in your GCP account, launch a Cloudshell session

• Navigate to the directory 2-CreateCluster

• Execute the script [Link] and confirm that your credentials are correct

This step will take about 10 minutes to complete

Confidential 19
Setup our GKE Cluster

• The [Link] script will do the following:

• Create a new GKE cluster
• Deploy the Dynatrace OneAgent Operator to the cluster
• Deploy the SockShop application to our cluster
• Including creating the required Namespaces, storage objects and LoadBalancers

Confidential 20
Validate our Cluster setup

• After about 10 minutes since running the [Link] script, let’s verify that everything is up and
running
• Let’s check and make sure our Deployments are Ready, run the following command in Cloudshell:
• kubectl get svc –n dev -o wide

• Note: This can also be validated in the GKE Console by selecting “Workloads” from the menu on
the left side of the screen

Confidential 21
Validate our Cluster setup

• Lets ensure that the SockShop application services are also created by running the following command:
• $kubectl get svc –n dev -o wide
• Our Carts and Front-end services should have External-IP addresses assigned
• Note: This can also be validated in the GKE Console by selecting “Services & Ingress” from the menu
on the left side of the screen

Confidential 22
Validate our Cluster setup

• Let’s launch our application in a web browser by copying and pasting the External-IP and Port of
the front-end service. In the previous example it would be: [Link]
• The SockShop homepage should appear in

your browser

Confidential 23
Clean up script placeholder

• In Terminal/Cloudshell execute the script: [Link]

• This will remove the dev, production and istio-system namespaces (and their associated objects) so
we have a clean cluster

• After this completes, we will redeploy the sock shop application in the dev namespace. We only
need to execute the script: [Link] and then wait a few minutes

• Then check if the SockShop app is back up and running

$kubectl get svc –n dev -o wide

Confidential 24
Validate Dynatrace

• Now that we know our application is deployed and running successfully, we can validate that
everything is being displayed in Dynatrace
• Navigate back to Dynatrace and you should see data for Host, Process and Services

Confidential 25
Deploying Jenkins

Confidential 26
Deploy Jenkins

• In Terminal/Cloudshell execute the script: [Link] in the directory 3-DeployJenkins

• This will deploy and startup Jenkins, import our pipeline and output the credentials

Note: You should change the default password!

Confidential 27
Jenkins Setup

• After Jenkins has successfully deployed, navigate to the Jenkins URL and login with the
previously provided username and password (again, recommend you change the password 

Confidential 28
Run our first pipeline

• On the Jenkins main screen we should see the “DeploySockShop” pipeline, select the pipeline by
clicking it

Click Build with Parameters

Select Build “One” and Build

Confidential 29
If all goes well…

• The pipeline should run successfully (Note: this will take about 10 min)

Confidential 30
Dynatrace Integration

Confidential 31
Dynatrace Integration

• GitHub Repo and Example:

[Link]
• Integration via the Dynatrace APIs (Timeseries and Events)

• Based on a Performance Signature of a release (monspec)

Confidential 32
Install Jenkins Plugin

• Navigate to Manage Jenkins -> Manage Plugins page and switch to the Available tab. Search for
the "Dynatrace" keyword, select the 3 options below and click “Install without restart”.

Confidential 33
Dynatrace API

• A Dynatrace API Token is needed for the Integration (Settings -> Integration -> Dynatrace API)
• Only need the default top 3 permissions (This can be the same API Token as created in earlier steps)

Confidential 34
 Jenkins Configuration
• Navigate to Manage Jenkins -> Configure System -> Performance Signature: Dynatrace SaaS/Managed

• Enter name of the

• Server (ex. Dynatrace Server)
• Your Tenant URL ([Link] – SaaS, ([Link] -
Managed
• API Token
• Note: You many need to Save and exit this screen and come back for the “Add” dropdown to works, thanks to a bug in
Jenkins

• Test the Connection and Save

Confidential 35
Adding Events to our Pipeline

• There are three main events we will be adding to our Deployment Pipeline:

• createDynatraceDeploymentEvent – Sends the Deployment Event to Dynatrace

• recordDynatrace Session – Sets start/end time for when Perf Test beings

• perfSigDynatraceReports – Queries all the metrics during the perf test timeframe and compares against
the spec file. If metrics violate the spec file, the pipeline stage is failed

Confidential 36
Adding Events to our Pipeline

• In the DeploySockShop pipeline, click “Configure” and scroll down to the Pipeline script

Confidential 37
Adding Events to our Pipeline – Deployment Events

• First we want to add a Deployment Event to our Pipeline so every time we deploy our build to
staging, Dynatrace will know about it

Confidential 38
Adding Events to our Pipeline – Record Perf Test

• Next we want to wrap our Loadtest stage with the recordDynatraceSession function, so we know when
tests start and stop

Confidential 39
Adding Events to our Pipeline – Compare results to monspec

• After the load test is completed, we want to validate the test against our monspec file (see next slide), to
determine if our build meets quality or not
• Add the below stage (“Dynatrace Quality Gate”) to the very end of our pipeline

Confidential 40
Sample Monspec File

"spec_version": "2.0",

"timeseries": [{

"timeseriesId": "[Link]:[Link]",

"aggregation": "avg",

"tags": “[Kubernetes]app:carts,[Kubernetes]stage:dev",

"upperWarning": 1000,

"upperSevere": 3000

}
Confidential 41
Adding Events to our Pipeline

• Once the events are added to our Pipeline we can apply and save

• Note: There is a sample Pipeline with the events already added in the root of the repository titled
“[Link]”

• You can use this as a reference when editing your pipeline

• Note: You will need to make sure you change the name of the Dynatrace Environment to the same thing as you
named your connection earlier. This file uses the name “Dynatrace Server” as the Dynatrace Environment
Name.

Confidential 42
Let’s test our Integration
Deploy Build Two

• After updating and saving our Pipeline to include the Dynatrace Performance Events, let’s re-run
our pipeline with build Two

Select Build “Two” and Build

Confidential 44
Now our build should fail

Confidential 45
Deploy Ansible Tower

• If we redeploy using Build #1, this build should pass our Dynatrace Quality Gate

Select Build “One” and Build

Confidential 46
Auto-remediation with Ansible Tower

Confidential 47
Ansible and Ansible Tower

• Runbook – definition: a compilation of routine procedures and

operations that the system operator (administrator) carries out

• Ansible
• Simple automation language + automation engine

• Human readable (yaml)

• Ansible Tower
• Web UI
• Management of runbooks
• Powerful API for automation Confidential 48
Promotional campaign

• We’ll be implementing a promotional

campaign for our socks

• This promotion will be randomly available to

customers

• The carts service has an endpoint allowing us

to provide the percentage of users that will
benefit from the promotional gift

• The problem is that the promotion

implementation is defective 

• Nobody knows about it yet and this could

potentially have disastrous effect on the brand
and business reputation

Confidential 49
Runbook automation workflow

Trigger promotional campaign

with Promotion Rate set to 50 %
Set Promotion Rate to 50 %

Send custom configuration event

Monitors Sockshop & detects

increase of failure rate
Sends problem notification

Set Promotion Rate to 0 %

Send custom configuration event

Send comment to problem

Confidential 50
Deploy Ansible Tower

• In Terminal/Cloudshell execute the script: [Link] in the directory 4-DeployTower

• This will deploy and startup Ansible Tower. This will also configure Tower, import projects,
inventories, credentials (DT API token) and playbook templates.
• Take a note of the Ansible Job URL displayed by the script. You will need it for a next step.

Confidential 51
Log in the Ansible Tower console

• Use the job template URL you copied earlier,

without the path, and copy it in your browser to
load the Ansible Tower console
• For example, if the job template URL was :
[Link]
then the console URL is : [Link]

• You will be prompted to provide a license file and

accept the license agreement
• If you don’t already have your license file, click
the button to request a free trial license from Red
Hat (delivered via email)

• Log in Ansible Tower

• username : admin
Confidential 52
• password : dynatrace
Configure Dynatrace Problem Notification

• In the Dynatrace UI, navigate to Settings -> Integration -> Problem Notification -> Ansible Tower

Confidential 53
Configure Ansible Tower integration

• Enter the Ansible Tower job template

URL you were provided after the
deployment
• Enter the credentials to access Ansible
• username : admin
• password : Dynatrace

• Click the Sent test notification button to

validate your configuration
• on success, a green confirmation
message will be displayed

• Save your configuration

Confidential 54
 Adjust Anomaly Detection

Both problem and anomaly detection in Dynatrace leverage AI technology. This means that the AI learns how each
and every microservice behaves and baselines them. Therefore, in a demo scenario like we have right now, we have
to override the AI engine with user-defined values to allow the creation of problems due to an artificial increase of a
failure rate. (Please note: if we would have the application running and simulate end-user traffic for a couple of
hours/days there would be no need for this step.)

In your Dynatrace tenant, navigate to “Transaction & services” and filter by:
[Kubernetes]app:carts and [Kubernetes]environment:production

Confidential 55
Adjust Anomaly Detection (Cont.)

• Click on the ItemsController and then on the three dots ( … ) next to the service name. Click on
Edit

Confidential 56
Adjust Anomaly Detection (Cont.)

On the next screen, edit the anomaly detection settings as seen in the following screenshot. - Global
anomaly detection has to be turned off - Detect increases in failure rate using fixed thresholds -
Alert if 0 % custom failure rate threshold is exceed during any 5-minute period. - Sensitivity: High

Confidential 57
Launch Campaign playbook

• Navigate back to the Ansible Tower UI

• From the side menu, navigate to Resources -> Templates

• Click on the rocket icon to launch the start-campaign playbook

• Hit Next on the prompt popup window and then Launch

• As the playbook runs, the output will be displayed. Validate that the recap shows OK=2

Confidential 58
Observe the promotional campaign

• In the Dynatrace console, navigate to Transactions & Services and drill-down into the production
ItemsController service
• Tip : filter by tag : [Kubernetes]stage:prod to only display services in production

Confidential 59
Observe the promotional campaign

• Look at the Service events.

• The Start Campaign playbook have

notified Dynatrace when the promotional
rate was changed to 50%

Confidential 60
Observe the promotional campaign

• You should see the Failure Rate

increasing, eventually leading to
Dynatrace detecting a Problem
• You might need to refresh your browser
a few times

Confidential 61
Observe the auto-remediation playbook actions

• Go back to the Service

view
• The comments section will
show the remediation
actions taken by Ansible
Tower

Confidential 62
Observe the auto-remediation playbook actions

• Drill-down in the Problem.

• You will see a new

configuration change event
reported by Ansible Tower
• The promotional rate has
been set back to 0% to
remediate to the transaction
failures

Confidential 63
Observe the auto-remediation playbook actions

• Jobs executed in Ansible Tower

• start-campaign (set rate to 50%)
• remediation
• push comment to Dynatrace Problem
• retrieve problem details
• launch remediation action related to
problem context
• update Dynatrace Problem
• stop-campaign (set rate to 0%)

Confidential 64
That’s it! Problem solved automatically

• Look at the Service Failure Rate. It has returned to 0%

Confidential 65
Runbook automation workflow recap

Trigger promotional campaign

with Promotion Rate set to 50 %
Set Promotion Rate to 50 %

Send custom configuration event

Monitors Sockshop & detects

increase of failure rate
Sends problem notification

Set Promotion Rate to 0 %

Send custom configuration event

Send comment to problem

Confidential 66
Clean up

Confidential 67
Clean up

• If you want to keep the cluster running but:

• remove Jenkins : execute the script [Link] contained in the utils directory
• remove Ansible Tower : execute the script [Link] container in the utils directory

• If you want to delete everything, feel free to delete the GKE cluster via the GKE
Console

Confidential 68
[Link]

Confidential 69