0% found this document useful (0 votes)

9 views101 pages

Openstack Ops Lab v2.1

This document provides a comprehensive guide on OpenStack operations, specifically focusing on connecting to an Ubuntu OpenStack Cloud, managing MAAS (Metal as a Service) operations, and performing Juju operations. It includes detailed instructions for SSH connections, creating admin accounts, modifying images, deploying virtual machines, creating VLANs, and backing up MAAS configurations. The document emphasizes the importance of using the provided commands and settings for effective management of cloud resources.

Uploaded by

Soumya IN

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

9 views101 pages

Openstack Ops Lab v2.1

Uploaded by

Soumya IN

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

OpenStack Operations

OpenStack Operations
This material is copyright of Canonical Limited. This material may be used for personal and noncommercial use only.

This documentation is copyright of Canonical Limited. You are welcome to display on your computer, download and print this
documentation or to use the hard copy provided to you for personal, education and non-commercial use only. You must retain copyright,
trademark and other notices unaltered on any copies or printouts you make. Any trademarks, logos an service marks displayed in this
document are property of their owners, whether Canonical or third parties.

This documentation is provided on an "as is" basis, without warranty of any kind, either express or implied. Your use of this documentation
is at your own risk. Canonical disclaims all warranties and liability that may result directly or indirectly from the use of this documentation.

1. Connect to your Ubuntu OpenStack Cloud

The Ubuntu OpenStack Cloud will be provided in the cloud. Here it is explained how to connect to it.

1.1 SSH connection

1.1 Create the SSH Tunnel

On Linux :

Use the terminal:

ssh -D 9999 ubuntu@<your public IP>

Please note the -D 9999 . This is a socks proxy. It is used to access internal UI services of the cloud for, like MAAS and JUJU.

On Windows open Putty :

1. In the Session section, add the public IP address you received on mail from the trainer, port should be 22.

2. On the left side, go to Connection > SSH > Tunnels

3. In Source Port enter 9999 , select Dynamic button, and click Add .

1 / 101
OpenStack Operations

4. Go back to Session section, add a name in Saved Sessions and click Save .

1.2 Set the proxy in browser

Because it's a SOCKS proxy, we need to set it in the browser. Firefox will be demonstrated here.

1. Go to Preferences or Options icon.

2. Navigate to Network Settings .

3. Select Manual proxy configuration , use localhost or [Link] for the SOCKS host with port 9999 . Click OK when done.

4. Check the Proxy DNS when using SOCKS v5 box a bit further down.

For Chrome , the proxy settings can be configured from the Operating System networking configurations,, the proxy settings can be
configured from the Operating System networking configurations, e.g. LAN Settings on Windows.

1.3 Lab architecture

In your lab environments, MAAS and Openstack are already deployed. Managing resources in both cloud providers will be done by
switching your Juju client to different cloud providers using the juju switch command.

2 / 101
2. MAAS Operations

2. MAAS Operations

2.1 Create a new admin account

There is already an admin account created during the deployment process, but let's create a more customized one:

sudo maas createadmin --username=student --password=ubuntu \

--email=student@[Link]

sudo maas apikey --username=student > ~/student-maas-apikey

maas login student-profile [Link] - < ~/student-maas-apikey

In this way, you'll be creating a new profile with your own API key and you'll be able to run commands in MAAS using maas CLI. Your
personal/company e-mail address will not be required and you're going to use a fake one.

To verify you properly added the profile configuration, please run:

maas list
# output
deployprofile [Link] t2wCPAa3ZEUVMYvW...
student-profile [Link] cXPWhJrpqdqnPqnvH...

The MAAS Web UI can be accessed on [Link] . The credentials are admin/ubuntu.

2.2 Modify imported images list and start the import

By default, MAAS will only fetch Ubuntu 18.04 (bionic) and Ubuntu 20.04 (focal) images from its repositories. Let's also fetch Ubuntu 16.04
(xenial):

maas student-profile boot-source-selections create 1 os="ubuntu" release="xenial" \

arches="amd64" subarches="*" labels="*"

Then, start the import process:

3 / 101
2. MAAS Operations

maas student-profile boot-resources import

# output
Success.
Machine-readable output follows:
Import of boot resources started

Usually, import is done automatically, every hour. Changes can also be checked in MAAS WebUI > Images .

2.3 Go through a deployment process with a VM created on top of a POD

Verify the existence of the KVM POD:

maas student-profile pods read

maas student-profile vm-hosts read

The output should be the same for both commands. Next, create a SSH keypair that you're going to use to login to your newly created
instance:

ssh-keygen -t rsa -N "" -q -f ~/.ssh/student_id_rsa

maas student-profile sshkeys create key="`cat ~/.ssh/student_id_rsa.pub`"

Next, get the KVM host ID from MAAS and compose a virtual machine:

VMHOST_ID=`maas student-profile vm-hosts read | jq ".[0].id"`

maas student-profile vm-host compose $VMHOST_ID cores=1 memory=2048 \

architecture="amd64/generic" "storage=mylabel:10(default)" hostname="student-test"

This will automatically create a virtual machine and also commission it. Let's wait for the VM to be commissioned, in aReady state and
turned off. You will also notice the inventory for that VM will also be populated in MAAS WebUI.

The output should be something like:

Success.
Machine-readable output follows:
{
"system_id": "qa7w6e",
"resource_uri": "/MAAS/api/2.0/machines/qa7w6e/"
}

For the next command, we'll need the generated systemd_id from the previous command. Next step will be to deploy an operating
system on the newly created VM after the commissioning process is finished. You can check the progress in the WebUI. Here's the
command to deploy the OS:

maas student-profile machine deploy <system_id> osystem="ubuntu" distro_series="focal"

Example:

maas student-profile machine deploy qa7w6e osystem="ubuntu" distro_series="focal"

After that, you'll need to wait 5-10 minutes for the machine to be deployed with an operating system and configured, before you'll be able
to login to it.

You can now use your terminal window to connect to your newly deployed VM, after getting its IP address from the web interface:

4 / 101
2. MAAS Operations

ssh -i ~/.ssh/student_id_rsa ubuntu@<IP_address>

exit

Now that you verified access, we can now safely release the VM.

SYSTEM_ID=$(maas student-profile machines read | jq ".[1].system_id" | sed 's/\"//g')

maas student-profile machine release ${SYSTEM_ID}

Once the machine is release, it's will go back into aReady state and will be turned off. Now, let's also delete the instance:

maas student-profile machine delete ${SYSTEM_ID}

2.4 Create VLANs/Subnets/Ranges

In this exercise, you’ll create a VLAN inside MAAS and attach it to an existing fabric (fabric-1):

FABRIC_ID=`maas student-profile fabrics read | jq ".[1].id"`

Now that we have the VLAN defined, together with some other variables needed for the next commands, let's create a new subnet, some
reserved and dynamic ranges and enable DHCP. Everything will be checked out in the WebUI after finishing the setup.

maas student-profile subnets create cidr=[Link]/24 VLAN=$VLAN_ID \

fabric=$FABRIC_ID vid=100 gateway_ip=[Link] dns_servers=[Link]

SUBNET_ID=`maas student-profile subnets read \

| jq -r '.[] | select(.name == "[Link]/24") | .id'`

maas student-profile ipranges create type=dynamic \

start_ip=[Link] end_ip=[Link]

maas student-profile vlan update $FABRIC_ID 100 relay_vlan=$VLAN_ID_TARGET mtu=1400

maas student-profile ipranges create type=reserved \

start_ip=[Link] end_ip=[Link]

2.5 Checking through the logs

All the logs stored by MAAS installed from a snap can be found in /var/snap/maas/common/logs/ . Let's list all the files available:

cd /var/snap/maas/common/log/

ls -l

Files of interest will be:

[Link]
[Link]
[Link]

5 / 101
2. MAAS Operations

[Link]
[Link]
[Link]
rsyslog folder

Different log files provide details about each MAAS component, while the rsyslog folder contains boot log files for every instance created
by maas.

Let's check the contents of each of the files:

tail -20 [Link]

tail -20 [Link]
sudo tail -20 [Link]
tail -20 [Link]
tail -20 [Link]
tail -20 [Link]

Files in the rsyslog folder are not world-readable, that means we're going to use sudo to be actually able to read them:

sudo less rsyslog/<vm_name>/<YYYY-MM-DD>/messages

cd ~

2.6 Backup MAAS

Backup of the database and all configuration files will be done with all MAAS services being stopped so there's going to be consistency in
its database and no interrupted transactions.

So first, let's stop all MAAS services. Since it's installed from a Snap, this can be done using:

sudo systemctl stop [Link]

Once all services are stopped, let's do a PostgreSQL database backup:

sudo pg_dumpall -U postgres -h /var/snap/maas-test-db/common/postgres/sockets -c > ~/[Link]

maas-test-db is the name of the database/snap package installed whenever using all-in-one installations. While this provides an easy
testing environment, it's not suitable for HA production environments.

Let's check the [Link] file is in your home folder:

du -hs ~/[Link]
# output
3.1G [Link]

Next step will be to create an archive containing all MAAS configuration files and PostgreSQL dump. We're going to exclude a folder
called /var/snap/maas/common/maas/boot-resources since it contains OS images that can be easily imported whenever doing a
restore and they only increase the backup size which is not desirable.

sudo tar cvpzf ~/[Link] --exclude=/var/snap/maas/common/maas/boot-resources \

/snap/maas/current/etc/maas /var/snap/maas/common/maas ~/[Link]

Since the [Link] file is not that small, it will take around 5 minutes to complete. Once it's finished, let's check the contents and size of
the newly created backup:

6 / 101
3. Juju Operations

tar -tzf ~/[Link]

# output
snap/maas/current/etc/maas/
snap/maas/current/etc/maas/[Link]
snap/maas/current/etc/maas/preseeds/
snap/maas/current/etc/maas/preseeds/commissioning
snap/maas/current/etc/maas/preseeds/curtin
snap/maas/current/etc/maas/preseeds/curtin_userdata
snap/maas/current/etc/maas/preseeds/curtin_userdata_centos
snap/maas/current/etc/maas/preseeds/curtin_userdata_custom
snap/maas/current/etc/maas/preseeds/curtin_userdata_suse
snap/maas/current/etc/maas/preseeds/curtin_userdata_windows
snap/maas/current/etc/maas/preseeds/enlist
var/snap/maas/common/maas/
var/snap/maas/common/maas/[Link]
var/snap/maas/common/maas/[Link]
var/snap/maas/common/maas/gnupg/
var/snap/maas/common/maas/dhcp/
var/snap/maas/common/maas/dhcp/[Link]
var/snap/maas/common/maas/dhcp/[Link]~
var/snap/maas/common/maas/dhcp/[Link]
var/snap/maas/common/maas/secret
var/snap/maas/common/maas/dhcpd-interfaces
var/snap/maas/common/maas/maas_id
home/ubuntu/[Link]

du -hs ~/[Link]
# output
1.7G /home/ubuntu/[Link]

Since storage is limited, let's delete the plain dump file to free up some space:

rm -rf ~/[Link]

After deleting the dump, let's start the MAAS services back, so everything becomes functional again:

sudo systemctl start [Link]

3. Juju Operations

3.1 Inspecting the current setup

LAB architecture from the first chapter presented the current setup and other models that we're going to create during this lab. There are
two controllers created, one managing MAAS resources, the other one managing LXD resources.

Let's check everything out. First, let's see the controllers:

juju list-controllers
# output
Controller Model User Access Cloud/Region Models Nodes HA Version
lxd-controller* default admin superuser localhost/localhost 2 1 none 2.9.5
maas-controller default admin superuser maas/default 2 1 none 2.9.5

As stated, each controller is installed in a different cloud provider, MAAS and LXD. Let's check their configuration:

7 / 101
3. Juju Operations

juju list-clouds
# output
Only clouds with registered credentials are shown.
There are more clouds, use --all to see them.

Clouds available on the controller:

Cloud Regions Default Type
localhost 1 localhost lxd

Clouds available on the client:

Cloud Regions Default Type Credentials Source Description
localhost 1 localhost lxd 1 built-in LXD Container Hypervisor
maas 1 default maas 1 local Metal As A Service

Also, let's check each cloud configuration:

localhost cloud represents the LXD cloud provider using lxd-controller to manage resources

juju show-cloud localhost

# output
Cloud "localhost" from controller "lxd-controller":

defined: public
type: lxd
auth-types: [certificate]
endpoint: [Link]
credential-count: 1
regions:
localhost: {}
users:
admin:
display-name: admin
access: admin

Client cloud "localhost":

defined: built-in
type: lxd
description: LXD Container Hypervisor
auth-types: [certificate]
credential-count: 1
regions:
localhost: {}

maas cloud represents MAAS installation using maas-controller to manage resources

juju show-cloud maas

# output
Client cloud "maas":

defined: local
type: maas
description: Metal As A Service
auth-types: [oauth1]
endpoint: [Link]
credential-count: 1
regions:
default: {}

Since lxd-controller is the currently selected, first, we'll need to switch back to maas-controller to manage applications, models,
relations using that specific controller, while lxd-controller will only be used to manage Openstack services.

juju switch maas-controller

# output
lxd-controller:admin/default -> maas-controller:admin/default

8 / 101
3. Juju Operations

For this specific controller, let's check the status, show all the models available and check the status of the controller model:

juju status
# output
Model Controller Cloud/Region Version SLA Timestamp
default maas-controller maas/default 2.9.5 unsupported 19:12:23Z

Model "admin/default" is empty.

juju list-models
# output
Controller: maas-controller

Model Cloud/Region Type Status Machines Cores Access Last connection

controller maas/default maas available 1 1 admin just now
default* maas/default maas available 0 - admin just now

juju status -m controller

# output
Model Controller Cloud/Region Version SLA Timestamp
controller maas-controller maas/default 2.9.5 unsupported 19:12:25Z

Machine State DNS Inst id Series AZ Message

0 started [Link] f6sgyc focal default Deployed

3.2 Application deployment

Now that you inspected the current configuration, you're ready to deploy new applications, scale and related them.

First, let's create a new model:

juju add-model webapp

# output
Added 'webapp' model on maas/default with credential 'admin' for user 'admin'

After that, make sure default-series is set to bionic , so everything that's getting deployed, will use Ubuntu 18.04 as OS:

juju model-config default-series=bionic

Check the model default configs:

9 / 101
3. Juju Operations

juju model-config
# output
Attribute From Value
agent-metadata-url default ""
agent-stream default released
agent-version model 2.9.19
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
apt-no-proxy default ""
automatically-retry-hooks default true
backup-dir default ""
charmhub-url default [Link]
cloudinit-userdata default ""
container-image-metadata-url default ""
container-image-stream default released
container-inherit-properties default ""
container-networking-method model provider
default-series model bionic
default-space default ""
development default false
disable-network-management default false
disable-telemetry default false
egress-subnets default ""
enable-os-refresh-update default true
enable-os-upgrade default true
fan-config default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream default released
juju-ftp-proxy default ""
juju-http-proxy default ""
juju-https-proxy default ""
juju-no-proxy default [Link],localhost,::1
logforward-enabled default false
logging-config model <root>=INFO
lxd-snap-channel default latest/stable
max-action-results-age default 336h
max-action-results-size default 5G
max-status-history-age default 336h
max-status-history-size default 5G
net-bond-reconfigure-delay default 17
no-proxy default [Link],localhost,::1
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
snap-http-proxy default ""
snap-https-proxy default ""
snap-store-assertions default ""
snap-store-proxy default ""
snap-store-proxy-url default ""
ssl-hostname-verification default true
storage-default-block-source model maas
test-mode default false
transmit-vendor-metrics default true
update-status-hook-interval default 5m

Next, deploy a MySQL charm, a Wordpress charm, relate them and expose Wordpress:

juju deploy wordpress

# output
Located charm "wordpress" in charm-hub, revision 0
Deploying "wordpress" from charm-hub charm "wordpress", revision 0 in channel stable

10 / 101
3. Juju Operations

juju deploy percona-cluster mysql

# output
Located charm "percona-cluster" in charm-hub, revision 297
Deploying "mysql" from charm-hub charm "percona-cluster", revision 297 in channel stable

juju add-relation wordpress mysql

juju expose wordpress

juju deploy apache2

# output
Located charm "apache2" in charm-hub, revision 36
Deploying "apache2" from charm-hub charm "apache2", revision 36 in channel stable

It will take around 15-20 minutes for it to complete. You can either use juju status or use the watch command. Apache2 charm will
remain in an unknown workload status, since it's not going to be configured by any other charm, through a relation. We'll ignore that for
now.

juju status

Everything is done deploying when Workload status is active and Agent status is idle .

After everything is deployed, to get the unit IP address, please use this command:

juju status wordpress --format line | grep -v ^$ | awk '{print $3}' | head -1

After it's finished, you'll be able to access the Wordpress install menu in your tunneled browser using the Wordpress unit IP address.

11 / 101
3. Juju Operations

3.3 Application scaling

Next step will be to scale out this app. We'll also need new charms, like HAProxy.

juju add-unit -n 2 wordpress

juju deploy haproxy

# output
Located charm "haproxy" in charm-hub, revision 61
Deploying "haproxy" from charm-hub charm "haproxy", revision 61 in channel stable

juju add-relation haproxy wordpress

HAProxy will act as a load balancer in front of all three Wordpress units. You no longer need to access Wordpress's IP address (or any of
the three), you'll just need to point your browser to the HAProxy unit.

Adding those two more units and the HAProxy charm will take around 15-20 minutes. Please be patient.

To get its IP address, run this command:

12 / 101
3. Juju Operations

juju status haproxy --format line | grep -v ^$ | awk '{print $3}' | head -1

Again, you'll need to wait for everything to be completed before being able to access anything. That means, Workload status shows
active everywhere and Agent status shows idle everywhere.

juju status
# output
Model Controller Cloud/Region Version SLA Timestamp
webapp maas-controller maas/default 2.9.19 unsupported 15:31:38Z

App Version Status Scale Charm Store Channel Rev OS

apache2 active 1 apache2 charmhub stable 38 ubuntu
haproxy waiting 0/1 haproxy charmhub stable 66 ubuntu
mysql 5.7.20 active 1 percona-cluster charmhub stable 302 ubuntu
wordpress waiting 1/3 wordpress charmhub stable 0 ubuntu

Unit Workload Agent Machine Public address Ports Message

apache2/0* active idle 2 [Link] Unit is ready
haproxy/0 waiting allocating 5 [Link] waiting for machine
mysql/0* active idle 1 [Link] 3306/tcp Unit is ready
wordpress/0* active idle 0 [Link] 80/tcp
wordpress/1 waiting allocating 3 [Link] waiting for machine
wordpress/2 waiting allocating 4 [Link] waiting for machine

Machine State DNS Inst id Series AZ Message

0 started [Link] saving-hound bionic default Deployed
1 started [Link] free-shark bionic default Deployed
2 started [Link] joint-mite bionic default Deployed
3 pending [Link] equal-wolf bionic default Deploying: Performing PXE boot
4 pending [Link] tender-ewe bionic default Deploying: Performing PXE boot
5 pending [Link] main-foal bionic default Deploying: Performing PXE boot

juju status
# output
Model Controller Cloud/Region Version SLA Timestamp
webapp maas-controller maas/default 2.9.19 unsupported 15:47:21Z

App Version Status Scale Charm Store Channel Rev OS

apache2 active 1 apache2 charmhub stable 38 ubuntu
haproxy active 1 haproxy charmhub stable 66 ubuntu
mysql 5.7.20 active 1 percona-cluster charmhub stable 302 ubuntu
wordpress active 3 wordpress charmhub stable 0 ubuntu

Unit Workload Agent Machine Public address Ports Message

apache2/0* active idle 2 [Link] Unit is ready
haproxy/0* active idle 5 [Link] 80/tcp Unit is ready
mysql/0* active idle 1 [Link] 3306/tcp Unit is ready
wordpress/0* active idle 0 [Link] 80/tcp
wordpress/1 active idle 3 [Link] 80/tcp
wordpress/2 active idle 4 [Link] 80/tcp

Machine State DNS Inst id Series AZ Message

0 started [Link] saving-hound bionic default Deployed
1 started [Link] free-shark bionic default Deployed
2 started [Link] joint-mite bionic default Deployed
3 started [Link] equal-wolf bionic default Deployed
4 started [Link] tender-ewe bionic default Deployed
5 started [Link] main-foal bionic default Deployed

Accessing HAProxy IP address will lead you to the same install menu of Wordpress.

To check current configuration of HAProxy and to make sure it's forwarding requests to all three Wordpress units, let's run the next
command:

13 / 101
3. Juju Operations

juju ssh haproxy/0 cat /etc/haproxy/[Link]

# output

...
frontend haproxy-0-80
bind [Link]:80
default_backend haproxy_service

backend haproxy_service
balance leastconn
cookie SRVNAME insert
server wordpress-0-8080 [Link]:8080 maxconn 100 cookie S0 check
server wordpress-1-8080 [Link]:8080 maxconn 100 cookie S1 check
server wordpress-2-8080 [Link]:8080 maxconn 100 cookie S2 check

So, HAProxy charm will forward requests on all Wordpress units, but on port tcp/8080 instead of normal tcp/80. That's because every
Wordpress unit also runs an NGinx reverse proxy what will forward requests to all other Wordpress units and will set different HTTP
headers, like: Host, X-Real-IP, X-Forwarded-For. After which, the requests go to the same NGinx on port tcp/80, which actually servers the
blog platform.

3.4 SCP, Actions, Config

Now that we have a load balanced Wordpress, let's fetch the [Link] file locally, for further investigation:

juju scp haproxy/0:/etc/haproxy/[Link] .

less [Link]

Using actions, different charm can perform different tasks. For Percona-Cluster charm, you can do either a mysqldump that's usually
compressed and you can copy it remotely for long term storage on tape, for example, or you can do a "backup" that can be used to import
the files elsewhere.

Let's list the actions available for the Percona Cluster charm:

juju actions mysql

# output
Action Description
backup Full database backup
bootstrap-pxc Bootstrap this unit of Percona.
*WARNING* This action will bootstrap this unit of Percona cluster. This
should only occur in a recovery scenario. Make sure this unit has the
highest sequence number in [Link] or data loss may occur.
See upstream Percona documentation for context
complete-cluster-series-upgrade Perform final operations post series upgrade. Inform all nodes in the
cluster the upgrade is complete cluster wide. Update configuration with all
peers for wsrep replication.
This action should be performed on the current leader. Note the leader may
have changed during the series upgrade process.
generate-nagios-password Re-generate the password for the nagios user. This action can only be run
on the leader unit,
which sets a new password and then the `leader-settings-changed` hook
will be activate.
mysqldump MySQL dump of databases. Action will return mysqldump-file location of the
requested backup in the results. If the databases parameter is unset all
databases will be dumped. If the databases parameter is set only the
databases specified will be dumped. Note it may be necessary to use the
set-pxc-strict-mode action first to set either PERMISSIVE or MASTER to
allow locking of tables for mysqldump to complete successfully.
for more detail.
notify-bootstrapped No description
pause Pause the MySQL service.
resume Resume the MySQL service.
set-pxc-strict-mode Set PXC strict mode.

This is the lightweight output of the command. It shows the actions, but it's not actually showing the parameters each action can take. For

14 / 101
3. Juju Operations

that information, you can run the next command:

juju actions mysql --schema

This is the most complete output, together with all parameters available for each of the action. We're going to use later.

Let's try the "backup" action first:

juju run-action mysql/0 --wait backup compress=true incremental=false

# output
...
211126 15:54:46 Executing UNLOCK BINLOG
211126 15:54:46 Executing UNLOCK TABLES
211126 15:54:46 All tables unlocked
211126 15:54:46 Backup created in directory '/opt/backups/mysql/2021-11-26_15-54-45/'
211126 15:54:46 [00] Compressing /opt/backups/mysql/2021-11-26_15-54-45/[Link]
211126 15:54:46 [00] ...done
211126 15:54:46 [00] Compressing /opt/backups/mysql/2021-11-26_15-54-45/xtrabackup_info.qp
211126 15:54:46 [00] ...done
xtrabackup: Transaction log of lsn (2573231) to (2573240) was copied.
211126 15:54:46 completed OK!
outcome: Success
time-completed: "2021-11-26 15:54:47"
status: completed

Backup file will be stored in a folder called /opt/backups/mysql/<timestamp/ . Let's list all files in that folder:

juju ssh mysql/0 sudo ls -al /opt/backups/mysql/<timestamp>/

As you might notice, permissions for the files do not allow access for others , that means it needs some extra commands for you to have
remote access to them, like using chmod . We're going to exemplify that later on for the mysqldump action, for the exact same reason.

Next, let's try the mysqldump action:

juju run-action mysql/0 --wait mysqldump

# output
[Link]: Command '['/usr/bin/mysqldump', '-u', 'root',
'--default-character-set=utf8', '--triggers', '--routines', '--events',
'--ignore-table=[Link]', '--result-file', '/var/backups/mysql/mysqldump-all-databases-202111261555',
'--all-databases']' returned non-zero exit status 2.
status: failed

Apparently, there's an error and mysqldump can't be taken, because of pxc-strict-mode . There are two ways of modifying that, either
by using actions or by using juju config . Let's do juju config this time.

First, let's check the current configuration of the parameter:

juju config mysql pxc-strict-mode

# output
enforcing

pxc-strict-mode set to enforcing will not allow us to do mysqldump. That means, we need to change this parameter to "permissive" or
"disabled". Let's do permissive:

juju config mysql pxc-strict-mode=permissive

Once that's set, let's try to do the mysqldump again:

15 / 101
3. Juju Operations

juju run-action mysql/0 --wait mysqldump

# output

unit-mysql-0:
UnitId: mysql/0
id: "6"
results:
mysqldump-file: /var/backups/mysql/[Link]
outcome: Success
status: completed
timing:
completed: 2021-11-26 15:57:13 +0000 UTC
enqueued: 2021-11-26 15:56:49 +0000 UTC
started: 2021-11-26 15:57:12 +0000 UTC

This time it worked. Let's try and fetch this archive locally:

juju scp mysql/0:/var/backups/mysql/mysqldump-all-databases-<timestamp>.gz .

# output
ERROR exit status 1 (scp: /var/backups/mysql/mysqldump-all-databases-<timestamp>.gz: Permission denied)

This is where chmod command comes into play. We need to add read + execute permissions for the other group.

juju ssh mysql/0 sudo chmod o+rx /var/backups/mysql

juju scp mysql/0:/var/backups/mysql/mysqldump-all-databases-<timestamp>.gz .

Both commands worked. First one, change permissions on the parent folder, while the second command was able to actually fetch the
archive from the MySQL unit. Let's see if the file actually exists locally and let's scroll through it:

ls -al | grep mysqldump

Inspecting the file can be done with:

zless mysqldump-all-databases-<timestamp>.gz

Dumps can be copied remotely and stored on some shared network device, tape device or some backup server.

3.5 Debug logging

Juju has different ways of checking what's happening to all of your charms deployed. The most used one, that you can check whenever
you have the juju client command is juju debug-log .

Example

juju debug-log
# output
unit-mysql-0: 09:55:11 INFO [Link] ran "update-status" hook
unit-wordpress-0: 09:56:00 INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-2: 09:56:41 INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-1: 09:58:25 INFO [Link] skipped "update-status" hook (missing)
unit-haproxy-0: 09:58:26 INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-0: 10:00:47 INFO [Link] skipped "update-status" hook (missing)
machine-1: 10:01:06 INFO [Link]/[Link]-log Updating status.
machine-1: 10:01:07 INFO [Link]/[Link]-log Unit is ready
unit-mysql-0: 10:01:07 INFO [Link] ran "update-status" hook

juju debug-log shows you information about all the charms, machines, actions and hooks that are getting run within your model. This is
a scrolling output, so to exit this command, you'll need to hit Ctrl + C .

16 / 101
3. Juju Operations

Since it's a scrolling output, and you might be interested in a specific unit or application, you can filter the result based on what you're
interested in. Let's see what's deployed in our model:

juju status
# output
Model Controller Cloud/Region Version SLA Timestamp
webapp maas-controller maas/default 2.9.19 unsupported 15:59:03Z

App Version Status Scale Charm Store Channel Rev OS

Unit Workload Agent Machine Public address Ports Message

Machine State DNS Inst id Series AZ Message

Let's check the latest debug logging coming from mysql/0 unit:

juju debug-log -i mysql/0

# output
unit-mysql-0: 09:13:44 INFO [Link] ran "update-status" hook (via explicit...
unit-mysql-0: 09:19:32 INFO [Link] ran "update-status" hook (via explicit...
unit-mysql-0: 09:24:39 INFO [Link] ran "update-status" hook (via explicit...

These are all the logs from hooks, actions that happen now. How about what happened from whenever the unit was initially installed? You
can do that by using:

juju debug-log -i mysql/0 --replay --no-tail

# output
unit-mysql-0: INFO juju Starting unit workers for "mysql/0"
unit-mysql-0: INFO [Link] [ec74e1] "unit-mysql-0" successfully connected to "[Link]:17070"
unit-mysql-0: INFO [Link] [ec74e1] password changed for "unit-mysql-0"
unit-mysql-0: INFO [Link] [ec74e1] "unit-mysql-0" successfully connected to "[Link]:17070"
unit-mysql-0: INFO [Link] migration phase is now: NONE
unit-mysql-0: INFO [Link] logger worker started
unit-mysql-0: INFO [Link] no waiter, upgrader is done
....

Also, you can filter by multiple units or applications. In the next example, let's check last debug logs for both mysql/0 unit and
haproxy/0 unit:

17 / 101
3. Juju Operations

juju debug-log -i mysql/0 -i haproxy/0 --no-tail

# output
unit-haproxy-0: INFO [Link] skipped "update-status" hook (missing)
unit-mysql-0: INFO [Link] ran "update-status" hook (via explicit, bespoke hook script)
unit-haproxy-0: INFO [Link] skipped "update-status" hook (missing)
unit-mysql-0: INFO [Link] ran "update-status" hook (via explicit, bespoke hook script)
unit-haproxy-0: INFO [Link] skipped "update-status" hook (missing)
unit-mysql-0: INFO [Link] ran "update-status" hook (via explicit, bespoke hook script)
unit-haproxy-0: INFO [Link] skipped "update-status" hook (missing)
unit-mysql-0: INFO [Link] ran "update-status" hook (via explicit, bespoke hook script)

Specifying --no-tail option will remove the scrolling output behavior and will not show you future events happening to your units.

Last but not least, you can check the debug logs for all the units of one single application. In our case, we have three units of wordpress.
To check its debug logs, let's do this:

juju debug-log -i wordpress --no-tail

# output
unit-wordpress-0: INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-2: INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-1: INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-0: INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-2: INFO [Link] skipped "update-status" hook (missing)
unit-wordpress-1: INFO [Link] skipped "update-status" hook (missing)

3.6 Debug hooks

Debugging hooks is helpful whenever you are creating your own charm, or you want to know why a specific charm is running into an error
while running a hook or action. You can use it like this:

juju debug-hooks <application/unit> [hook-name hook-name2 ...]

juju debug-hooks <service/unit> [action-name action-name2 ...]

During the last labs, we had some parameter we needed to change to mysql charm in order to be able to take a mysqldump of all the
databases. To be able to do that, we modified a config option called "pxcstrict- mode" from "enforcing" to "permissive". After we changed
that, we managed to be able to take a MySQL dump. Now, let's change the parameter back to "enforcing"

juju config mysql pxc-strict-mode=enforcing

Please wait for a minute for the change to propagate. Next, let's open a debug-hooks session on mysql/0 , searching for the
mysqldump action. To do that, please run:

juju debug-hooks mysql/0 mysqldump

You'll be presented with a TMUX session that waits for whatever hook or action you instructed it to wait for. In this case, it will be the
"mysqldump" action.

18 / 101
3. Juju Operations

In another terminal window, connected to your lab environment, let's run the action the TMUX session is waiting for:

juju run-action mysql/0 --wait mysqldump

As you will notice, the hook will no longer run in this session. Usually, you would have an output within seconds without running debug-
hooks . In this case, something changed in the TMUX session and it's waiting for some input.

19 / 101
3. Juju Operations

Our current folder is already changed to the home folder of the charm, in this case /var/lib/juju/agents/unit-mysql-0/charm .

Now, let's try and manually run the action, using the instructions presented above:

/var/lib/juju/agents/unit-mysql-0/charm# ./actions/mysqldump
# output
mysqldump: Got error: 1105: Percona-XtraDB-Cluster prohibits use of LOCK TABLE/FLUSH TABLE <table> WITH READ
LOCK with pxc_strict_mode = ENFORCING when using LOCK TABLES

Since we changed back the pxc-strict-mode parameter to enforcing, we are no longer able to take that mysqldump from our MySQL
unit and we're getting the same error as before.

While a charm is in debug-hooks mode, you'll not be able to change any of its configuration, or run other actions or hooks:

If you want to go back to permissive pxc-strict-mode, we need to first exit debug-hooks session and use juju config command again.
To exit the TMUX session, please it Ctrl + D and from the main "bash" session, use Ctrl + a and then d .

In the window you started the mysqldump action, once you close the TMUX session, you'll also get some error from the action itself:

20 / 101
3. Juju Operations

juju run-action mysql/0 --wait mysqldump

# output
unit-mysql-0:
UnitId: mysql/0
id: "10"
message: mysqldump failed
results:
Stderr: |
mysqldump: Got error: 1105: Percona-XtraDB-Cluster prohibits use of LOCK TABLE/FLUSH
TABLE <table> WITH READ LOCK with pxc_strict_mode = ENFORCING when using LOCK TABLES
....

Let's try and change back MySQL configuration and try debug-hooks again:

#1 terminal:

juju config mysql pxc-strict-mode=permissive

juju debug-hooks mysql/0 mysqldump

(this will open the TMUX session and wait for the action)

#2 terminal:

juju run-action mysql/0 --wait mysqldump

debug-hooks session already detected the action we tried to run. Last time we ran it, we got into an error.

Now that we've changed MySQL configuration, let's try and run the action again:

21 / 101
3. Juju Operations

root@main-flea:/var/lib/juju/agents/unit-mysql-0/charm# ./actions/mysqldump
root@main-flea:/var/lib/juju/agents/unit-mysql-0/charm#

This time, there was no error. Let's close this session with Ctrl + D and switch back to our second terminal window:

juju run-action mysql/0 --wait mysqldump

# output
unit-mysql-0:
UnitId: mysql/0
id: "12"
results:
mysqldump-file: /var/backups/mysql/[Link]
outcome: Success
status: completed
timing:
completed: 2021-11-26 16:04:26 +0000 UTC
enqueued: 2021-11-26 16:03:59 +0000 UTC
started: 2021-11-26 16:04:16 +0000 UTC

You'll notice that this time the action succeeded and you know have a new mysqldump waiting for you.

3.7 Maintenance tasks

In this chapter, you will perform different tasks, like removing machine, units, applications, models, taking backups of the controllers,
upgrading models and agents, enable High Availability on the controller. After everything is finished, we're going to remove maas-
controller altogether.

Removing an application is done by using juju remove-application command, and it will remove all units of an application:

juju remove-application apache2

Verify the Apache2 application was removed:

juju status
# output
Model Controller Cloud/Region Version SLA Timestamp
webapp maas-controller maas/default 2.9.19 unsupported 16:05:09Z

App Version Status Scale Charm Store Channel Rev OS

haproxy active 1 haproxy charmhub stable 66 ubuntu
mysql 5.7.20 active 1 percona-cluster charmhub stable 302 ubuntu
wordpress active 3 wordpress charmhub stable 0 ubuntu

Unit Workload Agent Machine Public address Ports Message

haproxy/0* active idle 5 [Link] 80/tcp Unit is ready
mysql/0* active idle 1 [Link] 3306/tcp Unit is ready
wordpress/0* active idle 0 [Link] 80/tcp
wordpress/1 active idle 3 [Link] 80/tcp
wordpress/2 active idle 4 [Link] 80/tcp

Machine State DNS Inst id Series AZ Message

0 started [Link] saving-hound bionic default Deployed
1 started [Link] free-shark bionic default Deployed
3 started [Link] equal-wolf bionic default Deployed
4 started [Link] tender-ewe bionic default Deployed
5 started [Link] main-foal bionic default Deployed

Removing application unit. We next have the HAProxy + Wordpress setup. HAProxy forwards requests to all three Wordpress units and
the backend configuration of HAProxy contains all three IP addresses for all Wordpress units. Let's delete a unit of Wordpress:

juju remove-unit wordpress/2 --force

22 / 101
3. Juju Operations

First, let's verify the unit has been removed:

juju status
# output
Model Controller Cloud/Region Version SLA Timestamp
webapp maas-controller maas/default 2.9.19 unsupported 16:05:45Z

App Version Status Scale Charm Store Channel Rev OS

haproxy active 1 haproxy charmhub stable 66 ubuntu
mysql 5.7.20 active 1 percona-cluster charmhub stable 302 ubuntu
wordpress active 2 wordpress charmhub stable 0 ubuntu

Unit Workload Agent Machine Public address Ports Message

haproxy/0* active idle 5 [Link] 80/tcp Unit is ready
mysql/0* active idle 1 [Link] 3306/tcp Unit is ready
wordpress/0* active idle 0 [Link] 80/tcp
wordpress/1 active idle 3 [Link] 80/tcp

Machine State DNS Inst id Series AZ Message

0 started [Link] saving-hound bionic default Deployed
1 started [Link] free-shark bionic default Deployed
3 started [Link] equal-wolf bionic default Deployed
5 started [Link] main-foal bionic default Deployed

Second, let's check current HAProxy backend configuration:

juju ssh haproxy/0 cat /etc/haproxy/[Link]

# output

...
frontend haproxy-0-80
bind [Link]:80
default_backend haproxy_service

backend haproxy_service
balance leastconn
cookie SRVNAME insert
server wordpress-0-8080 [Link]:8080 maxconn 100 cookie S0 check
server wordpress-1-8080 [Link]:8080 maxconn 100 cookie S1 check

As you might notice, the wordpress/2 unit has been removed and HAProxy now forwards requests to the only two remaining units.
These changes are done through the relation between HAProxy and Wordpress charms.

Juju controller backup. Juju controller backups can be done either on a regular basis, or before doing upgrades to agents and controllers.
Usually, it's downloaded on the machine you're running the command from (where juju client is installed), but can be also stored locally on
the Juju controller.

Let's take a backup of the maas-controller (the controller model):

23 / 101
3. Juju Operations

juju create-backup -m maas-controller:controller

# output
backup ID:
backup format version: 1
juju version: 2.9.19
series: focal

controller UUID: b122309b-0f6d-4471-80d1-d5adfdc9915e

model UUID: 08c44ea2-1753-4440-82fc-5df5578786c3
machine ID: 0
created on host: heroic-prawn

checksum: 3YaJkx9VbgcoEHrx0iVdlLLEgcE=
checksum format: SHA-1, base64 encoded
size (B): 98919479
stored: 0001-01-01 00:00:00 +0000 UTC
started: 2021-11-26 16:06:27.287211686 +0000 UTC
finished: 2021-11-26 16:06:54.619337261 +0000 UTC

notes:

Remote backup was not created.

Downloaded to [Link].

Upgrades. You can do the following upgrades:

1. controller - upgrades the software on the juju controller side

2. model - upgrades juju agents inside models
3. client - upgrades Juju client

Whenever doing upgrades in production, this should be the order to follow: controller, model(s), juju client.

Let's start by doing a controller upgrade. Since you're switched to maas-controller , this is the controller you're going to upgrade:

juju upgrade-model -m controller

# output
no upgrades available

In this case, there's no upgrade available, but can be during the lab.

Upgrading models is done by running juju upgrade-model -m <model> . If not specified, it will, again, upgrade the currently selected
model. To verify the selected model, type juju list-models or just juju models :

juju models
# output
Controller: maas-controller

Model Cloud/Region Type Status Machines Cores Units Access Last connection
controller maas/default maas available 1 1 - admin just now
default maas/default maas available 0 - - admin 50 minutes ago
webapp* maas/default maas available 4 4 4 admin 1 minute ago

In this case, juju upgrade-model with no specific model defined, will upgrade the webapp model:

juju upgrade-model
# output
no upgrades available

In case you want to upgrade a different mode, specify it with-m flag:

24 / 101
3. Juju Operations

juju upgrade-model -m default

# output
no upgrades available

Upgrading the client (if installed using snaps) is done by:

sudo snap refresh juju

# output
snap "juju" has no updates available

Snap uses channels for different releases of different software. snap refresh juju will upgrade to the newest version of the juju client,
from the currently installed channel. If you want to upgrade to a newer version, you'll also need to upgrade the source channel from which
it's taking the new version. To list all channels available, please type:

snap info juju --color=auto

# output

name: juju
summary: Juju - a model-driven operator lifecycle manager for K8s and machines
publisher: Canonical✓
store-url: [Link]
contact: [Link]
license: unset
...
snap-id: e2CPHpB1fUxcKtCyJTsm5t3hN9axJ0yj
tracking: 2.9/stable
refresh-date: today at 09:59 UTC
channels:
latest/stable: 2.9.19 2021-11-23 (17725) 109MB classic
latest/candidate: ↑
latest/beta: ↑
latest/edge: 3.0-beta1-ab8a327 2021-11-26 (17785) 92MB classic
2.9/stable: 2.9.19 2021-11-23 (17725) 109MB classic
2.9/candidate: ↑
2.9/beta: ↑
2.9/edge: 2.9.20-e34a4ba 2021-11-26 (17789) 109MB classic
2.8/stable: 2.8.13 2021-11-11 (17665) 74MB classic
2.8/candidate: ↑
2.8/beta: ↑
2.8/edge: 2.8.14-0445a6e 2021-11-09 (17673) 74MB classic
2.7/stable: 2.7.8 2020-07-22 (13563) 77MB classic
2.7/candidate: ↑
2.7/beta: ↑
2.7/edge: ↑
2.6/stable: 2.6.10 2019-10-30 (9484) 69MB classic
2.6/candidate: ↑
2.6/beta: ↑
2.6/edge: ↑
2.5/stable: 2.5.8 2019-06-27 (8361) 67MB classic
2.5/candidate: ↑
2.5/beta: ↑
2.5/edge: ↑
2.4/stable: 2.4.7 2019-05-23 (6012) 58MB classic
2.4/candidate: ↑
2.4/beta: ↑
2.4/edge: ↑
2.3/stable: 2.3.9 2018-09-19 (5267) 55MB classic
2.3/candidate: ↑
2.3/beta: ↑
2.3/edge: ↑
installed: 2.9.19 (17725) 109MB classic

So, you're tracking version 2.9. That means, whenever using snap refresh , it will install the latest 2.9 stable version available.

3.8 Cleanup & HA

25 / 101
3. Juju Operations

In this chapter, we're going to free up some resources, by deleting the webapp model and then enable controller HA, so Juju will build a
cluster of three controllers and will help with maintenance in production environments.

Let's list the models available, again. All apps are in the webapp model:

juju models
# output
Controller: maas-controller

Model Cloud/Region Type Status Machines Cores Units Access Last connection
controller maas/default maas available 1 1 - admin just now
default maas/default maas available 0 - - admin 53 seconds ago
webapp* maas/default maas available 4 4 4 admin 1 minute ago

To destroy the webapp model, please run:

juju destroy-model -y webapp

# output
Destroying model
Waiting for model to be removed, 4 machine(s), 3 application(s)........
Waiting for model to be removed, 4 machine(s), 2 application(s)...
Waiting for model to be removed, 2 machine(s), 1 application(s)...
Waiting for model to be removed....
Model destroyed.

While it's deleting the model, you can check MAAS web interface to actually see the VMs being release and deleted.

Let's wait a minute or two for the machines to be fully released. After that, let's enable controller High Availability. To do that, simply run:

juju enable-ha -n <number_of_units>

Example:

juju enable-ha
# output
maintaining machines: 0
adding machines: 1, 2

26 / 101
3. Juju Operations

Whenever enabling HA, you need to specify an odd number of units, like 3, 5, etc. If you're not specifying the number of units, it will
automatically build a cluster of three controllers. Let's check the status:

juju status -m controller

# output
Model Controller Cloud/Region Version SLA Timestamp
controller maas-controller maas/default 2.9.19 unsupported 16:11:23Z

Machine State DNS Inst id Series AZ Message

0 started [Link] wna7ye focal default Deployed
1 pending pending focal starting
2 pending pending focal

Two new VMs are being created. The complete setup process will take around 10 minutes. Please be patient. You can use this command
to watch the progress:

watch -c juju status -m controller --color

Once everything is green, let's check the status first:

juju status -m controller

# output
Model Controller Cloud/Region Version SLA Timestamp
controller maas-controller maas/default 2.9.19 unsupported 16:20:05Z

Machine State DNS Inst id Series AZ Message

0 started [Link] wna7ye focal default Deployed
1 started [Link] main-bat focal default Deployed
2 started [Link] good-horse focal default Deployed

And then, let's check if it's really an HA setup:

27 / 101
3. Juju Operations

juju show-controller maas-controller

# output
maas-controller:
details:
uuid: b122309b-0f6d-4471-80d1-d5adfdc9915e
controller-uuid: b122309b-0f6d-4471-80d1-d5adfdc9915e
api-endpoints: ['[Link]:17070', '[Link]:17070', '[Link]:17070']
cloud: maas
region: default
agent-version: 2.9.19
agent-git-commit: bae1644c23fd86e58dbf7249af55a236d0159099
controller-model-version: 2.9.19
...
controller-machines:
"0":
instance-id: wna7ye
ha-status: ha-enabled
ha-primary: true
"1":
instance-id: g3waw8
ha-status: ha-enabled
"2":
instance-id: qys4dk
ha-status: ha-enabled
models:
controller:
uuid: 08c44ea2-1753-4440-82fc-5df5578786c3
model-uuid: 08c44ea2-1753-4440-82fc-5df5578786c3
machine-count: 3
core-count: 3
default:
uuid: 717338e8-da25-4fb6-8667-1e7dfed130d1
model-uuid: 717338e8-da25-4fb6-8667-1e7dfed130d1
current-model: admin/webapp
account:
user: admin
access: superuser

api-endpoints will show you the IP/port numbers of all the units connected in the cluster. They match the IP addresses you see
whenever using "juju status -m controller"

controller machines show "ha-enabled", while the primary node is set to machine "0" (the first one created)

Now that we have HA up and running, let's break it somehow. Let's assume that you completely lost one machine and then see what
happens and try to achieve HA again.

To suddenly remove a machine from the cluster, we can use juju remove-machine -m controller <ID_of_machine>

juju remove-machine -m controller 2

# output
removing machine 2

So, we had machines "0", "1" and "2" and we just removed "2". juju status only shows two of them up and running:

juju status -m controller

#output
Model Controller Cloud/Region Version SLA Timestamp
controller maas-controller maas/default 2.9.19 unsupported 16:22:17Z

Machine State DNS Inst id Series AZ Message

0 started [Link] wna7ye focal default Deployed
1 started [Link] main-bat focal default Deployed

And let's check the HA status:

28 / 101
3. Juju Operations

juju show-controller maas-controller

# output
...
controller-machines:
"0":
instance-id: wna7ye
ha-status: ha-enabled
ha-primary: true
"1":
instance-id: g3waw8
ha-status: ha-pending
...

Since there's only two of them lost (even number), there's no "ha-enabled" setup anymore, they just work in active-passive mode. That
means if the primary node fails (machine "0"), the other node becomes primary (machine "1").

Let's assume you brought some resources back to your infrastructure and you want a third controller added to the cluster, so it's all "ha-
enabled" and you'd have an odd number. This is a very simple task and can be achieved by running again:

juju enable-ha -n 3
# output
maintaining machines: 0, 1
adding machines: 3

You already had machines "0" and "1", since "2" was removed. Now, there's a machine "3" being added to the cluster. The whole process
will take around 10 minutes, but you can watch the progress with:

watch -c juju status -m controller --color

Once that's finished, let's check the status of the "controller" model and also check the details of the "maas-controller"

juju status -m controller

# output
Model Controller Cloud/Region Version SLA Timestamp
controller maas-controller maas/default 2.9.19 unsupported 16:32:48Z

Machine State DNS Inst id Series AZ Message

0 started [Link] wna7ye focal default Deployed
1 started [Link] main-bat focal default Deployed
3 started [Link] useful-shrew focal default Deployed

juju show-controller maas-controller

# output
controller-machines:
"0":
instance-id: wna7ye
ha-status: ha-enabled
ha-primary: true
"1":
instance-id: g3waw8
ha-status: ha-enabled
"3":
instance-id: 6hcchn
ha-status: ha-enabled

Last but not least, let's do a bit of cleanup and let's destroy the controller altogether with all the models in it. After this is finished, the whole
controller will be deleted together with all models, apps and everything else related to "maas-controller". Also, if you check MAAS interface,
you'll see all controllers VMs being "released" and deleted and they're no longer required.

29 / 101
4. Keystone

juju destroy-controller maas-controller --destroy-all-models -y

# output
Destroying controller
Waiting for hosted model resources to be reclaimed
Waiting for 1 model
Waiting for 1 model
Waiting for 1 model
All hosted models reclaimed, cleaning up controller machines

4. Keystone
Keystone provides authentication and authorization for all the users and services inside your Openstack cloud. It also provides
authentication tokens, catalog of all the services and endpoints available and policy services.

In this chapter, you're going to manage users and projects, roles, policies and see where to identify different errors that may occur while
Keystone is operating.

4.1 openrc, endpoints, current setup

To access all the services inside your Openstack cloud, you'll need an openrc file. There are several provided in the lab, in the
~/resources folder:

ls -l ~/resources/*openrc
# output
-rw-r--r-- 1 ubuntu ubuntu 727 Jul 6 18:42 /home/ubuntu/resources/admin_openrc
-rw-r--r-- 1 ubuntu ubuntu 727 Jul 6 18:42 /home/ubuntu/resources/user1_openrc
-rw-r--r-- 1 ubuntu ubuntu 727 Jul 6 18:42 /home/ubuntu/resources/user2_openrc

For the moment, there's only the admin user available. The admin user gets created whenever you're deploying your cloud and it's the
administrator account that you're going to use whenever needed.

For now, let's source the file and see which environment variables get set. They should all start with OS_ :

30 / 101
4. Keystone

juju switch lxd-controller

source ~/resources/admin_openrc

env | grep OS_

# output
OS_REGION_NAME=RegionOne
OS_AUTH_VERSION=3
OS_CACERT=/home/ubuntu/resources/certs/[Link]
OS_AUTH_URL=[Link]
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=admin
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=admin
OS_PASSWORD=openstack
OS_IDENTITY_API_VERSION=3

All environment variables starting with OS_ are using by the Openstack clients (either "openstack" command, or the project command)
whenever authenticating to Keystone.

Let's check if these variables would be enough for you to get a list of all the endpoints available in your Openstack cloud:

31 / 101
4. Keystone

openstack endpoint list

As you might have noticed, all endpoints how use HTTPS based on a self-signed certificate generated during the deployment process.
Also, all endpoints use a DNS name. That DNS zone, [Link] is stored inside MAAS, as MAAS can also provide DNS services
for your infrastructure.

Let's also check the catalog of services:

openstack catalog list

32 / 101
4. Keystone

The information is the same, it's just structured differently.

There are some default domains that get created when deploying a new Openstack cloud using Juju. Still, you can create new domains,
projects, users and groups to ensure a more granular security subdivision for your cloud. Currently defined domains can be listed like this:

openstack domain list

The service_domain is used by internal Openstack services to be able to authenticate to one another. All services that will require
authentication to any other service inside your Openstack cloud, it will have a user inside this domain. This domain comes with a
services project created and service users. Let's list all of them:

openstack project list --domain service_domain

# output
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 1a3364cc60814b70813bcd4972f93c5e | services |
+----------------------------------+----------+

openstack user list --domain service_domain

The admin_domain is where you can find the administrator account. This only has a project called admin and the only user defined, it's
the admin user. Let's list that, too:

34 / 101
4. Keystone

openstack project list --domain admin_domain

# output
+----------------------------------+-------+
| ID | Name |
+----------------------------------+-------+
| fd4a4ff53b184df581b2a5b60d62fb7c | admin |
+----------------------------------+-------+

openstack user list --domain admin_domain

# output
+----------------------------------+-------+
| ID | Name |
+----------------------------------+-------+
| 1bbf44cdbdfd4d9ab7eda4323f596439 | admin |
+----------------------------------+-------+

4.2 Creating domains, projects and users

A domain in OpenStack is a way to partition users, groups and projects. Why do they exist? Without domains, the admin user is global, not
per project, so granting a user the admin role in any project gives the user administrative rights across the whole cloud. With domains,
multi-tenancy is achieved. An admin role can be assigned per domain, so that admin role affects only the projects in that specific domain,
not the other. Domains solve the Cloud Provider and Reseller paradigm issue.

User names also had global visibility and could also result in undesired collisions on user names if two different organizations both used
the same user name. In order for OpenStack to better support multiple user organizations at the same time, Keystone added a new
abstraction, called a Domain, that could provide the ability to isolate the visibility of a set of Projects and Users (and User Groups) to a
specific organization.

35 / 101
4. Keystone

Create a new domain called dev_domain . This domain can be reserved for the dev team in an organization.

source ~/resources/admin_openrc

openstack domain create dev_domain

Show domain information:

openstack domain show dev_domain

Domain can be disabled:

openstack domain set dev_domain --disable

Show domain information:

openstack domain show dev_domain

Enable the domain again:

openstack domain set dev_domain --enable

Let's create two different projects and two different users, each user being part of the previously created project. Both projects will be
created inside admin_domain domain.

openstack project create --enable --description "User1 Project" user1project --domain admin_domain

openstack project create --enable --description "User2 Project" user2project --domain admin_domain

Now that we have projects, we can create users and assign roles to those users. In this case, we're going to go with theMember role as
we want to extend its capabilities later on.

First, let's create two new users:

openstack user create --project user1project --email user1@[Link] \

--password openstack --enable user1 --domain admin_domain

openstack user create --project user2project --email user2@[Link] \

--password openstack --enable user2 --domain admin_domain

Lastly, let's assign the Member role for each user, relative to their own projects:

openstack role list

36 / 101
4. Keystone

openstack role add --project-domain admin_domain --user-domain admin_domain \

--user user1 --project user1project member

openstack role add --project-domain admin_domain --user-domain admin_domain \

--user user2 --project user2project member

So, right now, there are two new projects with two new users, each user having a Member role in each project. A user can have multiple
roles in the same project, but can also have multiple roles in different projects. Custom roles can be created, but they require thorough
testing in a testing environment before creating the same role in a production environment. Creating roles will imply policy changes for
each project affected by the change/role.

E.g. If you were to modify which role can list endpoints, that will affect the policy files for the Openstack service managing endpoints - in
this case, Keystone. If you want to create a custom role that can do more actions on instances created in Openstack, that means the policy
for the Nova service also need to be modified to accommodate new roles created.

4.3 Modifying policies

In this task, you're going to try and extend the capabilities for the Member role. The endpoint catalog listing is available only for the
administrator account, but we're going to try and get the list using one of the users you just created.

First, let's source the openrc file for user1 and check the environment variables being set:

source ~/resources/user1_openrc

env | grep OS_

# output
OS_REGION_NAME=RegionOne
OS_AUTH_VERSION=3
OS_CACERT=/home/ubuntu/resources/certs/[Link]
OS_AUTH_URL=[Link]
OS_PROJECT_DOMAIN_NAME=admin_domain
OS_AUTH_PROTOCOL=https
OS_USERNAME=user1
OS_AUTH_TYPE=password
OS_USER_DOMAIN_NAME=admin_domain
OS_PROJECT_NAME=user1project
OS_PASSWORD=openstack
OS_IDENTITY_API_VERSION=3

Every openstack command you'll run (unless you source another openrc file) it will run as the user1 user. Let's try and list the endpoint
catalog from Keystone:

openstack endpoint list

# output
You are not authorized to perform the requested action: identity:list_endpoints. (HTTP 403)

An error is seen as listing endpoints, apparently, is not available for a normal user with a Member role. Let's check the current policy file
for Keystone:

37 / 101
4. Keystone

juju ssh keystone/0 sudo head -30 /etc/keystone/[Link]

# output
{
"admin_required": "role:Admin",
"cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:e5f4e8e210f3...",
"service_role": "role:service",
"service_or_admin": "rule:admin_required or rule:service_role",
"owner": "user_id:%(user_id)s or user_id:%([Link].user_id)s",
"admin_or_owner": "(rule:admin_required and domain_id:%([Link])s) or rule:owner",
"admin_and_matching_domain_id": "rule:admin_required and domain_id:%(domain_id)s",
"service_admin_or_owner": "rule:service_or_admin or rule:owner",
"default": "rule:admin_required",
......
"identity:get_endpoint": "rule:admin_required",
"identity:list_endpoints": "rule:admin_required",
"identity:create_endpoint": "rule:cloud_admin",
"identity:update_endpoint": "rule:cloud_admin",
"identity:delete_endpoint": "rule:cloud_admin",
.......

Currently, there's admin_required rule that only matches the Admin role and listing endpoints will only be allowed based on this rule.
We're going to add a rule for the Member role and modify list_endpoints rule to take into account the Member role as well (based on
the new rule).

New rules should be:

"member_role": "role:Member",
"identity:list_endpoints": "rule:admin_required or rule:member_role",

The new policy should look like below:

juju ssh keystone/0 sudo head -30 /etc/keystone/[Link]

# output
{
"admin_required": "role:Admin",
-> "member_role": "role:member", <-
"cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:e5f4e8e210f34...",
"service_role": "role:service",
"service_or_admin": "rule:admin_required or rule:service_role",
"owner": "user_id:%(user_id)s or user_id:%([Link].user_id)s",
"admin_or_owner": "(rule:admin_required and domain_id:%([Link])s) or rule:owner",
"admin_and_matching_domain_id": "rule:admin_required and domain_id:%(domain_id)s",
"service_admin_or_owner": "rule:service_or_admin or rule:owner",
"default": "rule:admin_required",
......
"identity:get_service": "rule:admin_required",
"identity:list_services": "rule:admin_required",
"identity:create_service": "rule:cloud_admin",
"identity:update_service": "rule:cloud_admin",
"identity:delete_service": "rule:cloud_admin",
"identity:get_endpoint": "rule:admin_required",
-> "identity:list_endpoints": "rule:admin_required or rule:member_role", <-
"identity:create_endpoint": "rule:cloud_admin",
"identity:update_endpoint": "rule:cloud_admin",
"identity:delete_endpoint": "rule:cloud_admin",
...
}

NOTE: in the Juju charm context, application configuration options should NOT be edited directly from within the application, rather the
configuration should be enforced via juju config command. In this case, policy editing was done in this way in order to see the exact
policies. To see how the keystone policies can be overridden, please go [Link] Policy overrides section.

Use your favorite text editor to modify the current policy to match the example. (e.g. vim, nano) Restarting Keystone is not required after
modifying the policy, as the policy gets computed at every request coming to keystone. Let's see if what you modified was enough for
user1 to get the endpoint list:

38 / 101
4. Keystone

openstack endpoint list

# output
You are not authorized to perform the requested action: identity:list_services. (HTTP 403)

Apparently not. There's also a list_services rule that also needs to be modified. That's because, even though you only asked for
endpoints, the output also contain other information, like the services available, service name, etc. That requires a subsequent request to
Keystone to fetch the information. Listing services is not yet allowed. You're going to modify the list_services rule the same way you
modified list_endpoints rule and try again. Upon successful modification, the new policy file should look like this:

juju ssh keystone/0 sudo head -30 /etc/keystone/[Link]

# output
{
"admin_required": "role:Admin",
-> "member_role": "role:member", <-
"cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:e5f4e8e...",
"service_role": "role:service",
"service_or_admin": "rule:admin_required or rule:service_role",
"owner": "user_id:%(user_id)s or user_id:%([Link].user_id)s",
"admin_or_owner": "(rule:admin_required and domain_id:%([Link])s) or rule:owner",
"admin_and_matching_domain_id": "rule:admin_required and domain_id:%(domain_id)s",
"service_admin_or_owner": "rule:service_or_admin or rule:owner",
"default": "rule:admin_required",
.....
"identity:get_service": "rule:admin_required",
-> "identity:list_services": "rule:admin_required or rule:member_role", <-
"identity:create_service": "rule:cloud_admin",
"identity:update_service": "rule:cloud_admin",
"identity:delete_service": "rule:cloud_admin",
"identity:get_endpoint": "rule:admin_required",
-> "identity:list_endpoints": "rule:admin_required or rule:member_role", <-
"identity:create_endpoint": "rule:cloud_admin",
"identity:update_endpoint": "rule:cloud_admin",
"identity:delete_endpoint": "rule:cloud_admin",

Again, restarting Keystone is not needed. Let's try and get the endpoint list again:

openstack endpoint list

Everything works now.

4.4 Logging

There are two types of logs you can check whenever investigating Keystone. Since it's a WSGI application, requests go through an
Apache2 web server, so all API requests go through Apache2 and logs can be found in /var/log/apache2 . Next, internal Keystone logs.
These can be found /var/log/keystone .

First, let's inspect Apache2 logs:

39 / 101
5. OpenStack Dashboard

juju ssh keystone/0 sudo tail -10 /var/log/apache2/keystone_access.log

# output
[Link] - - [07/Jul/2021:16:34:41 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 9168 "-" "python-keystoneclient"
[Link] - - [07/Jul/2021:16:34:41 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 9168 "-" "python-keystoneclient"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 9168 "-" "python-keystoneclient"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/projects/0fe82391a69849859ba1a8875bc953b2 HTTP/1.1" 200
659 "-" "nova keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/projects/36047d020f9d49a6ae7456c9777b91e9 HTTP/1.1" 200
654 "-" "nova keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/projects/6a48c2f51bda4506b55a44aeda6f4061 HTTP/1.1" 200
634 "-" "nova keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/projects/90cd0569f5f448bbb938c37bce8c4b4c HTTP/1.1" 200
659 "-" "nova keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/projects/97fff49051b6459cb93588b892bb7170 HTTP/1.1" 200
657 "-" "nova keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
[Link] - - [07/Jul/2021:16:34:42 +0000] "GET /v3/projects/e0b92e9a9bfd4c8abbda437b5a085bbe HTTP/1.1" 200
607 "-" "nova keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
[Link] - - [07/Jul/2021:16:36:16 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 9219 "-" "python-keystoneclient"

juju ssh keystone/0 sudo tail -10 /var/log/apache2/keystone_error.log

Next, inspecting keystone logs:

juju ssh keystone/0 sudo tail -10 /var/log/keystone/[Link]

# output
([Link]): 2021-07-07 15:18:35,112 WARNING Could not find domain: admin_domain.
([Link]): 2021-07-07 15:18:35,159 WARNING Could not find project: user2project.
([Link]): 2021-07-07 15:23:42,331 WARNING You are not authorized to perform
the requested action. ([Link]): 2021-07-07 15:31:04,736 WARNING You are not
authorized to perform the requested action: identity:list_endpoints.
([Link]): 2021-07-07 15:38:42,460 WARNING You are not authorized to perform
the requested action. ([Link]): 2021-07-07 15:41:38,249 WARNING You are not
authorized to perform the requested action: identity:list_services.
([Link]): 2021-07-07 15:53:42,586 WARNING You are not authorized to perform
the requested action.

You can also filter for different strings. Since you had some issues with listing endpoints and services, let's see what exactly was written in
the logs:

juju ssh keystone/0 "sudo grep 'list_services\|list_endpoints' /var/log/keystone/[Link]"

# output
([Link]): 2021-07-07 14:54:34,366 WARNING /usr/lib/python3/dist-packages/oslo_policy/[Link]:
UserWarning: Policy identity:list_endpoints failed scope check. The token used to make the request
was project scoped but the policy requires ['system'] scope.
This behavior may change in the future where using the intended scope is required
([Link]): 2021-07-07 14:54:34,390 WARNING /usr/lib/python3/dist-packages/oslo_policy/[Link]:
UserWarning: Policy identity:list_services failed scope check. The token used to make the request was
project scoped but the policy requires ['system'] scope.
This behavior may change in the future where using the intended scope is required
([Link]): 2021-07-07 15:31:04,736 WARNING You are not authorized to perform
the requested action: identity:list_endpoints.
([Link]): 2021-07-07 15:41:38,249 WARNING You are not authorized to perform
the requested action: identity:list_services.

5. OpenStack Dashboard
Openstack dashboard (Horizon) provides a graphical way of you to interact with your Openstack Cloud. Horizon is a Django-based project
aimed at providing a complete OpenStack Dashboard along with an extensible framework for building new dashboards from reusable
components.

Horizon is deployed by the "openstack-dashboard" charm. It can also be connected to HACluster charm to provide high availability and
load balancing between units.

40 / 101
5. OpenStack Dashboard

5.1 Changing themes

By default, Openstack Dashboard charms comes preconfigured with Canonical-branded interface:

This can be easily changed to the default one, by using simple juju config commands. First, let's inspect the current theme.

NOTE: your client host (e.g. laptop) does not have MAAS as DNS server, so you’ll have to use IP address to access Horizon. This can be
done by using the IP address of the charm, the IP can be taken from:

dig +short [Link]

# or
juju status openstack-dashboard

To change the theme to the default Openstack theme, please run:

41 / 101
5. OpenStack Dashboard

juju config openstack-dashboard default-theme=default

juju config openstack-dashboard ubuntu-theme=no

Let's watch the status for openstack-dashboard, until everything become stable:

juju status openstack-dashboard

# output
Model Controller Cloud/Region Version SLA Timestamp
default lxd-controller localhost/localhost 2.9.18 unsupported 10:25:22Z

App Version Status Scale Charm Store Channel Rev OS

dashboard-mysql-router 8.0.27 active 1 mysql-router charmstore stable 15 ubuntu
filebeat 5.6.16 active 1 filebeat charmstore stable 24 ubuntu
nrpe-container active 1 nrpe charmstore stable 75 ubuntu
openstack-dashboard 18.3.4 active 1 openstack-dashboard charmstore stable 318 ubuntu

Unit Workload Agent Machine Public address Ports Message

openstack-dashboard/0* active idle 27 [Link] 80/tcp,443/tcp Unit is ready
dashboard-mysql-router/0* active idle [Link] Unit is ready
filebeat/32 active idle [Link] Filebeat ready.
nrpe-container/32 active idle [Link] icmp,5666/tcp Ready

Machine State DNS Inst id Series AZ Message

27 started [Link] juju-33abf5-27 focal Running

Once Workload is active and Agent is idle , please refresh your tunneled browser:

42 / 101
5. OpenStack Dashboard

5.2 Logging
Since it's a web application, traffic to Horizon goes through a HTTP server, in this case Apache2. That means, logs are found in
/var/log/apache2 . Let's check the last few lines of both access and error logs.

43 / 101
6. Glance

juju ssh openstack-dashboard/0 sudo tail -10 /var/log/apache2/[Link]

# output
[Link] - - [23/Nov/2021:09:41:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:09:46:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:09:51:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:09:56:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:10:01:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:10:06:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:10:11:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2...)"
[Link] - - [23/Nov/2021:10:16:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:10:21:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."
[Link] - - [23/Nov/2021:10:26:47 +0000] "GET / HTTP/1.1" 200 931 "-" "check_http/v2.2..."

juju ssh openstack-dashboard/0 sudo tail -10 /var/log/apache2/[Link]

6. Glance
The Image service (glance) project provides a service where users can upload and discover data assets that are meant to be used with
other services. This currently includes images and metadata definitions.

Glance image services include discovering, registering, and retrieving virtual machine (VM) images. Glance has a RESTful API that allows
querying of VM image metadata as well as retrieval of the actual image.

In this chapter, you're going to add some images to Glance and set up custom properties, you're going to inspect its storage backend
configuration and actually find the image inside its current configuration and inspect Glance's logs.

Glance is composed of multiple services:

glance-api : accepts image API calls for image discovery, retrieval, and storage

glance-registry : stores, processes, and retrieves metadata about images. Metadata includes items such as size and type

5.1 Create images, set custom properties

Images uploaded in Glance can be either private to the user/project that uploaded it, or can be public images, so every other user/project
can use them for creating different instances. While working with images, we're going to use the administrator account and we're going to
create public images.

First, let's install qemu-utils package, that will provide different imaging tools, like qemu-img command.

sudo apt install -y qemu-utils

Then, you're going to create a local folder in which we're going to store temporary images before conversion and before uploading them to
Glance:

mkdir ~/cloud_images

cd ~/cloud_images

Next, we're going to fetch both Ubuntu 20.04 (focal) and 18.04 (bionic) minimal images from Canonical repository and we're going to
convert them from QCOW2 to RAW format before uploading them to Glance:

wget [Link]

44 / 101
6. Glance

File format is QCOW2 and virtual size is around 2.2GB. That means, whenever using this image, disk size needs to have a minimum of
2.2GB. Let's also convert the image to RAW instead of QCOW2:

qemu-img convert -f qcow2 -O raw [Link] [Link]

Now, let's do the same, but for a bionic image:

wget [Link]

qemu-img convert -f qcow2 -O raw [Link] [Link]

We should now have two QCOW2 images and two RAW images in our current folder. Let's see if that's the case:

ls -la
# output
-rw-rw-r-- 1 ubuntu ubuntu 186318848 Nov 19 09:19 [Link]
-rw-rw-r-- 1 ubuntu ubuntu 257425408 Nov 19 08:05 [Link]
-rw-r--r-- 1 ubuntu ubuntu 2361393152 Nov 23 10:37 [Link]
-rw-r--r-- 1 ubuntu ubuntu 2361393152 Nov 23 10:36 [Link]

Upload the images to Glance:

source ~/resources/admin_openrc

openstack image create --public --min-disk 3 --container-format bare \

--disk-format raw --property architecture=x86_64 \
--file ~/cloud_images/[Link] \
"focal"

openstack image create --public --min-disk 3 --container-format bare \

--disk-format raw --property architecture=x86_64 \
--file ~/cloud_images/[Link] \
"bionic"

cd ~

45 / 101
6. Glance

Images are now uploaded to Glance. Listing them is done using:

openstack image list

Details on every image can be seen with openstack image show <image_name/image_ID> .

openstack image show focal

openstack image show bionic

Make sure you check out the properties section of each image. Image properties are set using openstack image set command. For
some of them, there are flags available (e.g. --os-distro ) and for others, you'll need to specify --property flag.

You can set image properties that can be consumed by other services to affect the behavior of those other services. For example:

Image properties can be used to override specific behaviors defined for Nova flavors
Image properties can be used to affect the behavior of the Nova scheduler
Image properties can be used to affect the behavior of particular Nova hypervisors

Some of the most important properties:

architecture : the CPU architecture that must be supported by the hypervisor. Values: x86_64 , arm , or ppc64

hypervisor_type : the hypervisor type. Note that qemu is used for both QEMU and KVM hypervisor types. This is useful in mixed
envs for the nova scheduler. values: hyperv , ironic , lxc , qemu , uml , vmware , or xen
os_type : useful for libvirt . Values: linux , windows

os_distro : common name of the operating system distribution in lowercase: arch , debian , centos , etc.

hw_vif_model : specifies the model of virtual network interface device to use. Different hypervisors have different models available.
For example, for qemu: e1000 , virtio , rtl8139 , etc.

Here is a list with more properties: [Link]

You're going to assign some new properties to the "focal" image and then use openstack image show to identify those properties:

openstack image set --os-distro ubuntu --os-version 20.04 focal

openstack image set --property os_type=linux focal

openstack image show focal -c properties -f value

# output
...
{'store': 'ceph'}}], 'direct_url': 'rbd://d6d36554-def7-11eb-9d9d-00163e3b70d7/glance/...',
'architecture': 'x86_64', 'os_distro': 'ubuntu', 'os_version': '20.04', 'os_type': 'linux'}

5.2 Storage backends

Glance can use multiple backend storage devices for its images. It can also use multiple backends at the same time. For all-in-one
installations, it can use the unit's local storage, but for production/HA setup, it needs access to a shared storage device. Let's check its
current storage backend configuration:

46 / 101
7. Neutron

juju ssh glance/0 "sudo grep -A 15 glance_store /etc/glance/[Link]"

# output
[glance_store]
default_backend = ceph
filesystem_store_datadir = /var/lib/glance/images/
stores = [Link],[Link],[Link]
default_store = rbd
rbd_store_ceph_conf = /etc/ceph/[Link]
rbd_store_user = glance
rbd_store_pool = glance

So current configuration points Glance to Ceph backend storage. Using Ceph as a storage backend, Glance can either store images as
RBD (Rados Block Device) or inside RadosGW as object-storage. In this case, Glance is set up to use RBD. Glance Ceph setup is done
by relating glance charm to ceph-mon charm. Once they're related, a new glance pool is created inside Ceph, a new authentication
user is created, as well and glance charm configures itself to use the new Ceph setup.

Now, let's identify the images. For that, we first need the image IDs:

openstack image list

Next, let's list the objects inside glance pool in Ceph:

juju ssh ceph-mon/0 "sudo rbd ls glance"

# output
88906fc3-176a-4c00-b46b-85797ccc21fe
bdbc90f3-946f-4d0a-a696-4a550b4fd467

RBDs created inside Ceph have the same name as the ID Glace stores in database. We can check the details of each image usingrbd
info <pool>/<object> command:

juju ssh ceph-mon/0 "sudo rbd info glance/88906fc3-176a-4c00-b46b-85797ccc21fe"

# output
rbd image '88906fc3-176a-4c00-b46b-85797ccc21fe':
size 2.2 GiB in 282 objects
order 23 (8 MiB objects)
id: 1ceadc0ff481
block_name_prefix: rbd_data.1ceadc0ff481
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Nov 23 11:01:43 2021

5.3 Logging

Glance relevant logs are found in [Link] file in /var/log/glance . Can be inspected with:

juju ssh glance/0 tail -10 /var/log/glance/[Link]

7. Neutron
Neutron is an OpenStack project to provide “network connectivity as a service” between interface devices (e.g. vNICs) managed by other
OpenStack services (e.g. nova).

47 / 101
7. Neutron

OpenStack Networking, Neutron, allows you to create and attach interface devices managed by other OpenStack services to networks.
Plug-ins can be implemented to accommodate different networking equipment and software, providing flexibility to OpenStack architecture
and deployment.

To connect instances to the outside world, we'll need a provider network. Provider networks are managed by the administrator account.
Also, instances need to be connected to a project/tenant network. Access to each instance is done by setting up different security groups
with different rules. Last but not least, to access an instance from outside OpenStack, you'll need floating IPs. A floating IP is 1:1 NAT to a
private IP address of an instance.

Neutron is implemented by multiple services:

neutron-server : the API server

ovn-controller (ovn-central) : ovn-northd, the OVN central control daemon, and ovsdb-server, the Open vSwitch Database
(OVSDB)
ovn-chassis : OVN local controller and Open vSwitch Database and Switch

7.1 Network environment

On the host machine, OpenStack was deployed in LXD containers. The subnet used for the deployment is [Link]/24 , which means
OpenStack services respond on IPs from that network.

If you run the ip a command, a long list of interfaces is displayed. Those are the OpenStack service containers:

ip a

Take a look on the bridge the OpenStack services are running:

sudo ip addr show lxdbr0

# output
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:06:9b:42 brd ff:ff:ff:ff:ff:ff
inet [Link]/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe06:9b42/64 scope link
valid_lft forever preferred_lft forever

Take a look at the services from juju, IPs from the [Link]/24 subnet are used:

juju status

OpenStack has 4 distinct networks:

management: services talk to each other here

data/internal: VM to VM communication
exterior: exterior networking
storage: storage networking

The 4 networks can be on logically separated networks, some on the same network, or all on the same network. We will use the options
with all the networks in one.

Within the OpenStack cloud, there are two types of networks:

tenant : private network, created by users on a project

provider : are directly associated with a physical network in the data center, provisioned by a cloud admin.

7.2 Create the provider network

Let's first create a provider network. This is the external network used to connect instances to the outside world and vice versa.

48 / 101
7. Neutron

source ~/resources/admin_openrc

openstack network create public_network --external --provider-physical-network physnet1 \

--provider-network-type flat --mtu 1300

openstack subnet create --ip-version 4 --allocation-pool start=[Link],end=[Link] \

--gateway [Link] --no-dhcp --dns-nameserver [Link] --dns-nameserver [Link] \
--network public_network --subnet-range [Link]/24 public_subnet

Inspect the network:

openstack network list

# output
+--------------------------------------+----------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+----------------+--------------------------------------+
| 585d86c8-705a-448e-8e2c-8cd1ac03869d | public_network | 16445f41-0cf2-4ac1-a92f-244423ae898c |
+--------------------------------------+----------------+--------------------------------------+

openstack network show public_network

7.3 Create project networks

Now that you have a provider network, you also need a project network. This network will be created inside "user1project", that means
you're going to use user1_openrc file to authenticate to Keystone:

source ~/resources/user1_openrc

openstack network create user1_network --mtu 1300

49 / 101
7. Neutron

openstack subnet create --ip-version 4 \

--allocation-pool start=[Link],end=[Link] \
--gateway=[Link] --dhcp \
--dns-nameserver [Link] --dns-nameserver [Link] \
--subnet-range [Link]/24 \
--network user1_network user1_subnet

Inspect the networks:

openstack network list

openstack subnet list

7.4 Create a router, security groups, floating IPs

Inside user1project , you'll also need a router to connect the project network to the provider network.

openstack router create user1_router

openstack router set --external-gateway public_network user1_router
openstack router add subnet user1_router user1_subnet

Here's your setup:

50 / 101
7. Neutron

All projects have a default security group which is applied to any instance that has no other defined security group. Unless you change the
default, this security group denies all incoming traffic and allows only outgoing traffic to your instance.

All you need now, is to allow access to your instances on different protocols and ports. For that, you're going to create two security groups,
one allowing ICMP traffic, the other one allowing SSH traffic.

source ~/resources/user1_openrc

openstack security group create --description 'Allow ICMP Traffic' user1_allow_icmp

openstack security group rule create --proto icmp user1_allow_icmp
openstack security group rule create --proto icmp --egress user1_allow_icmp

51 / 101
7. Neutron

openstack security group create --description 'Allow SSH Traffic' user1_allow_ssh

openstack security group rule create --proto tcp --dst-port 22 user1_allow_ssh
openstack security group rule create --proto tcp --dst-port 22 --egress user1_allow_ssh

List the existing security groups:

openstack security group list

Now list the rules of the user1_allow_icmp and user1_allow_ssh security groups:

openstack security group rule list user1_allow_icmp

openstack security group rule list user1_allow_ssh

To access instances from outside Openstack, you'll need floating IPs. You're not going to use it right now, but let's create one that you're
going to use later on:

source ~/resources/user1_openrc

openstack floating ip create public_network

List the floating IPs:

openstack floating ip list

Ports can also be listed:

52 / 101
7. Neutron

openstack port list

# output
+--------------------------------------+------+-------------------+---------------------------------------------+
| ID | Name | MAC Address | Fixed IP Addresses |
+--------------------------------------+------+-------------------+---------------------------------------------+
| b6c36b76-7ae9-48b2-8247-d27eefadf395 | | fa:16:3e:a7:3e:e7 | ip_address='[Link]', subnet_id='46d...' |
| bb88c8b6-29dc-47c7-ab7b-5f5bca0fdd38 | | fa:16:3e:f5:a8:8f | ip_address='[Link]', subnet_id='1644...' |
| bbf94c24-fc04-436f-ab40-b3724a2df541 | | fa:16:3e:7a:b9:8f | ip_address='[Link]', subnet_id='46...' |
+--------------------------------------+------+-------------------+---------------------------------------------+

When a new VM is created, a new port will be created automatically by Neutron an added to the VM.

7.5 Neutron Plugin, type drivers and mechanism drivers

The Modular Layer 2 (ml2) plugin is a framework allowing OpenStack to use the variety of layer 2 networking technologies found in
complex real-world contexts.

ML2 has two types of drivers that it can configure:

Mechanism drivers : it defines the mechanism to access an OpenStack network of a certain type, examples: ovn , openvswitch ,
linuxbridge , and hyperv

Type Drivers : it defines how an OpenStack network is technically realized, examples: flat , vlan , geneve , vxlan , gre

The available configurations can be inspected in the /etc/neutron/plugins/ml2/ml2_conf.ini file on the neutron-api container:

53 / 101
7. Neutron

juju ssh neutron-api/0 sudo cat /etc/neutron/plugins/ml2/ml2_conf.ini

# output

[ml2]
type_drivers = geneve,gre,vlan,flat,local
tenant_network_types = geneve,gre,vlan,flat,local
mechanism_drivers = ovn

path_mtu = 1500
[ml2_type_gre]
tunnel_id_ranges = 1:1000

[ml2_type_vxlan]
vni_ranges = 1001:2000

[ml2_type_vlan]
network_vlan_ranges = physnet1:1000:2000

[ml2_type_flat]
flat_networks = physnet1

[ovs]
enable_tunneling = True
local_ip = [Link]

[agent]
tunnel_types = gre
[securitygroup]
enable_security_group = True
[ovn]
ovn_nb_connection = ssl:[Link]:6641,ssl:[Link]:6641,ssl:[Link]:6641
ovn_nb_private_key = /etc/neutron/plugins/ml2/key_host
ovn_nb_certificate = /etc/neutron/plugins/ml2/cert_host
ovn_nb_ca_cert = /etc/neutron/plugins/ml2/[Link]
ovn_sb_connection = ssl:[Link]:16642,ssl:[Link]:16642,ssl:[Link]:16642
ovn_sb_private_key = /etc/neutron/plugins/ml2/key_host
ovn_sb_certificate = /etc/neutron/plugins/ml2/cert_host
ovn_sb_ca_cert = /etc/neutron/plugins/ml2/[Link]
ovn_l3_scheduler = leastloaded
ovn_metadata_enabled = True
enable_distributed_floating_ip = False
dns_servers =
dhcp_default_lease_time = 43200
ovn_dhcp4_global_options =
ovn_dhcp6_global_options =
vhost_sock_dir = /run/libvirt-vhost-user

[ml2_type_geneve]
vni_ranges = 1001:2000
max_header_size = 38

The following types are available:

local : works only on a single host, used in POCs

flat : no segmentation, traditional L2 network, anyone can see the broadcast traffic

vlan : VLAN segmentation takes place, VLAN ranges are specified network_vlan_ranges

geneve : overlay and traffic encapsulation and also segmentation used by OVN, ranges are specified vni_ranges

vxlan : overlay and traffic encapsulation and also segmentation, ranges are specified vni_ranges

gre : similar to vxlan

So what is physnet1 ? We used it when we provision the external network, it's an OpenStack internal object. This is a mapping to an
virtual switch provisioned by OpenStack. This bridge will be then connected to the external interface.

The virtual switch is called br-ex . So this means that every network we create and is attached to physnet1 will be attached to the
provider network.

54 / 101
7. Neutron

br-ex is connected to eth1 network interface, which is usually connected to the LAN network in the datacenter. All external traffic flows
via this interface. We will see more of this in the Open vSwitch chapter.

This was set in deploy time from the ovn-chassis charm:

juju config ovn-chassis ovn-bridge-mappings

# output
physnet1:br-ex

We can see the OVS bridged and the connection between br-ex and eth1 :

55 / 101
7. Neutron

juju ssh ovn-chassis/0 sudo ovs-vsctl show

# output
10030ef8-aa12-48a8-ad19-e7d9f0d37fab
Manager "ptcp:6640:[Link]"
is_connected: true
Bridge br-ex
fail_mode: standalone
datapath_type: system
Port eth1
Interface eth1
type: system
Port br-ex
Interface br-ex
type: internal
Port patch-provnet-b913001f-1784-4b51-a6b3-3a9b4aaccfa6-to-br-int
Interface patch-provnet-b913001f-1784-4b51-a6b3-3a9b4aaccfa6-to-br-int
type: patch
options: {peer=patch-br-int-to-provnet-b913001f-1784-4b51-a6b3-3a9b4aaccfa6}
Bridge br-int
fail_mode: secure
datapath_type: system
Port patch-br-int-to-provnet-b913001f-1784-4b51-a6b3-3a9b4aaccfa6
Interface patch-br-int-to-provnet-b913001f-1784-4b51-a6b3-3a9b4aaccfa6
type: patch
options: {peer=patch-provnet-b913001f-1784-4b51-a6b3-3a9b4aaccfa6-to-br-int}
Port br-int
Interface br-int
type: internal
ovs_version: "2.13.3"
Connection to [Link] closed.

7.6 OVN

Everything we created earlier, networks, subnets, routers, can be found under OVN. To connect to each of OVN's DBs, it would be easier
to connect to the unit running the database. Let's assign those units to two different variables:

56 / 101
7. Neutron

OVN_SB_UNIT=$(juju status ovn-central | grep ovnsb_db | tail -1 | awk '{print $1}' | cut -f 1 -d "*")
OVN_NB_UNIT=$(juju status ovn-central | grep ovnnb_db | tail -1 | awk '{print $1}' | cut -f 1 -d "*")

echo $OVN_SB_UNIT
echo $OVN_NB_UNIT

On the SB unit, let's check ovn-sbctl show command:

juju ssh $OVN_SB_UNIT sudo ovn-sbctl show

# output
Chassis [Link]
hostname: [Link]
Encap geneve
ip: "[Link]"
options: {csum="true"}
Port_Binding cr-lrp-93b105da-1b27-42e3-9eb9-24365e727840

On the NB unit, use the ovn-nbctl utility to see an overview of the logical topology:

juju ssh $OVN_NB_UNIT sudo ovn-nbctl show

# output
switch 0b0e4f75-ad40-470c-bb05-623c2e52380b (neutron-a99f47d0-f853-4b63-8433-a866ba5b7f86) (aka user1_network)
port b6c36b76-7ae9-48b2-8247-d27eefadf395
type: router
router-port: lrp-b6c36b76-7ae9-48b2-8247-d27eefadf395
port bbf94c24-fc04-436f-ab40-b3724a2df541
type: localport
addresses: ["fa:16:3e:7a:b9:8f [Link]"]
switch 7efba977-af0a-4f77-b9df-ecc49c7f5166 (neutron-585d86c8-705a-448e-8e2c-8cd1ac03869d) (aka public_network)
port 5b8fe496-6a06-49c0-9971-ed3437ff01bf
type: localport
addresses: ["fa:16:3e:16:ae:74"]
port provnet-9085584b-0188-48ee-9003-117b93e6ea5d
type: localnet
addresses: ["unknown"]
port 93b105da-1b27-42e3-9eb9-24365e727840
type: router
router-port: lrp-93b105da-1b27-42e3-9eb9-24365e727840
router be54a25e-cc4c-4881-ae76-4b080f9d5071 (neutron-bb2f3826-ca3a-4271-8e11-2afd4ab7fe23) (aka user1_router)
port lrp-b6c36b76-7ae9-48b2-8247-d27eefadf395
mac: "fa:16:3e:a7:3e:e7"
networks: ["[Link]/24"]
port lrp-93b105da-1b27-42e3-9eb9-24365e727840
mac: "fa:16:3e:b6:ed:2d"
networks: ["[Link]/24"]
gateway chassis: [[Link]]
nat e557b311-d6ff-4de2-99fd-eaa45131a0eb
external ip: "[Link]"
logical ip: "[Link]/24"
type: "snat"

Using ovn-northd , OVN translates the NB DB's high-level switch and router concepts into lower-level concepts of "logical datapaths" and
logical flows. There's one logical datapath for each logical switch or router. Logical datapaths are the logical version of an OpenFlow
switch. Logical switches and routers are both implemented as logical datapaths:

57 / 101
7. Neutron

juju ssh $OVN_SB_UNIT sudo ovn-sbctl list datapath_binding

# output
_uuid : bab08a92-c63d-40ba-8307-85f9e032bbb0
external_ids : {logical-switch="0b0e4f75-ad40-...", name=neutron-a99f47d0-..., name2=user1_network}
tunnel_key : 2

_uuid : f91f903b-349d-46f2-b71d-73f1c3708179
external_ids : {logical-switch="7efba977-af0a-...", name=neutron-585d86c8-..., name2=public_network}
tunnel_key : 1

_uuid : 65b128bb-7e75-4f7e-a1c1-348f83878b76
external_ids : {logical-router="be54a25e-cc4c-...", name=neutron-bb2f3826-..., name2=user1_router}
tunnel_key : 3

The logical flows of each datapath can be inspected. OVN creates logical flows to describe how the network should behave in logical
space. Each chassis then creates OpenFlow flows based on those logical flows that reflect its own local view of the network. The ovn-
sbctl command can show the logical flows:

juju ssh $OVN_SB_UNIT sudo ovn-sbctl lflow-list bab08a92-c63d-40ba-8307-85f9e032bbb0

As a final demonstration of the OVN architecture, let's examine the DHCP implementation. Like switching, routing, and NAT, the OVN
implementation of DHCP involves configuration in the NB DB and logical flows in the SB DB. We're going to use the UUID for the
user1_network to query all the flows.

juju ssh $OVN_NB_UNIT sudo ovn-nbctl list dhcp_options

# output
_uuid : fdcf2353-2994-436d-bf5f-fac3f3c45a03
cidr : "[Link]/24"
external_ids : {"neutron:revision_number"="0", subnet_id="46d31a6f-27c8-43dc-9c58-1b49d327dbb7"}
options : {classless_static_route="{[Link]/32,[Link], [Link]/0,[Link]}", ...}

Southbound DB, use the datapath binding uuid of the user1 network:

juju ssh $OVN_SB_UNIT sudo ovn-sbctl lflow-list bab08a92-c63d-40ba-8307-85f9e032bbb0 | grep dhcp

# output
table=14(ls_in_dhcp_options ), priority=0 , match=(1), action=(next;)
table=15(ls_in_dhcp_response), priority=0 , match=(1), action=(next;)

7.7 Logging
There are three types of units that can provide useful logging:

neutron-api
ovn-central

58 / 101
8. Nova

ovn-chassis
neutron-api - provides Neutron API services. It's logs are located in /var/log/neutron , and can be inspected using:

juju ssh neutron-api/0 "sudo tail -20 /var/log/neutron/[Link]"

ovn-central - provides OVN SB and NB databases. Not all of them are running on all of its units. They work in active/passive manner.
Their logs can be inspected using:

juju run -a ovn-central "sudo tail -20 /var/log/ovn/[Link]"

juju run -a ovn-central "sudo tail -20 /var/log/ovn/[Link]"
juju run -a ovn-central "sudo tail -20 /var/log/ovn/[Link]"

ovn-chassis - provides configuration for OVS on either dedicated network nodes or compute nodes. Logs can be inspected using:

juju run -a ovn-chassis "sudo tail -20 /var/log/ovn/[Link]"

juju run -a ovn-chassis "sudo tail -20 /var/log/openvswitch/[Link]"
juju run -a ovn-chassis "sudo tail -20 /var/log/openvswitch/[Link]"

8. Nova
Nova is the OpenStack project that provides a way to provision compute instances (aka virtual servers). Nova supports creating virtual
machines, bare metal servers (through the use of ironic), and has limited support for system containers. Nova runs as a set of daemons on
top of existing Linux servers to provide that service.

In this chapter, we're going to create our first instance. But, before doing that, we'll need to create some flavors, that will define the sizes of
every instance created. Also, we'll need keypairs, so we'll be able to access the instances once created. Cloud images usually don't come
with a default password, so unless we add a keypair to Openstack, we won't be able to access the instances. Custom images can also be
uploaded to Glance and you can use external tools like Packer to create images with custom default passwords. This is not recommended
from a security point of view.

Flavors are managed by the administrator account, so you're going to create them using "admin_openrc" file. Keypairs and instances,
you're going to add them inside "user1project", and that means you're going to use "user1_openrc" file.

Nova is composed of multiple services:

nova-api : accepts and responds to end user compute API calls

nova-compute : daemon that talks directly to the hypervisor

nova-scheduler : determines on which compute host a VM should be scheduled

nova-conductor : mediates interactions between the nova-compute service and the database

nova-novncproxy : provides a proxy for accessing running instances through a VNC connection. Supports browser-based novnc
clients.
nova-xvpvncproxy : provides a proxy for accessing running instances through a VNC connection. Supports an OpenStack-specific
Java client.
nova-spicehtml5proxy : provides a proxy for accessing running instances through a SPICE connection. Supports browser-based
HTML5 client.
nova-consoleauth : authorizes tokens for users that console proxies provide.

List the Nova services:

59 / 101
8. Nova

source ~/resources/admin_openrc

openstack compute service list

8.1 Creating flavors and keypairs

First, let's create some flavors:

openstack flavor create --vcpus 1 --ram 1024 --disk 5 --ephemeral 0 --swap 0 --public [Link]
openstack flavor create --vcpus 2 --ram 2048 --disk 10 --ephemeral 0 --swap 0 --public [Link]
openstack flavor create --vcpus 4 --ram 4096 --disk 20 --ephemeral 0 --swap 0 --public [Link]

Flavors are set. You now need a keypair, for you to be able to access your instances:

ssh-keygen -t rsa -N "" -q -f ~/.ssh/user1_id_rsa

source ~/resources/user1_openrc

openstack keypair create user1-keypair --public-key ~/.ssh/user1_id_rsa.pub

Now that you have everything set up, you need to create an instance and make sure you are able to access it through a floating IP (the
one generated during the Neutron lab).

List the keypair:

openstack keypair list

8.2 Creating instances

Creating an instance is done by using openstack server create command:

source ~/resources/user1_openrc

openstack server create --availability-zone nova --image 'focal' \

--flavor [Link] --key-name user1-keypair --security-group user1_allow_icmp \
--nic net-id=$(openstack network list | grep user1_network | awk '{print $2}') focal1

The server is building, wait a bit for the process to finish, notice the BUILD status:

60 / 101
8. Nova

openstack server list

Add a floating IP to the instance:

export INSTANCE_FLOATING_IP=$(openstack floating ip list | grep 10 | awk '{ print $4 }')

openstack server add floating ip focal1 $INSTANCE_FLOATING_IP

To verify the floating IP was actually assigned to the instance, please run:

openstack floating ip list

To verify connectivity to your instance:

ping $INSTANCE_FLOATING_IP

Let's also try SSH:

ssh -i ~/.ssh/user1_id_rsa $INSTANCE_FLOATING_IP

This command will time out with the following message:

ssh: connect to host [Link] port 22: Connection timed out

That's because, when you created the instance, you only assigned the security group allowing for ICMP traffic (ping), but not SSH. You
need to assign the second security group, as well:

openstack server add security group focal1 user1_allow_ssh

After running the command, let's try and login again:

ssh -i ~/.ssh/user1_id_rsa $INSTANCE_FLOATING_IP

This instance is a minimal Ubuntu instance. That means, it's missing simple commands, like ping . To have it available, we're going to
install two new packages, and we're going to test connectivity to the outside world:

sudo apt update

sudo apt install -y iputils-ping
ping [Link] -c 10

You should get replies from Cloudflare's DNS servers.

Exit the instance when you are done.

exit

8.3 Creating snapshots

Instances are the individual virtual machines that run on physical compute nodes inside the cloud. Users can launch any number of
instances from the same image. Each launched instance runs from a copy of the base image. Any changes made to the instance do not

61 / 101
8. Nova

affect the base image. Snapshots capture the state of an instances running disk. Users can create a snapshot, and build a new image
based on these snapshots. The Compute service controls instance, image, and snapshot storage and management.

Snapshots are created with the instance turned off. Snapshots are useful whenever you're trying to migrate instances to another cloud or
for backup purposes. Also, you can use them as "golden templates" whenever doing security tuning for a default image and you don't want
to do that again. The new snapshot will be uploaded to Glance as a new image.

First, let's stop the instance and verify its state is SHUTOFF :

openstack server stop focal1

openstack server list -c ID -c Name -c Status -f table

Next, create a snapshot using openstack server image create and then verify it's there:

openstack server image create --name focal1_snapshot focal1

The new snapshot will be uploaded to Glance as a new image.

Wait for the image creation process to finish, notice the saving status, wait for it to become active :

openstack image list

Now, start the VM:

openstack server start focal1

You can now save the image locally:

openstack image save --file [Link] focal1_snapshot

You can inspect the image using qemu-img command:

8.4 The metadata service

Nova presents configuration information to instances via a mechanism called metadata . These mechanisms are used by tools such as
cloud-init to specify things like hostname, root password the instance should use.

There are two types of data that can be passed to instances:

62 / 101
8. Nova

user provided data : ssh keys, scripts

nova provided data : hostname, IPs, MTU, availability zone

vendor provided data : AD post boot cryptographic token to register the instance

This data is made available via either a config drive or the metadata service and can be somewhat customized by the user using
the user data feature.

The metadata service provides a way for instances to retrieve instance-specific data via a REST API. Instances access this service at
[Link] and all types of metadata, be it user, nova or vendor provided, can be accessed via this service. Note that the
metadata API is a versioned API.

Connect to focal1 VM:

ssh -i ~/.ssh/user1_id_rsa $INSTANCE_FLOATING_IP

The different API version information is displayed, we'll use the latest one.

List what is the latest path:

curl [Link]
# output
meta_data.json
password
vendor_data.json
network_data.json
vendor_data2.json$

We are interested on the meta_data.json and network_data.json files:

curl [Link]

This is how the instance gets the information and cloud-init will use this info upon the boot process.

Exit the instance:

exit

8.5 The config drive

Config drives are special drives that are attached to an instance when it boots. The instance can mount this drive and read files from it
to get information that is otherwise available through the metadata service .

One use case for using the config drive is to pass a networking configuration when you do not use DHCP to assign IP addresses to

63 / 101
8. Nova

instances. For example, you might pass the IP address configuration for the instance through the config drive, which the instance can
mount and access before you configure the network settings for the instance.

To enable the config drive for an instance, pass the --config-drive true parameter to the openstack server create command.

Create a new instance using the config drive:

source ~/resources/user1_openrc

openstack server create --availability-zone nova --image 'focal' --config-drive true \

--flavor [Link] --key-name user1-keypair --security-group user1_allow_ssh \
--nic net-id=$(openstack network list | grep user1_network | awk '{print $2}') focal2

Create a new floating IP and associate it with the new instance:

openstack floating ip create public_network

openstack floating ip list

openstack server add floating ip focal2 <free floating IP>

openstack server list

Connect to the new instance:

ssh -i ~/.ssh/user1_id_rsa <free floating IP>

List the block devices present:

sudo lsblk
# output
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 486K 0 rom
vda 254:0 0 10G 0 disk
├─vda1 254:1 0 9.9G 0 part /
├─vda14 254:14 0 4M 0 part
└─vda15 254:15 0 106M 0 part /boot/efi

sr0 is the block device that was used to pass the metadata information during boot. Mount it and see what is inside.

sudo mkdir -p /mnt/config

sudo mount /dev/sr0 /mnt/config/
ls -l /mnt/config/openstack/

As you can see, the metadata info is present.

Exit the instance:

exit

8.6 User data

The user-data key is a special key in the metadata service that holds a file that cloud-aware applications within the guest instance
can access. For example, cloud-init is an open source package from Ubuntu, but available in most distributions, that handles early
initialization of a cloud instance that makes use of this user data .

64 / 101
8. Nova

This user data can be put in a file on your local system and then passed in at instance creation with the flag--user-data <user-data-
file> . The typical use case is to pass something like a shell script or a configuration file as user data.

Create a script that runs on the first instance boot. Call it [Link] and make it executable:

touch ~/[Link]
chmod +x ~/[Link]

Add the following content to the script:

sudo tee ~/[Link] <<EOF

#!/bin/sh
echo "Hello World!" | tee /home/ubuntu/[Link]
EOF

Create a new instance and check if the script was executed:

openstack server create --availability-zone nova --image 'focal' --user-data ./[Link] \

--flavor [Link] --key-name user1-keypair --security-group user1_allow_ssh \
--nic net-id=$(openstack network list | grep user1_network | awk '{print $2}') focal3

Create a new floating IP and associate it with the new instance:

openstack floating ip create public_network

openstack floating ip list

openstack server add floating ip focal3 <free floating IP>

openstack server list

Connect to the new instance:

ssh -i ~/.ssh/user1_id_rsa <free floating IP>

Once connected, check of the script was executed:

cat /home/ubuntu/[Link]
# output
Hello World!

Exit the instance:

exit

Delete instance focal2 and focal3:

openstack server delete focal2 focal3

8.7 Console access, console logs and events

VNC or SPICE are used to view the console output of an instance. This allows relaying keyboard and mouse activity to and from an
instance.

65 / 101
8. Nova

There are three remote console access methods commonly used with OpenStack:

novnc: in-browser VNC client implemented using HTML5 Canvas and WebSockets
spice: complete in-browser client solution for interaction with virtualized instances
xvpvnc: a Java client offering console access to an instance

To get the console logs of an instance. The logs can be inspected to see if the instance booted properly:

openstack console log show focal1

The instance events can be inspected:

openstack server event list focal1

8.8 Hypervisors

KVM + QEMU is used as hypervisor. The virtual disk formats that it supports is inherited from QEMU since it uses a modified QEMU
program to launch the virtual machine. The supported formats include raw images, the qcow2, and VMware formats.

From the perspective of the Nova, the QEMU hypervisor is very similar to the KVM hypervisor. Both are controlled through libvirt , both
support the same feature set, and all virtual machine images that are compatible with KVM are also compatible with QEMU. The main
difference is that QEMU does not support native virtualization. Consequently, QEMU has worse performance than KVM and is a poor
choice for a production deployment.

The compute node is where the nova-compute service is running, talking directly with QEMU.

List the and inspect the hypervisor:

source ~/resources/admin_openrc

openstack hypervisor list

openstack hypervisor show [Link]

Hypervisor stats details can also be displayed:

openstack hypervisor stats show

# output
+----------------------+-------+
| Field | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 110 |
| free_disk_gb | 115 |
| free_ram_mb | 61760 |
| local_gb | 125 |
| local_gb_used | 10 |
| memory_mb | 64320 |
| memory_mb_used | 2560 |
| running_vms | 1 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+

66 / 101
8. Nova

8.9 Overcommitting resources

OpenStack allows you to overcommit CPU and RAM on compute nodes. This allows you to increase the number of instances running on
your cloud at the cost of reducing the performance of the instances. The Compute service uses the following ratios:

cpu_allocation_ratio : the per physical core to virtual core ratio to use in the Nova scheduler.

disk_allocation_ratio : increase the amount of disk space that nova can overcommit to guests.

ram_allocation_ratio : the physical ram to virtual ram ratio to use in the Nova scheduler.

Increasing the above values will increase instance density on compute nodes at the potential expense of instance performance.

Check the current ration:

juju ssh nova-cloud-controller/0 sudo cat /etc/nova/[Link] | grep allocation

# output
cpu_allocation_ratio = 16
ram_allocation_ratio = 1
disk_allocation_ratio = 1

The default CPU allocation ratio of 16:1 means that the scheduler allocates up to 16 virtual cores per physical core.

8.10 Config and log files

We have Nova services deployed in two distinct containers nova-cloud-controller and nova-compute . Both containers have the
config files in the same location, /etc/nova/ .

List the config files:

juju ssh nova-cloud-controller/0 sudo ls -l /etc/nova/

juju ssh nova-cloud-controller/0 sudo cat /etc/nova/[Link]

juju ssh nova-compute/0 sudo ls -l /etc/nova/

Check the virtualization type in nova-compute service:

juju ssh nova-compute/0 sudo cat /etc/nova/[Link]

# output
[libvirt]
virt_type=kvm
...

For logging, log in the Nova compute container and make yourself root:

juju ssh nova-compute/0 sudo su -

Change directory to the logs:

cd /var/log/nova/
ls -l

Interactively tail all the logs:

tail -f ./*.log

Hit CTRL+C to stop the output.

67 / 101
9. Cinder

After you are done with the inspection, go back to the host:

exit

9. Cinder
Cinder is a Block Storage service for OpenStack. It's designed to present storage resources to end users that can be consumed by the
OpenStack Compute Project (Nova). This is done through the use of either a reference implementation (LVM) or plugin drivers for other
storage. The short description of Cinder is that it virtualizes the management of block storage devices and provides end users with a self
service API to request and consume those resources without requiring any knowledge of where their storage is actually deployed or on
what type of device

In this chapter, you're going to create a volume, attach it to an existing instance, resize it, take a snapshot and transfer the volume
between projects.

Cinder is composed of multiple services:

cinder-api : accepts API requests, and routes them to the cinder-volume for action

cinder-volume : interacts directly with the Block Storage backend service

cinder-scheduler : selects the optimal storage provider node on which to create the volume

cinder-backup : provides backing up volumes of any type to a backup storage provider

List the available services:

68 / 101
9. Cinder

source ~/resources/admin_openrc

openstack volume service list

9.1 Creating volumes

Creating volumes is done by using openstack volume create command. You'll need to specify the name of that volume and the size.
Optionally, if using multiple storage backends, you'll also need to specify a volume type.

source ~/resources/user1_openrc

openstack volume create --size 5 user1_volume

Once you have the volume created, let's attach it to your existing instance and let's verify the device is present in the OS:

openstack server add volume focal1 user1_volume --device /dev/vdb

export INSTANCE_FLOATING_IP=$(openstack server list | grep focal1 | awk '{ print $9 }')
ssh -i ~/.ssh/user1_id_rsa $INSTANCE_FLOATING_IP

Once connected to your instance:

69 / 101
9. Cinder

sudo apt update

sudo apt install -y parted
sudo fdisk -l
sudo mkdir /mnt/vdb
sudo parted /dev/vdb mklabel gpt
sudo parted -a optimal /dev/vdb mkpart primary ext4 1 100%
sudo mkfs.ext4 /dev/vdb1
sudo mount /dev/vdb1 /mnt/vdb

df -h

sudo umount /dev/vdb1

exit

9.2 Resizing volumes

Last command should show you the 5GB device being mounted in /mnt/vdb. Let's resize the volume to 10GB. For that, you'll need to
follow these steps:

un-attach the volume

resize the volume (cinder)
re-attach the volume
resize partition
resize filesystem

First, let's un-attach the volume:

openstack server remove volume focal1 user1_volume

Then, resize the volume in Cinder:

openstack volume set user1_volume --size 10

And then, re-attach the volume to the instance:

openstack server add volume focal1 user1_volume --device /dev/vdb

ssh -i ~/.ssh/user1_id_rsa $INSTANCE_FLOATING_IP

sudo parted /dev/vdb resizepart 1 100%

sudo e2fsck -f /dev/vdb1
sudo resize2fs /dev/vdb1
sudo mount /dev/vdb1 /mnt/vdb

df -h

You should now see a 10GB partition being mounted as /mnt/vdb.

exit

9.3 Snapshots

A Cinder snapshot is a point-in-time, read-only copy of a Cinder volume. Snapshots can be created from an existing Cinder volume that is
operational and either attached to an instance or in a detached state. A Cinder snapshot can serve as the content source for a new Cinder

70 / 101
9. Cinder

volume when the Cinder volume is created with the create from snapshot option specified.

To create a snapshot, simply run openstack volume snapshot create command:

openstack volume snapshot create --volume user1_volume --force user1_volume_snapshot

To list all the snapshots available:

openstack volume snapshot list

To show details about a specific snapshot:

openstack volume snapshot show user1_volume_snapshot

To create a new volume based on a snapshot:

openstack volume create --snapshot user1_volume_snapshot --size 15 user1_volume2

Let's see the details of the volume:

71 / 101
9. Cinder

openstack volume show user1_volume2

9.4 Volume transfers

You can transfer ownership of a volume from one project to another one. This is one way to move data from one project to another one,
since it's not something you could actually do, without volume transfer. Since we have two projects created, we're going to create a volume
in one of them, transfer it to the other one and see if we can manage the volume from the second project.

First, let's create the initial volume:

source ~/resources/user1_openrc

openstack volume create --size 10 user1_volume3

openstack volume list
openstack volume show user1_volume3

Next, create a volume transfer request:

openstack volume transfer request create --name transfer_volume3 user1_volume3

Please write down the auth_key and id fields. You're going to use both in the next commands. After creating the volume transfer
request, you need to log in as the other user (user2) and accept the transfer. You'll need both auth_key and transfer request id . To
accept the transfer, you'll need to run a openstack volume transfer request accept command.

source ~/resources/user2_openrc

openstack volume transfer request accept --auth-key <auth_key> <volume_transfer_ID>

72 / 101
9. Cinder

Example

source ~/resources/user2_openrc

openstack volume transfer request accept --auth-key 71aa1a71eb56db51 7d107001-2787-49e4-b951-769d4f307a57

Now the volume (user1_volume3) is owned by user2.

Verify who has ownership for the volume:

# as user2
source ~/resources/user2_openrc

openstack volume list

# as user1
source ~/resources/user1_openrc

openstack volume list

9.5 Bootable volumes

An instance can be created from a Glance image. In this case the root disk, sda , is not persistent, meaning that if the instance is deleted,
the disk goes as well. Cinder has no notion of this disk.

Another way to create an instance is to boot it from a volume.

Create a bootable volume from the existing focal image:

source ~/resources/user1_openrc

openstack volume create --image focal --size 5 focal_boot_volume

List and inspect the volume:

openstack volume list

openstack volume show focal_boot_volume

The bootable is set to True , making the volume a bootable volume.

Create a new instance from the volume:

openstack server create --availability-zone nova --volume 'focal_boot_volume' \

--flavor [Link] --key-name user1-keypair --security-group user1_allow_icmp \
--nic net-id=$(openstack network list | grep user1_network | awk '{print $2}') focal4

In this case, instead of --image , we used --volume .

List and inspect the instance, wait for it to become active :

73 / 101
9. Cinder

openstack server list

openstack server show focal4

Inspect the volume, should be in-use :

openstack volume list

Delete the VM, this will make the volume available again:

openstack server delete focal4

openstack volume list

Volumes can also be uploaded to Glance as Images, also called volume-backed images.

Create a new image from the bootable volume. The command template is openstack image create --volume <source_volume>
<image_name> :

openstack image create --volume focal_boot_volume focal_volume_image

List and inspect the image:

openstack image list

The volume is currently being saved in Glance. This process can take a minute or two, depending on the volume size.

Inspect the image:

openstack image show focal_volume_image

9.6 Identify storage backends configuration

74 / 101
10. Ceph

Once the Cinder charm is related to Ceph, its configuration is set in a way to be able to authenticate and communicate with Ceph monitors
and OSDs. Also, Ceph configures a pool in which Cinder can store its volumes. Let's check current configuration:

juju ssh cinder/0 "sudo grep -A 15 enabled_backends /etc/cinder/[Link]"

# output

enabled_backends = cinder-ceph

[cinder-ceph]
volume_backend_name = cinder-ceph
volume_driver = [Link]
rbd_pool = cinder-ceph
rbd_user = cinder-ceph
rbd_secret_uuid = ddabb334-85fc-46d6-a16e-b8f7a00cdf47
rbd_ceph_conf = /var/lib/charm/cinder-ceph/[Link]
report_discard_supported = True
rbd_exclusive_cinder_pool = True
rbd_flatten_volume_from_snapshot = False

So, it's using cinder-ceph as the default backend, and that default backend is using a Ceph pool called cinder-ceph . If you want to
check all the pools created inside Ceph:

juju ssh ceph-mon/0 sudo ceph osd lspools

# output

1 [Link]
2 [Link]
3 [Link]
4 [Link]
5 [Link]
6 [Link]-log
7 [Link]
8 [Link]
9 [Link]
10 [Link]
11 [Link]
12 [Link]
13 [Link]
14 [Link]
15 .[Link]
16 gnocchi
17 glance
18 cinder-ceph

9.7 Logging

There are two services providing Cinder, cinder-scheduler and cinder-volume , each of them having their own log file:

/var/log/cinder/[Link]
/var/log/cinder/[Link]

To inspect the last few lines of each of the log files:

juju ssh cinder/0 "sudo tail -10 /var/log/cinder/[Link]"

10. Ceph
Ceph is an open-source, software-defined and distributed storage system. Ceph is a unified storage solution that provides access to files,
blocks as well as objects from a single platform along with their storage

In this chapter, you're going to manage OSDs, check Ceph versions for each component and check the monitor map and CRUSH map.
Monitor map is useful whenever you're not able to recover the majority of monitors, they don't have quorum, and you absolutely need to

75 / 101
10. Ceph

bring back your cluster. The CRUSH map decides where to store data based on some rules. Some of the rules are there by default and
can be modified by using Ceph CLI, and some can be modified directly by modifying the CRUSH map. Once modified, it needs to be
recompiled and fed back to the Ceph monitors.

10.1 Add/remove OSDs

First. let's get the status of all Ceph related charms: ceph-mon, ceph-osd and ceph-radosgw.

76 / 101
10. Ceph

juju status ceph-mon ceph-osd ceph-radosgw

# output
Model Controller Cloud/Region Version SLA Timestamp
default lxd-controller localhost/localhost 2.9.19 unsupported 13:17:05Z

App Version Status Scale Charm Store Channel Rev OS Message

ceph-mon 13.2.9 active 3 ceph-mon charmstore stable 61 ubuntu Unit is ...
ceph-osd 13.2.9 active 3 ceph-osd charmstore stable 316 ubuntu Unit is ...
ceph-radosgw 13.2.9 active 1 ceph-radosgw charmstore stable 300 ubuntu Unit is ...
filebeat 5.6.16 active 7 filebeat charmstore stable 24 ubuntu Filebeat ...
nrpe-container active 7 nrpe charmstore stable 75 ubuntu Ready

Unit Workload Agent Machine Public address Ports Message

ceph-mon/0* active idle 1 [Link] Unit is ready and clustered
filebeat/14 active idle [Link] Filebeat ready.
nrpe-container/14 active idle [Link] icmp,5666/tcp Ready
ceph-mon/1 active idle 2 [Link] Unit is ready and clustered
filebeat/15 active idle [Link] Filebeat ready.
nrpe-container/16 active idle [Link] icmp,5666/tcp Ready
ceph-mon/2 active idle 3 [Link] Unit is ready and clustered
filebeat/22 active idle [Link] Filebeat ready.
nrpe-container/22 active idle [Link] icmp,5666/tcp Ready
ceph-osd/0 active idle 4 [Link] Unit is ready (1 OSD)
filebeat/17 active idle [Link] Filebeat ready.
nrpe-container/17 active idle [Link] icmp,5666/tcp Ready
ceph-osd/1* active idle 5 [Link] Unit is ready (1 OSD)
filebeat/16 active idle [Link] Filebeat ready.
nrpe-container/15 active idle [Link] icmp,5666/tcp Ready
ceph-osd/2 active idle 6 [Link] Unit is ready (1 OSD)
filebeat/18 active idle [Link] Filebeat ready.
nrpe-container/18 active idle [Link] icmp,5666/tcp Ready
ceph-radosgw/0* active idle 7 [Link] 443/tcp Unit is ready
filebeat/0* active idle [Link] Filebeat ready.
nrpe-container/0* active idle [Link] icmp,5666/tcp Ready

Machine State DNS Inst id Series AZ Message

1 started [Link] juju-4a4b92-1 bionic Running
2 started [Link] juju-4a4b92-2 bionic Running
3 started [Link] juju-4a4b92-3 bionic Running
4 started [Link] juju-4a4b92-4 bionic Running
5 started [Link] juju-4a4b92-5 bionic Running
6 started [Link] juju-4a4b92-6 bionic Running
7 started [Link] juju-4a4b92-7 bionic Running

In this case, you'll notice that each ceph-osd unit provides 1 OSD to the cluster. That means a total of 3 OSDs.

The CRUSH algorithm assigns a weight value per device with the objective of approximating a uniform distribution for I/O requests. Pools
are created, usually, with devices of the same type and size, and the same relative weight. Since this is not always practical, you may
incorporate devices of different size and use a relative weight so that Ceph will distribute more data to larger drives and less data to
smaller drives.

The initial crush weight, crush-initial-weight , for newly added OSDs into crush map . Use this option only if you wish to set the
weight for newly added OSDs in order to gradually increase the weight over time. Be very aware that setting this overrides the default
setting, which can lead to imbalance in the cluster, especially if there are OSDs of different sizes in use. By default, the initial crush weight
for the newly added OSD is set to its volume size in TB. Leave this option unset to use the default provided by Ceph itself. This option only
affects NEW OSDs, not existing ones.

This is a parameter you can set to ceph-osd charm. It provides a way to manage the rebalancing that would take place in case of adding
more OSDs to your cluster. By default, it will let Ceph manage weights of newly added OSDs, but if you don't want rebalancing to take
place, you need to set this to "0". Let's do that now:

juju config ceph-osd crush-initial-weight=0

Once that's finished reconfiguring, we can freely add/remove OSDs without Ceph moving any data to them. First, let's see which devices
are used as OSDs:

77 / 101
10. Ceph

juju config ceph-osd osd-devices

# output
/srv/osd

It's a folder based OSD. Let's add a new folder. But before that, you’ll get the current osd tree :

juju ssh ceph-mon/0 sudo ceph osd tree

# output
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.42519 root default
-7 0.14169 host juju-4a4b92-4
2 hdd 0.14169 osd.2 up 1.00000 1.00000
-3 0.14180 host juju-4a4b92-5
1 hdd 0.14180 osd.1 up 1.00000 1.00000
-5 0.14169 host juju-4a4b92-6
0 hdd 0.14169 osd.0 up 1.00000 1.00000

juju config ceph-osd osd-devices="/srv/osd /srv/osd2"

Check juju status and wait for the ceph charms to finish execution.

After it's finished setting up, you should have two OSDs / ceph-osd unit. That means a total of six OSDs available. Newly added OSDs
should be empty, since it's not doing any rebalancing and the weight for newly added OSDs is 0, that means they're not going to be used
in the cluster. You can check that using:

juju ssh ceph-mon/0 sudo ceph osd tree

# output
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.42519 root default
-7 0.14169 host juju-4a4b92-4
2 hdd 0.14169 osd.2 up 1.00000 1.00000
4 hdd 0 osd.4 up 1.00000 1.00000
-3 0.14180 host juju-4a4b92-5
1 hdd 0.14180 osd.1 up 1.00000 1.00000
5 hdd 0 osd.5 up 1.00000 1.00000
-5 0.14169 host juju-4a4b92-6
0 hdd 0.14169 osd.0 up 1.00000 1.00000
3 hdd 0 osd.3 up 1.00000 1.00000

So, osd.3 , osd.4 and osd.5 are the newly added OSDs, all of them having 0 weight.

Check the Ceph cluster status:

juju ssh ceph-mon/0 sudo ceph -s

# output
cluster:
id: 11226674-4d16-11ec-a8c7-00163e9aaf6f
health: HEALTH_OK

services:
mon: 3 daemons, quorum juju-4a4b92-1,juju-4a4b92-2,juju-4a4b92-3
mgr: juju-4a4b92-3(active), standbys: juju-4a4b92-2, juju-4a4b92-1
osd: 6 osds: 6 up, 6 in
rgw: 1 daemon active

data:
pools: 19 pools, 92 pgs
objects: 1.92 k objects, 11 GiB
usage: 89 GiB used, 604 GiB / 692 GiB avail
pgs: 92 active+clean

io:
client: 3.7 KiB/s rd, 3 op/s rd, 0 op/s wr

78 / 101
10. Ceph

Now that we tested adding new OSDs, let's remove them. First, you need to mark them as out :

juju ssh ceph-mon/0 sudo ceph osd out 3

juju ssh ceph-mon/0 sudo ceph osd out 4
juju ssh ceph-mon/0 sudo ceph osd out 5

Let's use ceph osd tree to see the changes, notice the REWEIGHT values are 0:

juju ssh ceph-mon/0 sudo ceph osd tree

# output
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.42519 root default
-7 0.14169 host juju-4a4b92-4
2 hdd 0.14169 osd.2 up 1.00000 1.00000
4 hdd 0 osd.4 up 0 1.00000
-3 0.14180 host juju-4a4b92-5
1 hdd 0.14180 osd.1 up 1.00000 1.00000
5 hdd 0 osd.5 up 0 1.00000
-5 0.14169 host juju-4a4b92-6
0 hdd 0.14169 osd.0 up 1.00000 1.00000
3 hdd 0 osd.3 up 0 1.00000

Next, let's reconfigure ceph-osd to only use one folder as OSD, not two:

juju config ceph-osd osd-devices="/srv/osd"

Then, let's stop the new OSDs, make sure you stop the correct OSD from the correct node:

juju run-action ceph-osd/0 --wait stop osds=<osd_ID>

juju run-action ceph-osd/1 --wait stop osds=<osd_ID>
juju run-action ceph-osd/2 --wait stop osds=<osd_ID>

In this case:

juju run-action ceph-osd/0 --wait stop osds=4

juju run-action ceph-osd/1 --wait stop osds=5
juju run-action ceph-osd/2 --wait stop osds=3

Let's check ceph osd tree :

juju ssh ceph-mon/0 sudo ceph osd tree

# output
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.42519 root default
-7 0.14169 host juju-4a4b92-4
2 hdd 0.14169 osd.2 up 1.00000 1.00000
4 hdd 0 osd.4 down 0 1.00000
-3 0.14180 host juju-4a4b92-5
1 hdd 0.14180 osd.1 up 1.00000 1.00000
5 hdd 0 osd.5 down 0 1.00000
-5 0.14169 host juju-4a4b92-6
0 hdd 0.14169 osd.0 up 1.00000 1.00000
3 hdd 0 osd.3 down 0 1.00000

Last step, remove the OSDs from the monitors:

juju run-action --wait ceph-mon/leader purge-osd osd=3 i-really-mean-it=yes

juju run-action --wait ceph-mon/leader purge-osd osd=4 i-really-mean-it=yes
juju run-action --wait ceph-mon/leader purge-osd osd=5 i-really-mean-it=yes

Verify Ceph health and the number of OSDs available:

79 / 101
10. Ceph

juju run -u ceph-mon/leader "sudo ceph osd tree"

juju run -u ceph-mon/leader "sudo ceph --status"

# output
cluster:
id: 11226674-4d16-11ec-a8c7-00163e9aaf6f
health: HEALTH_OK

services:
mon: 3 daemons, quorum juju-4a4b92-1,juju-4a4b92-2,juju-4a4b92-3
mgr: juju-4a4b92-3(active), standbys: juju-4a4b92-2, juju-4a4b92-1
osd: 3 osds: 3 up, 3 in
rgw: 1 daemon active

data:
pools: 19 pools, 92 pgs
objects: 1.92 k objects, 11 GiB
usage: 44 GiB used, 302 GiB / 346 GiB avail
pgs: 92 active+clean

10.2 Ceph versions

There is a way of you finding out the version of each Ceph daemon running in your Ceph cluster. This is useful whenever doing package
upgrades. Package upgrades can be either part of some security maintenance window or can be related to upgrading Openstack to a new
version. Either way, you need a way to identify which versions of each daemon is installed in your cluster. To do that, you can run ceph
versions command:

juju ssh ceph-mon/0 "sudo ceph versions"

# output
{
"mon": {
"ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable)": 3
},
"mgr": {
"ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable)": 3
},
"osd": {
"ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable)": 3
},
"mds": {},
"rgw": {
"ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable)": 1
},
"overall": {
"ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable)": 10
}
}

Apparently, everything is on v13.2.9. When doing upgrades, each type of daemon will be upgraded to a newer version. The recommended
order of upgrading is:

ceph-mon
ceph-fs (if exists)
ceph-radosgw

80 / 101
10. Ceph

ceph-osd

10.3 Monitor and CRUSH maps

Ceph monitors maintain a "master copy" of the cluster map. That means a Ceph client can determine the location of all Ceph monitors and
Ceph OSDs just by connecting to one Ceph monitor and retrieving a current cluster map.

Before Ceph clients can read from or write to Ceph OSDs, they must connect to a Ceph monitor first. With a current copy of the cluster
map and the CRUSH algorithm, a Ceph client can compute the location for any object. The ability to compute object locations allows a
Ceph client to talk directly to Ceph OSDs, which is a very important aspect of Ceph high scalability and performance.

The primary role of the Ceph monitor is to maintain a master copy of the cluster map.

Both monitor and CRUSH maps can be fetched from the monitors. To extract the monitor map (monmap), you'll need to stop the monitor
service first ([Link])

juju ssh ceph-mon/0 "sudo systemctl stop [Link]"

juju ssh ceph-mon/0 "hostname"
juju ssh ceph-mon/0 "sudo ceph-mon -i <hostname> -d --extract-monmap /tmp/[Link]"
juju ssh ceph-mon/0 "sudo systemctl start [Link]"

For the CRUSH map, you won't need to stop any services. Again, it can be taken from any of the monitors.

juju ssh ceph-mon/0 "sudo ceph osd getcrushmap -o /tmp/compiled-crushmap"

juju ssh ceph-mon/0 "sudo crushtool -d /tmp/compiled-crushmap -o /tmp/decompiled-crushmap"
juju ssh ceph-mon/0 "cat /tmp/decompiled-crushmap"

To upload a new CRUSH map to the monitors, you'll need to re-compile it and use ceph osd setcrushmap command:

# crushtool -c decompiled-crushmap -o compiled-crushmap

# ceph osd setcrushmap -i compiled-crushmap

This can be done from any of the monitor units.

10.4 Logging

Logs are located on each unit, either monitors, OSDs or radosgw, in /var/log/ceph . Ceph-mon units also contain the logs for the mgr
(manager) service and audit logs. Ceph-osd units contain all the log files for all OSDs running on that unit.

juju ssh ceph-mon/0 sudo ls -al /var/log/ceph/

# output
-rw-r--r-- 1 ceph ceph 342994 Nov 24 14:05 [Link]
-rw-r--r-- 1 ceph ceph 1260521 Nov 24 14:05 [Link]
-rw------- 1 ceph ceph 2112768 Nov 24 14:05 [Link]
-rw------- 1 ceph ceph 1047895 Nov 24 14:05 [Link]

81 / 101
11. Swift/RadosGW

juju ssh ceph-osd/0 sudo ls -al /var/log/ceph/

# output
-rw-r--r-- 1 ceph ceph 1356 Nov 24 13:30 [Link]
-rw-r--r-- 1 ceph ceph 65616 Nov 24 11:19 [Link]
-rw-r--r-- 1 ceph ceph 46064 Nov 24 13:51 [Link]

juju ssh ceph-radosgw/0 sudo ls -al /var/log/ceph/

# output
-rw-r--r-- 1 ceph ceph 3786 Nov 24 11:23 [Link]

11. Swift/RadosGW
Ceph RadosGW provides Amazon S3/Openstack Swift compatible API. It integrates with Keystone to provide authentication and
authorization for users working with containers and objects.

In this chapter, we're going to use all three alternatives in working with RadosGW:

openstack container/object command

swift command
curl using an authentication token generated from Keystone

11.1 openstack, swift and curl commands

First, let's create some test files that we're going to use throughout the lab:

mkdir -p ~/swift_lab
cd ~/swift_lab
for i in $(seq 1 10); do touch file${i}.txt; echo "this is file ${i}" >> file${i}.txt; done

These are ten files, each of them having different content. All actions in this lab, you're going to do them as user1 . First, we're going to
create a container:

source ~/resources/user1_openrc

openstack container create container1

openstack container list

openstack container show container1

Next, get the endpoint for Swift and get an authentication token. Running the script will output two export commands. Copy and paste
them in your terminal:

bash ~/resources/get_token_and_storage_url.sh

Example:

82 / 101
11. Swift/RadosGW

export OS_AUTH_TOKEN=gAAAAABg6sYCyFR712UEOi_mmCWnpUsOwNA4...
export OS_STORAGE_URL=[Link]

Now, let's create a second container, this time using curl :

curl -i -k -X PUT -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/container2

# output
HTTP/1.1 201 Created
Date: Thu, 25 Nov 2021 14:14:14 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Length: 0
X-Trans-Id: tx000000000000000000004-00619f9a36-15b4-default
X-Openstack-Request-Id: tx000000000000000000004-00619f9a36-15b4-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8

Next command will make the container world readable:

curl -i -k -X POST -H "X-Auth-Token: $OS_AUTH_TOKEN" \

-H "X-Container-Read: .r:*" $OS_STORAGE_URL/container2

Then, let's upload 5 files from the ones we create earlier to container1 using the "openstack" command, and the other five to
container2 using "curl":

openstack object create container1 file{1,2,3,4,5}.txt

for i in $(seq 6 10); do

curl -i -k -T file${i}.txt -X PUT \
-H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/container2/file${i}.txt
done

Let's list the objects in container2 , this time, using the "openstack" command:

openstack object list container2

The same command, but using "curl":

83 / 101
11. Swift/RadosGW

curl -k -X GET -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/container2/

# output
[Link]
[Link]
[Link]
[Link]
[Link]

Next, we're going to create a download folder in which you're going to retrieve objects from your containers in different ways, "openstack"
and "curl":

mkdir -p ~/swift_lab/download
cd ~/swift_lab/download

openstack object save container1 [Link]

curl -s -S -k -X GET -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/container1/[Link] -O

Check the contents of the files, just to make sure they're not empty:

ls -l
# output
-rw-rw-r-- 1 ubuntu ubuntu 15 Nov 25 14:16 [Link]
-rw-rw-r-- 1 ubuntu ubuntu 15 Nov 25 14:16 [Link]

cat [Link]
cat [Link]

Deleting objects can be done using both commands:

openstack object delete container2 [Link]

curl -s -S -k -X DELETE -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/container1/[Link]

Let's check the contents of each container:

openstack object list container1

openstack object list container2

84 / 101
12. Heat

Deleting containers can be easily done using the swift command. "openstack" and "curl" will require the container to be empty. At the
moment, both containers have objects. Let's try and delete a container:

swift delete container2 --insecure

Let's try and delete "container1" using "openstack" and "curl":

openstack container delete container1

# output
Conflict (HTTP 409) (Request-ID: tx000000000000000000023-00619f9b1c-15b4-default)

curl -s -S -k -X DELETE -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/container1

# output
There was a conflict when trying to complete your request.

Neither of them worked, since only the "swift" command can delete the container recursively. Let's try "swift":

swift delete container1 --insecure

Last command, let's make sure the cleanup is done properly:

openstack container list

cd ~

11.2 Logging

Ceph RadosGW logs, as stated in the Ceph chapter, can be found on each RadosGW unit, in /var/log/ceph . To list the files and check
their contents:

juju ssh ceph-radosgw/0 "sudo ls -la /var/log/ceph/"

juju ssh ceph-radosgw/0 "sudo tail -20 /var/log/ceph/[Link].<unit_hostname>.log"

And, in case of using multiple units and HACluster charm:

juju ssh hacluster-radosgw/<unit_number> "sudo journalctl -u [Link] -b"

juju ssh hacluster-radosgw/<unit_number> "sudo journalctl -u [Link] -b"
juju ssh hacluster-radosgw/<unit_number> "sudo journalctl -u [Link] -b"

12. Heat
Heat is a service to orchestrate composite cloud applications using a declarative template format through an OpenStack-native REST API.
Heat provides a template based orchestration for describing a cloud application by executing appropriate Openstack API calls to generate
running cloud applications. A Heat template describes the infrastructure for a cloud application in text files which are readable and writable
by humans, and can be managed by version control tools.

Templates specify the relationships between resources (e.g. this volume is connected to this server). This enables Heat to call out to the
OpenStack APIs to create all of your infrastructure in the correct order to completely launch your application.

The software integrates other components of OpenStack. The templates allow creation of most OpenStack resource types (such as
instances, floating ips, volumes, security groups, users, etc), as well as some more advanced functionality such as instance high
availability, instance autoscaling, and nested stacks.

In this chapter, you're going to check the contents of a stack that you're going to deploy. You'll replace different variables in the
environment file and then connect to the instance you're creating through Heat.

85 / 101
12. Heat

12.1 Creating stacks

First, let's inspect the files we're going to deploy:

cat ~/resources/[Link]
cat ~/resources/[Link]

The first one contains all the resources that you're going to deploy in your Openstack cloud:

an instance with security groups

a private network with a subnet
a router
a floating IP associated with the instance
a list of parameters used throughout the template

The second file contains changes to those parameters that you're going to use to deploy the template:

public network name

floating IP ID
image
flavor
SSH key added to the instance

Some of them are preconfigured and some of them, you'll need to configure yourself: floating_ip and ssh_keys. So, let's get the
information required:

source ~/resources/user1_openrc

cat ~/.ssh/user1_id_rsa.pub

openstack floating ip create public_network

Now, let's edit [Link] with the new information provided.

nano ~/resources/[Link]

Whenever finished, verify the contents using the cat command:

cat ~/resources/[Link]
# output
parameters:
public_network: public_network
floating_ip: cb49ca96-9eae-4ead-ac92-9b88cb311e9d
image: focal
flavor: [Link]
ssh_keys: |
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABgQDJo
+nbqOvPeYeNSE8dII9TsLLC9fT8Y5AuVdwOs4cEphwHxnyjhHOmvoMn8LeTbOELUGTro
6g6zfabwVuO0mUBm3tuyuHkJfCc1X4EafCZzPuxCVtrNQNtlMYrf0DpH684Na4wcH6iW
oud+HTaDNARpNI82ztwflRNTkEdYSqPE/Ms4FlyAcjyvNsIKUwbe6U0rK+3/dTsBUpM1
dWEk1z4IFX/...

Now that your environment file has all the necessary information, let's deploy the stack:

86 / 101
12. Heat

openstack stack create -t ~/resources/[Link] -e ~/resources/[Link] test-stack

You can watch the progress of your stack being created using:

watch openstack stack resource list test-stack

Once it's finished, you can double check with:

openstack stack show test-stack | grep stack_status

# output
stack_status | CREATE_COMPLETE
stack_status_reason | Stack CREATE completed successfully

Stack creation has finished. Let's check the resources, like the instance, network, subnet, router, etc.

openstack server list | grep 'test-stack'

openstack network list | grep 'test-stack'
openstack subnet list | grep 'test-stack'
openstack router list | grep 'test-stack'
openstack security group list | grep 'test-stack'

Now that we checked everything is in order, let's try and connect to the instance. If you don't remember the floating IP, issue an
openstack floating ip list command. The floating IP is the one assigned to a fixed IP address from [Link]/24 subnet. Once you
discover the floating IP, run the following command to connect to your newly created instance:

ssh -i ~/.ssh/user1_id_rsa ubuntu@<floating_IP>

Once connected to the instance, install a package called inetutils-ping , so you'll have the "ping" command available, and check
connectivity to [Link]:

sudo apt update

sudo apt install -y inetutils-ping
ping -c 5 [Link]

exit

All resource types available can be found here: [Link]

Details on how to write a HOT file: [Link]

12.2 Logging
There are three files of interest regarding Heat. All these files are present on all Heat units, in/var/log/heat :

[Link] - logs for CloudFormation compatible API

[Link] - logs for Heat API

87 / 101
13. Designate

[Link] - logs for the Heat engine

To inspect these files, use the following commands:

juju ssh heat/0 "tail -10 /var/log/heat/[Link]"

juju ssh heat/0 "tail -10 /var/log/heat/[Link]"
juju ssh heat/0 "tail -10 /var/log/heat/[Link]"

13. Designate
Designate is a multi-tenant DNSaaS service for OpenStack. It provides a REST API with integrated Keystone authentication. It can be
configured to auto-generate records based on Nova and Neutron actions. Designate supports a variety of DNS servers including Bind9
and PowerDNS 4.

In this chapter, you'll work with zones, records, blacklists. You'll also check Designate's backend configuration and verify the changes
you're doing in its backend server (designate-bind). Designate acts as a master for every zone you're creating, while the backend Bind,
installed by the designate-bind charm, will act as a slave.

13.1 Creating zones, records

Let's create a zone called [Link] . You'll also add some entries to that zone and verify everything on its Bind backend.

source ~/resources/user1_openrc

openstack zone create --email user1@[Link] --ttl 3600 [Link].

Even if you don't add any records, there should be already a NS record present. That means, the nameservers for the zone. To verify the
entry in Designate, please run:

openstack recordset list --type NS [Link].

The entry is present in Designate (master), but let's see if it's present in Bind (slave):

88 / 101
13. Designate

BIND_IP=$(juju status designate-bind | grep -A 1 Machine | grep started | awk '{print $3}')

dig @${BIND_IP} [Link] ns +short

# output
[Link].

That means the slave has replicated from the master and now has the proper entry already available. Let's add some more entries to the
master and see if the slave replicates:

openstack recordset create --type A --record [Link] [Link]. www

openstack recordset create --type A --record [Link] [Link]. mail
openstack recordset create --type A --record [Link] [Link]. mail2

You can check the entries in Designate with:

openstack recordset list --type A [Link].

Let's also check the slave:

dig @${BIND_IP} [Link] a +short

dig @${BIND_IP} [Link] a +short
dig @${BIND_IP} [Link] a +short

Ok, so the slave also replicated the new entries you just added. Now, let's see how to add multiple MX (mail exchange) records with
different priorities:

openstack recordset create --type MX --record '10 [Link].' \

--record '20 [Link].' [Link]. [Link].

This command will add [Link] with a priority of 10 and [Link] with a priority of 20. That means, whenever some
other mail server searches for whatever handles email for this domain, [Link] , it will first go to [Link] and then, if
inaccessible, it will go to [Link] .

Let's verify these entries in both master and slave. First, the master (Designate):

openstack recordset list --type MX [Link].

Next, the slave (Bind):

dig @${BIND_IP} [Link] mx +short

# output
20 [Link].
10 [Link].

To list all entries in the [Link] zone, you can use something like:

89 / 101
13. Designate

openstack recordset list [Link]. -f table -c name -c type -c records

Last but not least, let's check the recursiveness of the Bind server:

dig @${BIND_IP} [Link] a +short

# output
[Link].
[Link].
[Link]

That means Bind is configured as recursive. That means it can provide you with name resolution for every domain you're creating in
Designate, but also to everything from outside (e.g. [Link]).

13.2 Blacklisting
You can blacklist domains that you don't want users to have in Designate. Blacklisting requires you to use the administrator account.

Let's blacklist [Link] domain and then switch to user1 and try to create it.

source ~/resources/admin_openrc

Next, we're going to switch to user1 and try to create a zone called [Link] :

source ~/resources/user1_openrc

openstack zone create --email user1@[Link] --ttl 3600 [Link].

# output
Blacklisted zone name

13.3 DNS Backends

DNS backends are configured in a special file in /etc/designate/[Link] . If not using charmed OpenStack, it needs to be
configured manually with all backend DNS servers (Bind, PowerDNS, etc). Once modified, you'll need to run a pool update command.

First, let's check current configuration. It should contain the IP address of the backend Bind server installed by the designate-bind charm:

90 / 101
14. Telemetry - Ceilometer and Gnocchi

juju ssh designate/0 "sudo cat /etc/designate/[Link]"

# output

- id: 794ccc2c-d751-44fe-b57f-8894c9f5c842
name: default
description: Pool genergated by Juju
ns_records:
- hostname: [Link].
priority: 10

nameservers:
- host: [Link]
port: 53

targets:
- type: bind9
masters:
- host: [Link]
port: 5354
options:
host: [Link]
rndc_host: [Link]
rndc_key_file: /etc/designate/[Link]
port: 53
also_notifies: []

[Link] is automatically configured by relating designate to designate-bind. If, for some reason, you need to modify the file manually,
you should run:

# juju ssh designate/0 "sudo designate-manage pool update"

13.4 Logging
All the log files are present on the "designate" units, in /var/log/designate. To check which logs are available in that folder, run:

juju ssh designate/0 "sudo ls -al /var/log/designate/"

To inspect the contents of each file, please run:

juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"

juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"
juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"
juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"
juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"
juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"
juju ssh designate/0 "sudo tail -n 10 /var/log/designate/[Link]"

14. Telemetry - Ceilometer and Gnocchi

The Ceilometer project is a data collection service that provides the ability to normalize and transform data across all current OpenStack
core components with work underway to support future OpenStack components.

Ceilometer is a component of the Telemetry project. Its data can be used to provide customer billing, resource tracking, and alarming
capabilities across all OpenStack core components.

The main object types in Telemetry are:

archive policies: a measure storage policy attached to a metric. It determines how long measures will be kept in a metric and how
they will be aggregated

metrics: an entity storing measures identified by an UUID. It can be attached to a resource using a name. How a metric stores its

91 / 101
14. Telemetry - Ceilometer and Gnocchi

measure is defined by the archive policy it is associated to

measures: a datapoint composed of a timestamp and a value

resource types: a schema that describes a resource: attributes, types, constraints

resources: an entity that can represent anything, will have metrics associated with them. E.g. instances, images, volumes

14.1 Resource types and archive policies

First, install the metric client for OpenStack:

sudo apt install python3-pip -y

pip install gnocchiclient

List the resource types available in Ceilometer:

92 / 101
14. Telemetry - Ceilometer and Gnocchi

source ~/resources/admin_openrc

openstack metric resource-type list

# output
+----------------------------+---------------------------------------------------------------------------------------+
| name | attributes
+----------------------------+---------------------------------------------------------------------------------------+
| ceph_account |
| generic |
| host | - host_name: max_length: 255 , required: True , type: string , min_length: 0
| host_disk | - host_name: max_length: 255 , required: True , type: string , min_length: 0
| | - device_name: max_length: 255 , required: False , type: string , min_length: 0
| host_network_interface | - host_name: max_length: 255 , required: True , type: string , min_length: 0
| | - device_name: max_length: 255 , required: False , type: string , min_length: 0
| identity |
| image | - container_format: max_length: 255 , required: True , type: string , min_length: 0
| | - disk_format: max_length: 255 , required: True , type: string , min_length: 0
| | - name: max_length: 255 , required: True , type: string , min_length: 0
| instance | - display_name: max_length: 255 , required: True , type: string , min_length: 0
| | - availability_zone: max_length: 255 , required: False , type: string , min_length: 0
| | - created_at: required: False , type: datetime
| | - host: max_length: 255 , required: True , type: string , min_length: 0
| | - image_ref: max_length: 255 , required: False , type: string , min_length: 0
| | - flavor_name: max_length: 255 , required: True , type: string , min_length: 0
| | - server_group: max_length: 255 , required: False , type: string , min_length: 0
| | - deleted_at: required: False , type: datetime
| | - launched_at: required: False , type: datetime
| | - flavor_id: max_length: 255 , required: True , type: string , min_length: 0
| instance_disk | - instance_id: required: True , type: uuid
| | - name: max_length: 255 , required: True , type: string , min_length: 0
| instance_network_interface | - instance_id: required: True , type: uuid
| | - name: max_length: 255 , required: True , type: string , min_length: 0
| ipmi |
| ipmi_sensor | - node: max_length: 255 , required: True , type: string , min_length: 0
| loadbalancer |
| manila_share | - status: max_length: 255 , required: True , type: string , min_length: 0
| | - host: max_length: 255 , required: True , type: string , min_length: 0
| | - protocol: max_length: 255 , required: False , type: string , min_length: 0
| | - name: max_length: 255 , required: False , type: string , min_length: 0
| | - availability_zone: max_length: 255 , required: False , type: string , min_length: 0
| network |
| nova_compute | - host_name: max_length: 255 , required: True , type: string , min_length: 0
| port | - controller: max_length: 255 , required: True , type: string , min_length: 0
| stack |
| swift_account |
| switch | - controller: max_length: 255 , required: True , type: string , min_length: 0
| switch_port | - neutron_port_id: max_length: 255 , required: False , type: string , min_length: 0
| | - port_number_on_switch: max: 4294967295 , required: False , type: number , min: 0
| | - switch: max_length: 64 , required: True , type: string , min_length: 0
| | - controller: max_length: 255 , required: True , type: string , min_length: 0
| switch_table | - switch: max_length: 64 , required: True , type: string , min_length: 0
| | - controller: max_length: 255 , required: True , type: string , min_length: 0
| volume | - instance_id: required: False , type: uuid
| | - image_id: required: False , type: uuid
| | - display_name: max_length: 255 , required: False , type: string , min_length: 0
| | - volume_type: max_length: 255 , required: False , type: string , min_length: 0
| volume_provider |
| volume_provider_pool | - provider: max_length: 255 , required: True , type: string , min_length: 0
+----------------------------+---------------------------------------------------------------------------------------+

During the lab, we will check values for different resource types, like images and instances. Now, let's see which metrics are available for
some of those resource types.

The archive policy can be inspected.

openstack metric archive-policy list

93 / 101
14. Telemetry - Ceilometer and Gnocchi

Let's inspects ceilometer-low :

openstack metric archive-policy show ceilometer-low

The archive policies define how the metrics are aggregated and how long they are stored. Each archive policy definition is expressed as
the number of points over a timespan.

This means that Gnocchi will keep the data of 8640 points, each point representing an aggregation of 5 minutes or 300 seconds, this for 30
days. The aggregation method used is mean or average.

(8640 point x 5 minutes) / 1440 minutes in a day = 30 days

14.2 Metrics and measures

Let's check the metrics for a specific resource, like a Glance Image, focal for example:

openstack image list

NOTE: 95601e2b-2c83-4dbc-b166-df9d9de186e7 is the ID of the image.

openstack metric resource show 95601e2b-2c83-4dbc-b166-df9d9de186e7

Specific resource metrics can also be seen by listing all metrics and grep for the resource ID:

94 / 101
14. Telemetry - Ceilometer and Gnocchi

openstack metric list | grep 95601e2b-2c83-4dbc-b166-df9d9de186e7

NOTE: 95601e2b-2c83-4dbc-b166-df9d9de186e7 is the ID of the image.

So the available metrics are [Link] [Link] [Link] . Those metrics will have measures associated with them.

openstack metric measures show [Link] --resource-id 95601e2b-2c83-4dbc-b166-df9d9de186e7

# output
+---------------------------+-------------+--------------+
| timestamp | granularity | value |
+---------------------------+-------------+--------------+
| 2021-11-24T11:55:00+00:00 | 300.0 | 2361393152.0 |
| 2021-11-24T12:00:00+00:00 | 300.0 | 2361393152.0 |
+---------------------------+-------------+--------------+

Let's check the metrics for an instance. Let's do this for the test-stack-server VM. Get the VM ID:

source ~/resources/user1_openrc

openstack server list | grep test-stack-server

# output
| df2e162f-e15b-4778-bfef-e53fd1c72765 | test-stack-server | ACTIVE | ...

NOTE: df2e162f-e15b-4778-bfef-e53fd1c72765 is the ID of the image.

Let's check the metrics for a specific resource:

95 / 101
14. Telemetry - Ceilometer and Gnocchi

openstack metric resource show df2e162f-e15b-4778-bfef-e53fd1c72765

openstack metric list | grep df2e162f-e15b-4778-bfef-e53fd1c72765

As we can see, an instance has more metrics compared to an image: cpu, memory, disk utilization.

Get the measures for the vcpus metric:

openstack metric measures show vcpus --resource-id df2e162f-e15b-4778-bfef-e53fd1c72765

# output
+---------------------------+-------------+-------+
| timestamp | granularity | value |
+---------------------------+-------------+-------+
| 2021-11-25T14:30:00+00:00 | 300.0 | 2.0 |
| 2021-11-25T15:00:00+00:00 | 300.0 | 2.0 |
| 2021-11-25T16:00:00+00:00 | 300.0 | 2.0 |
+---------------------------+-------------+-------+

Get the measures for the [Link] metric:

96 / 101
15. Backup

openstack metric measures show [Link] --resource-id df2e162f-e15b-4778-bfef-e53fd1c72765

# output
+---------------------------+-------------+-------------+
| timestamp | granularity | value |
+---------------------------+-------------+-------------+
| 2021-11-25T14:30:00+00:00 | 300.0 | 80.3984375 |
| 2021-11-25T14:35:00+00:00 | 300.0 | 81.51171875 |
| 2021-11-25T14:40:00+00:00 | 300.0 | 81.71875 |
| 2021-11-25T14:45:00+00:00 | 300.0 | 81.46484375 |
| 2021-11-25T14:50:00+00:00 | 300.0 | 81.890625 |
| 2021-11-25T14:55:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:00:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:05:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:10:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:15:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:20:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:25:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:30:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:35:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:40:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:45:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:50:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T15:55:00+00:00 | 300.0 | 81.91015625 |
| 2021-11-25T16:00:00+00:00 | 300.0 | 81.91015625 |
+---------------------------+-------------+-------------+

What you can do next, is sending that data to 3rd party monitoring tools, like Grafana and Prometheus.

15. Backup
In this chapter, you're going to go through MySQL and Juju controller backups again. Also, you're going to back up the bundle deployed in
your lab environment and inspect that using your favorite text editor.

15.1 Database backup

First, let's take a backup of the MySQL InnoDB cluster. To do that, simply run:

juju run-action mysql-innodb-cluster/0 --wait mysqldump

# output
unit-mysql-innodb-cluster-0:
UnitId: mysql-innodb-cluster/0
id: "104"
results:
mysqldump-file: /var/backups/mysql/[Link]
outcome: Success
status: completed
timing:
completed: 2021-11-25 15:12:13 +0000 UTC
enqueued: 2021-11-25 15:12:07 +0000 UTC
started: 2021-11-25 15:12:07 +0000 UTC

The MySQL dump can be fetched from the unit and copied elsewhere, like a backup server.

15.2 Juju bundle backup

You can also take a backup of a bundle. In this case, you're going to take a backup of the Openstack bundle deployed in your lab
environment. Also, use your favorite editor to inspect its contents:

juju export-bundle --filename openstack_bundle.yaml -m default

There is an option to export the bundle, but with default values being set (available in v2.9):

97 / 101
16. Monitoring and centralized logging

juju export-bundle --filename openstack_bundle_defaults.yaml -m default --include-charm-defaults

16. Monitoring and centralized logging

For monitoring, there's a combination of multiple solutions, all working together. Everything infrastructure related, can be monitored in
Nagios. This includes containers and physical hosts. Then, there's Prometheus. Prometheus gathers information from multiple agents,
including Ceph and OpenStack. So, some metrics are only available in there.

Nagios has its own graphs based on the metrics it gathers from NRPE agents, while Prometheus sends all of its data to Grafana. Let's see
how to access all web interfaces for all the services.

Centralized logging is done using Filebeat, Elasticsearch and Graylog. Filebeat is a daemon running on each host/container and gathers
all the logs from /var/log/ . It can also be configured to search for logs in custom locations. Those logs are sent to an elasticsearch
server (cluster). And Graylog reads the data from Elasticsearch and can do filters and graphs based on custom searches.

16.1 Nagios

To access Nagios web interface, you can either use the DNS name, [Link] , or get the IP address from its unit. The IP
address can be found using:

juju status nagios

Nagios runs on port tcp/80, so there's no special thing to do, just enter the URL in your tunneled browser:

[Link]

Username is nagiosadmin. Password is randomly generated and can be found with the following command:

juju ssh nagios/0 sudo cat /var/lib/juju/[Link]

16.2 Prometheus

Prometheus gathers information from different targets . Those are agents running either in their own containers, or they are subordinate
charms for other main charms (e.g. nova-compute). To access its web interface, you can either use the DNS name,
[Link] , or use the IP address of the unit. To get the IP address, just run:

juju status prometheus

Prometheus web interface runs on port tcp/9090, so the URL should be:

98 / 101
16. Monitoring and centralized logging

[Link]

The web interface does not require authentication.

You can execute queries and see what data gets stored in Prometheus, like:

libvirt_domain_info_virtual_cpus
libvirt_domain_interface_stats_transmit_bytes_total
openstack_allocation_ratio

Or you can check the agents sending data back to Prometheus in Status > Targets .

16.3 Grafana
Grafana uses Prometheus as a source of data. Once related, Grafana and Prometheus, Prometheus is automatically set up as a source of
data for all the dashboards preconfigured in Grafana. Those dashboards also come from Grafana being related to different Prometheus
agents.

To access its web interface, you can either use [Link] or use the IP address of the unit. To get the IP address, just
run:

juju status grafana

Grafana web interface runs on port tcp/3000, so you'll need to enter the following URL in your tunneled browser:

[Link]

Username is admin and the password is randomly generated. The password can be fetched using:

juju run-action --wait grafana/0 get-admin-password

Once you login to the web interface, go to Dashboards > Manage . You should see a list of dashboards automatically added to your
Grafana just by relating Grafana to other charms.

99 / 101
16. Monitoring and centralized logging

16.4 Graylog

Graylog can read the logs from an Elasticsearch cluster. The unit's IP address is already added to DNS as [Link] . You
can also get the IP address by using:

juju status graylog

Graylog's web interface listens on port tcp/9000, so the URL is:

[Link]

Username is admin and the password is dynamically generated. It can be found using:

juju run-action --wait graylog/0 show-admin-password

Once logged in, you can go to the Search menu and you can search for an application name, a unit name, etc.

100 / 101
16. Monitoring and centralized logging

101 / 101

Openstack Ops Lab v2.2
No ratings yet
Openstack Ops Lab v2.2
101 pages
Openstack Foundation Lab 2.1.9
No ratings yet
Openstack Foundation Lab 2.1.9
65 pages
OpenStack Setup Guide on Coursera
No ratings yet
OpenStack Setup Guide on Coursera
10 pages
Os Slides v2.1.7 PP
No ratings yet
Os Slides v2.1.7 PP
315 pages
OpenStack Cluster Setup on Google Cloud
No ratings yet
OpenStack Cluster Setup on Google Cloud
9 pages
Openstack Multinode Guide
No ratings yet
Openstack Multinode Guide
8 pages
OpenStack Setup on VirtualBox Guide
No ratings yet
OpenStack Setup on VirtualBox Guide
4 pages
OpenStack Operations and VM Setup Guide
No ratings yet
OpenStack Operations and VM Setup Guide
15 pages
Openstack Ops Slides 1.3 PP
No ratings yet
Openstack Ops Slides 1.3 PP
394 pages
MAAS CLI Login and Management Guide
No ratings yet
MAAS CLI Login and Management Guide
8 pages
Trystack
No ratings yet
Trystack
16 pages
OpenStack DevStack Setup Guide
No ratings yet
OpenStack DevStack Setup Guide
24 pages
OpenStack Lab Setup on Google Cloud
No ratings yet
OpenStack Lab Setup on Google Cloud
10 pages
Multi-Node OpenStack Deployment Guide
100% (1)
Multi-Node OpenStack Deployment Guide
12 pages
Cloudcomputinglabmanual Cse Lab Manual-39-69
No ratings yet
Cloudcomputinglabmanual Cse Lab Manual-39-69
31 pages
OpenShift User-Provisioned Setup Guide
No ratings yet
OpenShift User-Provisioned Setup Guide
18 pages
OpenStack Lab Setup Guide
No ratings yet
OpenStack Lab Setup Guide
3 pages
Install OpenShift 4.18 on Azure
No ratings yet
Install OpenShift 4.18 on Azure
414 pages
OpenStack Kolla-Ansible Setup Guide
No ratings yet
OpenStack Kolla-Ansible Setup Guide
2 pages
OpenStack Installation Guide for T.E.-A
No ratings yet
OpenStack Installation Guide for T.E.-A
11 pages
Setting Up Meghdoot OpenStack Cloud
No ratings yet
Setting Up Meghdoot OpenStack Cloud
5 pages
OpenShift Container Platform-4.6-Installing On Azure-en-US
No ratings yet
OpenShift Container Platform-4.6-Installing On Azure-en-US
199 pages
Procedure To Launch Virtual Machine Using Trystack
100% (2)
Procedure To Launch Virtual Machine Using Trystack
5 pages
Lab Report Cloud Computing Practical Assignment
No ratings yet
Lab Report Cloud Computing Practical Assignment
26 pages
OpenShift Container Platform-4.4-Installing On AWS-en-US
No ratings yet
OpenShift Container Platform-4.4-Installing On AWS-en-US
260 pages
Install OpenStack on CentOS 8 Guide
No ratings yet
Install OpenStack on CentOS 8 Guide
14 pages
OpenStack Installation on Ubuntu Guide
No ratings yet
OpenStack Installation on Ubuntu Guide
153 pages
Juju Openstack Project
No ratings yet
Juju Openstack Project
41 pages
OpenStack Nested Virtualization Setup Guide
No ratings yet
OpenStack Nested Virtualization Setup Guide
4 pages
Install
No ratings yet
Install
575 pages
OpenStack Kilo Installation Guide
No ratings yet
OpenStack Kilo Installation Guide
177 pages
OpenStack 15.0.0 Install Guide
No ratings yet
OpenStack 15.0.0 Install Guide
155 pages
Launch VM with TryStack OpenStack Guide
No ratings yet
Launch VM with TryStack OpenStack Guide
62 pages
CloudStack Zone and Pod Setup Guide
No ratings yet
CloudStack Zone and Pod Setup Guide
20 pages
OpenShift Container Platform 4.5 Installing en US
No ratings yet
OpenShift Container Platform 4.5 Installing en US
1,553 pages
OpenStack 3-Node High Availability Cluster Install
No ratings yet
OpenStack 3-Node High Availability Cluster Install
11 pages
OpenStack Cloud Computing Overview
No ratings yet
OpenStack Cloud Computing Overview
18 pages
Set Up Zone
No ratings yet
Set Up Zone
20 pages
OpenStack Installation and Configuration Guide
No ratings yet
OpenStack Installation and Configuration Guide
44 pages
Multi-Node OpenStack Installation Guide
No ratings yet
Multi-Node OpenStack Installation Guide
9 pages
Launch VM with TryStack Guide
No ratings yet
Launch VM with TryStack Guide
8 pages
Launch VM on TryStack OpenStack Demo
No ratings yet
Launch VM on TryStack OpenStack Demo
42 pages
Managing OpenShift with Ansible Automation
No ratings yet
Managing OpenShift with Ansible Automation
47 pages
Install OpenStack on Ubuntu 18.04
No ratings yet
Install OpenStack on Ubuntu 18.04
5 pages
DevStack OpenStack Setup Guide
No ratings yet
DevStack OpenStack Setup Guide
9 pages
OpenStack Dashboard Setup Guide
No ratings yet
OpenStack Dashboard Setup Guide
13 pages
Install Your Own OpenStack Cloud Essex Edition
No ratings yet
Install Your Own OpenStack Cloud Essex Edition
49 pages
Openshift_UPI_Installation practice
No ratings yet
Openshift_UPI_Installation practice
18 pages
OpenStack-Ansible Deployment Guide
No ratings yet
OpenStack-Ansible Deployment Guide
28 pages
OpenShift Container Platform-4.6-Installing On GCP-en-US
No ratings yet
OpenShift Container Platform-4.6-Installing On GCP-en-US
248 pages
Openshift Container Platform 4.3: Installing On Openstack
No ratings yet
Openshift Container Platform 4.3: Installing On Openstack
47 pages
Launching Virtual Machines in OpenStack
No ratings yet
Launching Virtual Machines in OpenStack
14 pages
Private Cloud Network Project Guide
No ratings yet
Private Cloud Network Project Guide
26 pages
OpenStack Installation Guide For (RHEL, CentOS, Fedora)
100% (1)
OpenStack Installation Guide For (RHEL, CentOS, Fedora)
140 pages
Openshift Virtualization Example Architecture - 1000 Virtual Machines
No ratings yet
Openshift Virtualization Example Architecture - 1000 Virtual Machines
57 pages
Detailed Command Walkthrough For A Simplified Multinode OpenStack Setup (Using DevStack)
No ratings yet
Detailed Command Walkthrough For A Simplified Multinode OpenStack Setup (Using DevStack)
12 pages
OpenShift Container Platform 3.7 Installation and Configuration en US
0% (1)
OpenShift Container Platform 3.7 Installation and Configuration en US
651 pages
Install OpenStack on Ubuntu 18.04
No ratings yet
Install OpenStack on Ubuntu 18.04
11 pages
OpenShift - Container - Platform 3.11 Installing - Clusters en US PDF
No ratings yet
OpenShift - Container - Platform 3.11 Installing - Clusters en US PDF
124 pages
Configuring Cisco Proxy Server
No ratings yet
Configuring Cisco Proxy Server
6 pages
Secure SSH Connection Guide
No ratings yet
Secure SSH Connection Guide
4 pages
Class 10 Computer Marking Scheme 2024-25
No ratings yet
Class 10 Computer Marking Scheme 2024-25
5 pages
Comptia Linux Xk0 005 Exam Objectives (2 0)
No ratings yet
Comptia Linux Xk0 005 Exam Objectives (2 0)
16 pages
CCNA Foundations with Kevin Wallace
0% (1)
CCNA Foundations with Kevin Wallace
126 pages
FortiOS 6.4.0 Best Practices Guide
No ratings yet
FortiOS 6.4.0 Best Practices Guide
47 pages
Internet Basics for Grade 10 Students
No ratings yet
Internet Basics for Grade 10 Students
3 pages
Comprehensive Cybersecurity Commands Guide
No ratings yet
Comprehensive Cybersecurity Commands Guide
3 pages
Update ONIE Firmware for Dell OS10
No ratings yet
Update ONIE Firmware for Dell OS10
3 pages
Palo Alto Networks Firewall Setup Guide
No ratings yet
Palo Alto Networks Firewall Setup Guide
30 pages
Flashing TM-AC1900 to RT-AC68U Guide
No ratings yet
Flashing TM-AC1900 to RT-AC68U Guide
3 pages
LEP Data Encryption Guidelines
No ratings yet
LEP Data Encryption Guidelines
6 pages
Network Management Essentials Guide
No ratings yet
Network Management Essentials Guide
37 pages
Github
No ratings yet
Github
93 pages
Data Encryption Policy Overview
No ratings yet
Data Encryption Policy Overview
6 pages
iPhone Activation Bypass Script
83% (6)
iPhone Activation Bypass Script
6 pages
GSEP Installation Guide Overview
No ratings yet
GSEP Installation Guide Overview
259 pages
Basic Linux Commands Overview
No ratings yet
Basic Linux Commands Overview
53 pages
PYNQ-Z2 FPGA Project Deployment Guide
No ratings yet
PYNQ-Z2 FPGA Project Deployment Guide
38 pages
Linux Server Configuration Guide
No ratings yet
Linux Server Configuration Guide
11 pages
CACU NGTJ Backup and Installation Guide
No ratings yet
CACU NGTJ Backup and Installation Guide
11 pages
Organizations Userguide
No ratings yet
Organizations Userguide
329 pages
SSH2 Usersguide
No ratings yet
SSH2 Usersguide
117 pages
FW8015 19.0v1 Managing Logs and Notifications On Sophos Firewall
No ratings yet
FW8015 19.0v1 Managing Logs and Notifications On Sophos Firewall
18 pages
Abinitio Architecture and Features Overview
No ratings yet
Abinitio Architecture and Features Overview
2 pages
Navigating Cisco EXEC Modes
No ratings yet
Navigating Cisco EXEC Modes
8 pages
Cloud Computing Virtualization and GAE Guide
No ratings yet
Cloud Computing Virtualization and GAE Guide
6 pages
Cisco ASA 9.19 New Features Overview
No ratings yet
Cisco ASA 9.19 New Features Overview
321 pages
Adding a Node in Oracle RAC 19c
No ratings yet
Adding a Node in Oracle RAC 19c
35 pages