NetDevOps – Reducing the

Attack Surface of IOS XE

with Ansible

Tim Glen, Systems Architect

DEVNET-2111
 Merci de Hvala vam što
Gracias por
Spanish Tak fordi du
French
m'accueillir. steCroatian
me primili.
recibirme.
Danish
byder mig Go raibh maith
velkommen. Kiitos, että toivotit agat asIrish
fáilte a
Finnish chur romham.
minut tervetulleeksi.
Дякую, що
Ukrainian
прийняли мене. Vă mulțumesc
Romanian
Bedankt voor de că m-ați primit.
Dutch
Takk for at du uitnodiging.
Norwegian
ønsker meg
velkommen. Obrigado por me Köszönöm, hogy GrazieItalian
per avermi
Portuguese Hungarian accolto.
receber. befogadtál.
Vielen Dank, dass Sie
michGerman
willkommen ‫דאנק איר ֿפאַ ר‬
Yiddish
Tapadh leibh airson .‫באַ גריסן מיר‬
geheißen haben. ‫شكرا لك لحسن‬
Arabic fàilte a chuir orm.
.‫استضافتي‬
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Tim Glen (Personal)
• Husband to Hillary
• Father to Jenna
• 2 Dogs (Snickerdoodle & Autumn)
• Love the Outdoors
• Fitness, Running, Hiking
• Travel
• Driving Fast Cars,
• esp Porsche!
Attempt to add
pic of me on bike
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Tim Glen (Professional)
• Started in IT in 1995 Telephone Tech Support
• Worked 23 years at Advertising Specialty
• Web hosting provider, 500 employees, 30,000 hosted site
• Managed all routers, switches, firewalls, wireless, security

• Employed at Cisco 4+ years

• Started September 2019
• Systems Architect in Philadelphia, Pennsylvania

[Link]/timmayg

[Link]/in/timglen

[Link]/TimGlen
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 • Secure Ansible Usage
• IOS XE Attack Surface
• Local User Hardening
• Protocol Hardening

Agenda • MACSec Over the Wire

Encryption
• Conclusion

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
IOS XE Day Zero
Config
Day 0 IOS XE Config (page 1)
• IP Address !
interface GigabitEthernet1/0/24
no switchport
• Default Gateway ip address [Link] [Link]
!
• DNS ip route [Link] [Link] [Link]
!
ip name-server [Link] [Link]
!
clock timezone EST -5 0
• Time Zone clock summer-time EDT recurring
!
ntp server [Link]
ntp server [Link]
• NTP ntp server [Link]
ntp server [Link]
ntp server [Link]
!

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Day 0 IOS XE Config (page 2)
!
username timmyg privilege 15 secret 8 $8$CvofI3VTja.....
!
aaa new-model
• AAA, Local User !
aaa authentication login CON-LOCAL local
aaa authorization exec CON-LOCAL local
aaa authorization console
!
line vty 0 15
login authentication CON-LOCAL
authorization exec CON-LOCAL
!
• NetConf-YANG netconf
netconf-yang
!
yang-interfaces aaa authentication method-list CON-LOCAL
yang-interfaces aaa authorization method-list CON-LOCAL
• YANG AAA !

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
POSSIBLY DELETE

What is the IOS

XE Attack
Surface
POSSIBLY DELETE
What is the IOS XE Attack Surface
• Insecure Local DB Username and Password
• Open UDP \ TCP Ports
• Network facing services not optimized
• Con, Aux, VTY lines
• Clear text data in Transit

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Secure Ansible
Usage
 SSH Key Checking
• Have you all seen this error ?
• First time Ansible connects to IOS XE, no host key

The authenticity of host IP_ADDRESS can’t be

established due to Host is Unknown
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
[Link]

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
!!! UPDATE WITH MORE INFO !!!
Managing your ~/.ssh/known_hosts

• ssh-keyscan
• Allows you to fetch the SSH Key of Hosts.
• Can append these SSH Keys

• ssh-keygen
• Allows for checking and removal of keys from ~/.ssh/known_hosts

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Ansible Inventory \ Playbook Files

• Use Ansible Vault Instead

• Ansible Vault encrypts variables that you

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
SSH Key Checking
• Host key checking enabled by default

• Do NOT change this to false!

• Host keys stored in ~/.ssh/known_hosts
• Use ssh-keyscan to ‘import’ the SSH host key!

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 SSH Known Hosts
• Host keys stored in ~/.ssh/known_hosts

1st host key Hashed IP Addr \ Hostname

2nd host key Algorithm

tcp/830 Base64 Pub Key

5th host key, manually revoked

You can also remove known host keys

ssh-keygen -f "/home/ciscolive/.ssh/known_hosts" -R "[Link]"
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 ssh-keyscan options
• Let’s add our ‘unknown’ device to ~/.ssh/known_hosts
ssh-keyscan -H {{FQDN or IP_ADDRESS}} >> ~/.ssh/known_hosts

-H parameter will add Hashed IP Addr

ssh-keyscan {{FQDN or IP_ADDRESS}} >> ~/.ssh/known_hosts

ssh-keyscan –p 830 {{FQDN or IP_ADDRESS}} >> ~/.ssh/known_hosts

NETCONF Port tcp/830

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
SSH Key
Demo
ssh-keyscan Demo Steps
• Run simple playbook, watch fail
ansible-playbook -i inventories/[Link] playbooks/[Link]

• cat ~/.ssh/known_hosts
• ssh-keyscan [Link]
• ssh-keyscan [Link] >> ~/.ssh/known_hosts
• ssh-keyscan –p 830 [Link] >> ~/.ssh/known_hosts
• nano ~/.ssh/known_hosts
• This only has to be done once!
• Run simple playbook, watch succeed
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
ssh-keyscan Demo Results

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 Ansible Vault Demo Steps
• Show Inventory File, inventories/[Link]
• Discuss Username & Pass in the clear
• Remove U&P from Inventory
• Copy \ Paste into vaults/[Link]
ansible-vault encrypt vaults/[Link]

• Update Playbook
vars_files:
- ~/clus2023-devnet-2111/vaults/[Link]

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Common Criteria
&
Local Users Secrets
&
Login Block
 scan here for
IOS XE Local Passwords \ Secrets config guide!!!

Weak password hashing?

Weak methods for securing \ storing them?
Do these comply with business rules \ Password Compliance?

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 scan here for
Cisco Password Types config guide!!!

• 0,5,6,7,8,9 ??? What do we do ?

• Type 5 – MD5 hash, crypto broken, replace where possible

• Type 6 - reversable, Bulk Data Encryption, RADIUS, TACACS keys
• Type 7 - obfuscation, outdated stop using immediately
• Type 8 - SHA256 Hash, NIST Approved!
• Type 9 - SCRYPT Hash, not NIST approved, IOS XE default

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Common Criteria Policy
• Allows NetEng to enforce strong password policies on local users
• Enables InfoSec & Auditors to see validate password policy

Password must contain at least 12

characters and no more than 127
PW change req’s > 5 chars changed
Same character cannot be used 3x
qwer or asdf not permitted

These are just a few strong

password options.

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 scan here for
Local Users & Masked Secret config guide!!!
• Must use strong password, Common Criteria !!!
• Must use strong hashing algorithms, Type 8 NIST approved!
• Use masked-secret so ‘secret’ is NEVER displayed in the clear

Type 8

Password should never be

displayed or logged

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Login Block
• Slows down attackers attempting dictionary attacks
• Quiet period after multiple incorrect password
• Quiet period bypass for known admin IP addr with ACL

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Common Criteria
&
Local Users Secrets
&
Login Block
Demo
 PW Type & CC Policy User Demo Steps
• Show the PW Type & Common Criteria Playbook in VS Code
• Explain Connection Type = NETCONF
• Explain how the RPC was built in YANG Suite
• Run the PW Type & Common Criteria Playbook.
ansible-playbook -i inventories/[Link] playbooks/[Link]

• Note how we forgot to add –-ask-vault-pass

• Run Playbook again with ask-vault this time ☺
• On Cat9K Show Common Criteria Policy
show runn | sec aaa common-criteria
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 Local User Demo Steps
• Creating this local user could occur on hundreds or thousands
boxes
• We shouldn’t ever have the password displayed in the clear
• Show Playbook and Copy \ Paste Type 8
• Run Playbook
ansible-playbook -i inventories/[Link] playbooks/[Link] --
ask-vault-pass
• On Cat9K Show Newly Created Local User
show runn | inc username femke

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Enable Type 6
Encryption
 scan here for
Type 6 Encryption config guide!!!

• Supported since 2006 \ IOS 12.3 and possibly earlier

• Strong AES 128-bit encryption
• Encrypts RADIUS & TACACS, MACsec & IPsec PSK
• Type 6 passwords need to be reversed by IOS XE

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 THIS MAY NOT BE SUPPORTED AND
IOS XE Secure Storage THIS SLIDE MAY NEED TO BE
REMOVED

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Enable Type 6
Encryption
 Type 6 Demo Steps
• Add RADIUS servers to IOS XE
• Show Type 0 or Type 7
• Show Type 6 Playbook in VS Code
• Run Type 6 Playbook
ansible-playbook -i inventories/[Link] playbooks/[Link] --ask-vault-
pass

• No Type 6 Passwords exist now

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Harden
SSH & HTTPS
 What is the TCP Attack Surface of IOS XE
What TCP Ports are listening ?

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
HTTPS Cipher Suites in IOS XE 17.13(1)

https //[Link]

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
 What is the UDP Attack Surface of IOS XE
What UDP Ports are listening ?

udp/123 - NTP
udp/2228 - L2 traceroute
udp/161, 162 - SNMP

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
SSH Service Listener

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
HTTPS Service Listener

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Harden
SSH & HTTPS
Demo
 Harden SSH & HTTP Demo Steps
• Show the Hardening Playbook in VS Code
• On Cat9K Show SSH & HTTP Server Status
show ip ssh

show ip http server status | sec ciphersuite

• Run Playbooks
ansible-playbook -i inventories/[Link] playbooks/03-config-hard-
[Link] --ask-vault-pass
ansible-playbook -i inventories/[Link] playbooks/04-config-hard-
[Link] --ask-vault-pass

• On Cat9K Show SSH & HTTP Server Status

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
SNMPv2
vs
SNMPv3
SNMPv2

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
SNMPv2 Packet Capture

SNMP Response

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
SNMPv3 Packet Capture

SNMP Response

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 scan here for
SNMPv3 Basics config guide!!!

• View – What OIDs can be queried

• Group – Logical grouping of users, RW or RO
• User – Assign a user for each SNMP Monitoring Workstation
• Auth Protocol - MD5 \ SHA \ SHA256 \ SHA384 \ SHA512
• Auth Key – Shared Secret, verifies the integrity of the messages
• Privacy Protocol – DES \ AES-128 \ AES-192 \ AES-256
• Privacy Key – Shared Secret, used for bulk data encryption

See the
hidden
slides for
DEVNET-2111
more info!
© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
SNMP v3 SHA-2 Info (pg 1)
[Link]
17-x/snmp-xe-17-book/[Link]

© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
SNMP v3 SHA-2 Info (pg 2)

© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
SNMP v3 SHA-2 Info (pg 3)

© 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
SNMPv3 Template
Demo
RECORD THIS DEMO NEXT

SNMP Demo Steps

• Show SNMPv2 Config
• Run Simple SNMP Get
snmpget -v2c -c cisco123 [Link] SNMPv2-MIB::sysName.0

• Show SNMPv3 Playbook in VS Code

• Run Playbook
ansible-playbook -i inventories/[Link] playbooks/05-add-snmp-v3-
[Link] --ask-vault-pass
• Show SNMPv3 Config
• Run same Simple SNMP Get using SNMPv3
snmpget -v3 -l authPriv -a SHA -u willem-ReadOnly -A ciscolive123 -
x AES -X ciscolive123 [Link] SNMPv2-MIB::sysName.0
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
MACsec PSK
 scan here for
MACsec Switch to Switch with PSK config guide!!!
• Dance like no one is watching
• Encrypt like everyone is watching
• Lightweight L2 encryption
• Cat9K Line Rate 1 -400Gb encryption AES-128 AES-256
• <1% CPU impact only during Rekey
• Bulk Data encryption occurs on PHY

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
MACsec PSK
Demo
 MACsec Demo Steps
• Show Inteface Config
• Show MACsec Playbook in VS Code
• Run MACsec Playbook
ansible-playbook -i inventories/[Link] playbooks/[Link] --
ask-vault-pass

• Show MKA session on Cat9K

• Show MACsec session on Cat9K

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 • Harden NTP
• VTY ACLs
Other Hardening • YANG Interface ACL
Best Practices … • HTTP Modules
• Config Archiving
• SYSLOG over TLS

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 Question &
Answer Time

DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 • Breathe, Relax

3,2,1 • Scan QR Codes

Action Items
• Cisco Communities

• GitHub Repo
DEVNET-2111

• Test Playbooks

• Execute
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
DEVNET-2111 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Thank you