NSX Advanced Load

Balancer: Install,
Configure, Manage
Lecture Manual
NSX Advanced Load Balancer 18.2

VMware® Education Services

VMware, Inc.
[Link]/education
VMware Confidential - Internal Only
VMware NSX Advanced Load Balancer: Install, Configure, Manage

Lecture Manual

NSX Advanced Load Balancer 18.2

EDU-EN-NSXALBICM-LAB (08/2020)

Copyright © 2020 VMware, Inc. All rights reserved. This manual and its accompanying materials are
protected by U.S. and international copyright and intellectual property laws. VMware products are covered
by one or more patents listed at [Link] VMware is a registered trademark or
trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names
mentioned herein may be trademarks of their respective companies. VMware Cloud™ on AWS, VMware
Cloud™ on AWS GovCloud (US, VMware Cloud™ on AWS Outposts, VMware ESX®, VMware ESXi™,
VMware Go™, VMware Horizon® View™, VMware NSX®, VMware NSX® Advanced Load Balancer
Controller™, VMware NSX® Advanced Load Balancer™, VMware NSX® Data Center, VMware NSX® Data
Center for vSphere®, VMware NSX® Manager™, VMware NSX-T™ Data Center, VMware vCenter Server®,
VMware vCenter®, VMware Verify™, VMware View®, VMware vSphere®, VMware vSphere® Client™,
VMware vSphere® Distributed Resource Scheduler™, VMware vSphere® Distributed Switch™, VMware
vSphere® vMotion®, are registered trademarks or trademarks of VMware, Inc. in the United States and/or
other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective
companies.

The training material is provided “as is,” and all express or implied conditions, representations, and warranties,
including any implied warranty of merchantability, fitness for a particular purpose or noninfringement, are
disclaimed, even if VMware, Inc., has been advised of the possibility of such claims. This training material is
designed to support an instructor-led training course and is intended to be used for reference purposes in
conjunction with the instructor-led training course.

The training material is not a standalone training tool. Use of the training material for self-study without class
attendance is not recommended. These materials and the computer programs to which it relates are the
property of, and embody trade secrets and confidential information proprietary to, VMware, Inc., and may
not be reproduced, copied, disclosed, transferred, adapted or modified without the express written approval
of VMware, Inc.

[Link]/education

VMware Confidential - Internal Only

Contents

Module 1 Course Introduction ........................................................................................... 1

1-2 Course Introduction ............................................................................................................................... 1
1-3 Importance ................................................................................................................................................ 1
1-4 Learner Objectives (1) .......................................................................................................................... 2
1-5 Learner Objectives (2) ......................................................................................................................... 2
1-6 Course Outline ........................................................................................................................................ 3
1-7 Typographical Conventions ............................................................................................................... 4
1-8 Reference (1) ........................................................................................................................................... 5
1-9 Reference (2) .......................................................................................................................................... 5
1-10 Reference (3) .......................................................................................................................................... 6
1-11 Reference (4) .......................................................................................................................................... 6
1-12 VMware Online Resources (1) ........................................................................................................... 7
1-13 VMware Online Resources (2) .......................................................................................................... 7
1-14 Environment Considerations ............................................................................................................. 8
1-15 VMware Education Overview ........................................................................................................... 9
1-16 VMware Certification Overview..................................................................................................... 10

Module 2 Introduction to NSX Advanced Load Balancer (Avi Networks) ... 11

2-2 Module Lessons..................................................................................................................................... 11
2-3 Lesson 1: Introduction to NSX Advanced Load Balancer (Avi Networks) .................... 12
2-4 Unique Features of NSX Advanced Load Balancer ............................................................... 12
2-5 Lesson 2: NSX Advanced Load Balancer (Avi Networks) Architecture and Components
13
2-6 Modern Distributed Architecture ................................................................................................... 13
2-7 NSX Advanced Load Balancer (Avi Networks) Components ........................................... 15
2-8 Lesson 3: NSX Advanced Load Balancer (Avi Networks) Controller ............................. 17
2-9 NSX Advanced Load Balancer (Avi Networks) Controller Details ................................... 17

VMware Confidential - Internal Only iii

2-10 NSX Advanced Load Balancer (Avi Networks) Controller Services (1) ......................... 19
2-11 NSX Advanced Load Balancer (Avi Networks) Controller Services (2) ........................ 19
2-12 NSX Advanced Load Balancer (Avi Networks) Controller Services (3) ....................... 20
2-13 Controller High Availability: Cluster ............................................................................................... 21
2-14 NSX Advanced Load Balancer (Avi Networks) Controller Sizing .................................... 22
2-15 Lesson 4: NSX Advanced Load Balancer (Avi Networks) Service Engine .................. 23
2-16 NSX Advanced Load Balancer (Avi Networks) Service Engine Details (1) .................. 23
2-17 NSX Advanced Load Balancer (Avi Networks) Service Engine Details (2) ................. 23
2-18 NSX Advanced Load Balancer (Avi Networks) Service Engine Details (3) ................. 24
2-19 NSX Advanced Load Balancer (Avi Networks) Service Engine Details (4) ................. 25
2-20 NSX Advanced Load Balancer (Avi Networks) Service Engine Group (1) ................... 26
2-21 NSX Advanced Load Balancer (Avi Networks) Service Engine Group (2) .................. 27
2-22 SE High Availability Modes .............................................................................................................. 29
2-23 NSX Advanced Load Balancer (Avi Networks) Service Engine Sizing .......................... 30
2-24 Lesson 5: NSX Advanced Load Balancer (Avi Networks) Common Terminologies . 31
2-25 NSX Advanced Load Balancer Terminology and System Components ........................ 31
2-26 Terminology Comparison ................................................................................................................ 32
2-27 Lesson 6: NSX Advanced Load Balancer (Avi Networks) Service Elements ............. 33
2-28 About NSX Advanced Load Balancer (Avi Networks) Service Elements.................... 33
2-29 Lesson 7: NSX Advanced Load Balancer (Avi Networks) Clouds .................................. 35
2-30 About NSX Advanced Load Balancer (Avi Networks) Clouds......................................... 35
2-31 Deployment Types ............................................................................................................................ 36
2-32 SE Image Types (1) ............................................................................................................................ 37
2-33 SE Image Types (2) ........................................................................................................................... 38
2-34 SE Image Types (3) ........................................................................................................................... 39
2-35 Lesson 8: Overview of Tenants .................................................................................................... 40
2-36 About Tenants .....................................................................................................................................40
2-37 Lesson 9: NSX Advanced Load Balancer (Avi Networks) Logs ....................................... 41
2-38 About Logs (1) ...................................................................................................................................... 41
2-39 About Logs (2) .................................................................................................................................... 42
2-40 NSX Advanced Load Balancer (Avi Networks) Logs .......................................................... 43
2-41 Lesson 10: NSX Advanced Load Balancer (Avi Networks) Service Health ................. 44
2-42 Health Score (1) ................................................................................................................................... 44
2-43 Health Score (2) .................................................................................................................................. 45
2-44 Lesson 11: Overview of the GUI .................................................................................................... 46
2-45 Main Elements ...................................................................................................................................... 46

iv
VMware Confidential - Internal Only
2-46 Table Views .......................................................................................................................................... 47
2-47 Analytics and Informational Views ............................................................................................... 48
2-48 Configuration Editing Views............................................................................................................ 49
2-49 Activity: GUI Orientation .................................................................................................................. 50
2-50 References ............................................................................................................................................. 51

Module 3 Virtual Services Configuration Concepts .............................................. 53

3-2 Module Lessons................................................................................................................................... 53
3-3 Lesson 1: Virtual Services Configuration Concepts................................................................ 54
3-4 About Virtual Service ........................................................................................................................ 54
3-5 Components of a Virtual Service .................................................................................................. 55
3-6 About a Pool ........................................................................................................................................ 56
3-7 Components of a Pool...................................................................................................................... 57
3-8 Default Configuration ........................................................................................................................ 58
3-9 Lesson 2: Understanding Virtual Services ................................................................................. 59
3-10 Listening Services (1) ......................................................................................................................... 59
3-11 Listening Services (2) ........................................................................................................................ 59
3-12 Application Types............................................................................................................................... 60
3-13 Virtual Service Types (1) ................................................................................................................... 61
3-14 Virtual Service Types (2) ................................................................................................................. 62
3-15 Application Types............................................................................................................................... 63
3-16 Back-End Communications: Introduction to Pools ................................................................. 64
3-17 Back-End Communications: Load-Balancing Algorithms ..................................................... 65
3-18 Back-End Communications: Session Persistence ................................................................... 66
3-19 Handling Failure: Health Monitors.................................................................................................. 67
3-20 Handling Failure: Passive Health Monitor ................................................................................... 68
3-21 Handling Failure: Active Health Monitors ................................................................................... 69
3-22 Lesson 2: Virtual Service Creation ............................................................................................... 70
3-23 Virtual Service Dashboard ............................................................................................................... 70
3-24 Virtual Services Tab ............................................................................................................................ 71
3-25 Virtual Service Creation.................................................................................................................... 72
3-26 Virtual Service Creation: Basic Mode (1) .................................................................................... 73
3-27 Virtual Service Creation: Basic Mode (2) ................................................................................... 74

Module 4 Profiles and Policies ...................................................................................... 77

4-2 Module Lessons....................................................................................................................................77
4-3 Lesson 1: Introduction to Profiles (1)............................................................................................ 78

VMware Confidential - Internal Only v

4-4 Introduction to Profiles (2) .............................................................................................................. 78
4-5 Advanced Virtual Service Creation.............................................................................................. 79
4-6 Advanced Virtual Service Creation: Settings (1) ..................................................................... 80
4-7 Advanced Virtual Service Creation: Settings (2) ..................................................................... 81
4-8 Introduction to Profiles ..................................................................................................................... 82
4-9 Introduction to Profiles: Application and Network Profiles ................................................ 83
4-10 Introduction to Profiles: Persistence Profiles ........................................................................... 85
4-11 Introduction to Profiles: Health Monitor Profile ....................................................................... 87
4-12 Introduction to Profiles: Analytics Profile .................................................................................. 88
4-13 Lesson 2: Configuring and Using Profiles .................................................................................. 89
4-14 Configuring and Using Profiles (1) ................................................................................................. 89
4-15 Configuring and Using Profiles (2) ................................................................................................ 90
4-16 Configuring and Using Profiles: Application Profile ................................................................. 91
4-17 Configuring and Using Profiles: TCP/UDP (1) .......................................................................... 92
4-18 Configuring and Using Profiles: TCP/UDP (2) ......................................................................... 93
4-19 Configuring and Using Profiles: Persistence Profile (1) ......................................................... 94
4-20 Configuring and Using Profiles: Persistence Profile (2) ........................................................ 95
4-21 Lesson 3: TCP/UDP Profiles .......................................................................................................... 96
4-22 L4: Proxy ............................................................................................................................................... 96
4-23 TCP/UDP Profile Creation (1) ........................................................................................................ 97
4-24 Network Profile: TCP Proxy Versus TCP Fast Path Flow (1) ........................................... 98
4-25 Network Profile: TCP Proxy Versus TCP Fast Path Flow (2) .......................................... 99
4-26 Network Profile: L4 UDP Fast and UDP Per Packet .......................................................... 100
4-27 TCP/UDP Profile Creation (2) ...................................................................................................... 101
4-28 TCP/UDP Profile Creation (3) ...................................................................................................... 101
4-29 Advanced Topics: Server Network Profile ............................................................................. 104
4-30 TCP/UDP Profile Creation ............................................................................................................ 104
4-31 Lesson 4: Application Profiles...................................................................................................... 105
4-32 L4 and L4 SSL/TLS Application Profiles................................................................................. 105
4-33 DNS Application Profile .................................................................................................................. 106
4-34 Introduction to HTTP Application Profile ................................................................................ 107
4-35 Introduction to HTTP Profile: General Tab ............................................................................. 108
4-36 Introduction to HTTP Profile: Security Tab ............................................................................ 109
4-37 Introduction to HTTP Profile: Compression ............................................................................ 110
4-38 Introduction to HTTP Profile: Caching ......................................................................................... 111
4-39 Introduction to HTTP Profile: DDoS ............................................................................................ 112

vi
VMware Confidential - Internal Only
4-40 HTTP Profile Use Cases .................................................................................................................. 113
4-41 Lesson 5: SSL Profile and Certificate Overview.....................................................................114
4-42 Introduction to SSL Profile and Certificate ...............................................................................114
4-43 SSL Profile Overview (1) .................................................................................................................. 115
4-44 SSL Profile Overview (2) .................................................................................................................116
4-45 SSL Certificate Overview (1) .......................................................................................................... 117
4-46 SSL Certificate Overview (2) .........................................................................................................118
4-47 SSL Certificate Use Cases ..............................................................................................................118
4-48 Lesson 6: SSL Re-Encryption ........................................................................................................119
4-49 SSL Re-Encryption Overview ........................................................................................................119
4-50 Configuring SSL Re-Encryption................................................................................................... 120
4-51 SSL Re-Encryption Use Case ........................................................................................................ 121
4-52 Lesson 7: Acceleration Technologies ........................................................................................ 122
4-53 Introduction Acceleration Technologies ................................................................................... 122
4-54 Lesson 8: HTTP Compression ...................................................................................................... 123
4-55 HTTP Compression Tab ................................................................................................................. 123
4-56 HTTP Compression Mode and Options ....................................................................................124
4-57 HTTP Compression: Custom Setting ......................................................................................... 125
4-58 HTTP Compression Filter and Options .....................................................................................126
4-59 Lesson 9: HTTP Caching ................................................................................................................128
4-60 HTTP Caching (1) ...............................................................................................................................128
4-61 HTTP Caching (2) ..............................................................................................................................129
4-62 HTTP Caching (3) ............................................................................................................................. 130
4-63 HTTP Caching Tab and Configuration ........................................................................................ 131
4-64 HTTP Caching CLI Commands ..................................................................................................... 133
4-65 Lesson 10: Connection Multiplexing ............................................................................................134
4-66 Introduction to Connection Multiplexing (1) .............................................................................134
4-67 Introduction to Connection Multiplexing (2) ............................................................................134
4-68 Lesson 11: Policies Overview .........................................................................................................136
4-69 Virtual Service Policies: Overview ...............................................................................................136
4-70 Virtual Service Policies: Rules ........................................................................................................ 137
4-71 HTTP Virtual Service Policies (1)..................................................................................................138
4-72 HTTP Virtual Service Policies (2).................................................................................................138
4-73 HTTP Virtual Service Policies: Network Security Rule (1) .................................................. 139
4-74 HTTP Virtual Service Policies: Network Security Rule (2) ................................................ 140
4-75 HTTP Virtual Service Policies: HTTP Security Rule (1) .........................................................141

VMware Confidential - Internal Only vii

4-76 HTTP Virtual Service Policies: HTTP Security Rule (2) .......................................................142
4-77 HTTP Virtual Service Policies: HTTP Security Rule (3) .......................................................143
4-78 HTTP Virtual Service Policies: HTTP Security Rule (4)...................................................... 144
4-79 HTTP Virtual Service Policies: HTTP Security Rule (5) .......................................................145
4-80 HTTP Virtual Service Policies: HTTP Security Rule (6) .......................................................146
4-81 HTTP Virtual Service Policies: HTTP Request Rule (1)........................................................147
4-82 HTTP Virtual Service Policies: HTTP Request Rule (2).......................................................148
4-83 HTTP Virtual Service Policies: HTTP Request Rule (3).......................................................149
4-84 HTTP Virtual Service Policies: HTTP Request Rule (4) ..................................................... 150
4-85 HTTP Virtual Service Policies: HTTP Request Rule (5)........................................................ 151
4-86 HTTP Virtual Service Policies: HTTP Request Rule (6) ...................................................... 152
4-87 HTTP Virtual Service Policies: HTTP Request Rule (7)....................................................... 153
4-88 HTTP Virtual Service Policies: HTTP Response Rule (1) ....................................................154
4-89 HTTP Virtual Service Policies: HTTP Response Rule (2) ................................................... 155
4-90 HTTP Virtual Service Policies: HTTP Response Rule (3) ................................................... 156
4-91 HTTP Virtual Service Policies: HTTP Response Rule (4) ................................................... 157
4-92 HTTP Virtual Service Policies: HTTP Response Rule (5) ................................................... 158
4-93 HTTP Virtual Service Policies: HTTP Response Rule (6) ................................................... 159
4-94 DNS Virtual Service Policies (1) ................................................................................................... 160
4-95 DNS Virtual Service Policies (2) .................................................................................................. 160
4-96 DNS Virtual Service Policies: DNS Rule (1) ................................................................................161
4-97 DNS Virtual Service Policies: DNS Rule (2) ..............................................................................162
4-98 DNS Virtual Service Policies: DNS Rule (3) ..............................................................................163
4-99 DNS Virtual Service Policies: DNS Rule (4)..............................................................................164
4-100 DNS Virtual Service Policies: DNS Rule (5) ..............................................................................165
4-101 DNS Virtual Service Policies: DNS Rule (6) ..............................................................................166
4-102 DNS Virtual Service Policies: DNS Rule (7) .............................................................................. 167
4-103 References ...........................................................................................................................................168

Module 5 Pools Configuration Concepts ................................................................ 169

5-2 Module Overview ..............................................................................................................................169
5-3 Lesson 1: Pools Overview ............................................................................................................. 170
5-4 About Pools (1) .................................................................................................................................. 170
5-5 About Pools (2) ................................................................................................................................. 170
5-6 Lesson 2: Pool Configuration ......................................................................................................... 171
5-7 Creating a Pool: Settings ................................................................................................................. 171
5-8 Creating a Pool: Servers ................................................................................................................. 172
viii
VMware Confidential - Internal Only
5-9 Creating a Pool: Advanced ............................................................................................................ 173
5-10 Creating a Pool: Review .................................................................................................................. 176
5-11 Load-Balancing Algorithms (1) ...................................................................................................... 179
5-12 Load-Balancing Algorithms (2) .................................................................................................... 180
5-13 Persistence Algorithms (1)...............................................................................................................181
5-14 Persistence Algorithms (2)..............................................................................................................181
5-15 Health Monitors ..................................................................................................................................182
5-16 Passive Health Monitor ....................................................................................................................183
5-17 Active Health Monitors ....................................................................................................................184
5-18 DNS Monitor ........................................................................................................................................185
5-19 Ping Monitor.........................................................................................................................................186
5-20 TCP Monitor ........................................................................................................................................187
5-21 UDP Monitor ........................................................................................................................................188
5-22 HTTP Monitor (1) ...............................................................................................................................189
5-23 HTTP Monitor (2) ..............................................................................................................................189
5-24 HTTP Monitor (3) ............................................................................................................................. 190
5-25 External Monitor (1) ............................................................................................................................191
5-26 External Monitor (2) ...........................................................................................................................191
5-27 External Monitor (3) ...........................................................................................................................191
5-28 SIP Monitor...........................................................................................................................................192
5-29 RADIUS Monitor .................................................................................................................................193
5-30 Lesson 3: Pool Groups ....................................................................................................................194
5-31 Pool Groups Overview (1) ..............................................................................................................194
5-32 Pool Groups Overview (2) .............................................................................................................195
5-33 Pool Groups Configuration.............................................................................................................196
5-34 Pool Group Best Practices .............................................................................................................196
5-35 Pool or Pool Groups .........................................................................................................................197
5-36 References ...........................................................................................................................................197

Module 6 Modifying Application Behavior .............................................................. 199

6-2 Module Lessons..................................................................................................................................199
6-3 Lesson 1: Controlling Traffic with NSX Advanced Load Balancer (1) ........................... 200
6-4 Controlling Traffic with NSX Advanced Load Balancer (2) .............................................. 201
6-5 Controlling Traffic with NSX Advanced Load Balancer (3) ............................................. 202
6-6 Controlling Traffic with NSX Advanced Load Balancer (4) ............................................. 202
6-7 Controlling Traffic with NSX Advanced Load Balancer (5) ............................................. 203
6-8 Controlling Traffic with NSX Advanced Load Balancer (6) ............................................. 203

VMware Confidential - Internal Only ix

6-9 Lesson 2: L7 Traffic Manipulation .............................................................................................. 204
6-10 App Teams Need to Know Who Is Connecting .................................................................. 204
6-11 X-Forwarded-For (1) ...................................................................................................................... 204
6-12 X-Forwarded-For (2) ..................................................................................................................... 205
6-13 X-Forwarded-For (3) ..................................................................................................................... 206
6-14 Validating That Headers Are Inserted ..................................................................................... 207
6-15 Capturing and Examining Headers (1) ...................................................................................... 207
6-16 Capturing and Examining Headers (2) ..................................................................................... 208
6-17 Capturing and Examining Headers (3) ..................................................................................... 208
6-18 Capturing and Examining Headers (4) ..................................................................................... 209
6-19 Minimizing the Number of Connections to Back-End Servers ......................................... 210
6-20 Connection Multiplexing (1) ............................................................................................................. 211
6-21 Connection Multiplexing (2) ............................................................................................................ 211
6-22 Traffic Needs to Be HTTPS but HTTP Traffic Cannot Be Missed .................................. 212
6-23 HTTP to HTTPS Redirection ........................................................................................................ 212
6-24 HTTP to HTTPS Redirection: App Profile ............................................................................... 213
6-25 HTTP to HTTPS Redirection: HTTP Security Policy ...........................................................214
6-26 Blocking Bad Actors ......................................................................................................................... 215
6-27 Using Network Security Policies to Denylist (1) ..................................................................... 215
6-28 Using Network Security Policies to Denylist (2) .................................................................... 216
6-29 Using Network Security Policies to Denylist (3) .................................................................... 217
6-30 Using Network Security Policies to Denylist (4) .................................................................... 218
6-31 Using Network Security Policies to Denylist (5) .................................................................... 219
6-32 App Team Wants Certain Traffic to Be Processed Apart from Their Main App .... 220
6-33 HTTP Redirects ................................................................................................................................ 220
6-34 Finding Other Options ..................................................................................................................... 221
6-35 Content Switching (1) ....................................................................................................................... 221
6-36 Content Switching (2) ..................................................................................................................... 222
6-37 Options Other Than Policies......................................................................................................... 223
6-38 DataScripts..........................................................................................................................................223
6-39 Example DataScripts .......................................................................................................................224
6-40 DataScripts..........................................................................................................................................225
6-41 App Owner Wants to Authenticate Client Connections (1) ............................................. 226
6-42 App Owner Wants to Authenticate Client Connections (2) ............................................ 226
6-43 Client SSL Certificate Validation ................................................................................................. 227
6-44 Client SSL Certificate Validation: Requirements (1) ............................................................. 227

x
VMware Confidential - Internal Only
6-45 Client SSL Certificate Validation: Requirements (2) ............................................................ 228
6-46 Client SSL Certificate Validation: PKI Profile (1) .................................................................... 229
6-47 Client SSL Certificate Validation: PKI Profile (2) ................................................................... 229
6-48 Client SSL Certificate Validation: PKI Profile (3) .................................................................. 230
6-49 Client SSL Certificate Validation: PKI Profile (4) .................................................................... 231
6-50 Client SSL Certificate Validation: Putting It All Together (1) ............................................ 232
6-51 Client SSL Certificate Validation: Putting It All Together (2) ........................................... 233
6-52 Client SSL Certificate Validation: Putting It All Together (3) ...........................................234
6-53 Client SSL Certificate Validation: Putting It All Together (4) ........................................... 235
6-54 Client SSL Certificate Validation: Putting It All Together (5) ........................................... 236
6-55 Traffic Speed...................................................................................................................................... 237
6-56 Rate-Limiting Traffic ........................................................................................................................ 237
6-57 Throttling: Application Profile (1) .................................................................................................238
6-58 Throttling: Application Profile (2) ................................................................................................239
6-59 Throttling: Virtual Service ............................................................................................................. 240
6-60 App Teams Complain That Posts Are Too Big .....................................................................241
6-61 App Team Needs to Limit POST Message Bodies ..............................................................241
6-62 Client Post Body Size .....................................................................................................................242
6-63 Lesson 3: L4 Traffic Manipulation ...............................................................................................243
6-64 Handling Idle Connections .............................................................................................................243
6-65 Idle Connection Behavior...............................................................................................................243
6-66 Idle Connection Types (1) ............................................................................................................. 244
6-67 Idle Connection Types (2) .............................................................................................................245
6-68 Users Are Seeing 403s After Successful Authentication .................................................246
6-69 Session Persistence .........................................................................................................................246
6-70 Lesson 4: Customizing Application Delivery with DataScripts ........................................247
6-71 Overview .............................................................................................................................................247
6-72 Lesson 5: Getting Started with DataScripts ...........................................................................248
6-73 DataScripts..........................................................................................................................................248
6-74 DataScripts for the Data Plane (1) ..............................................................................................249
6-75 DataScripts for the Data Plane (2) .............................................................................................249
6-76 Supported Events ........................................................................................................................... 250
6-77 Lesson 6: DataScripts Constructs ............................................................................................... 251
6-78 DataScripts Constructs: Logic and Operators........................................................................ 251
6-79 DataScripts Constructs: Variables .............................................................................................. 252
6-80 DataScripts Constructs: Functions (1) ....................................................................................... 252

VMware Confidential - Internal Only xi

6-81 DataScripts Constructs: Functions (2) ...................................................................................... 253
6-82 DataScripts Constructs: Functions (3) ......................................................................................254
6-83 DataScripts: Attaching Objects ................................................................................................... 255
6-84 Lesson 7: Using DataScripts Examples ....................................................................................256
6-85 DataScript Example .........................................................................................................................256
6-86 Public DataScript Examples Library (1) ..................................................................................... 257
6-87 Public DataScript Examples Library (2) ....................................................................................258
6-88 Public DataScript Examples Library (3) ....................................................................................259

Module 7 NSX Advanced Load Balancer Infrastructure Architecture ........ 261

7-2 Module Lessons..................................................................................................................................261
7-3 Lesson 1: Architecture Overview ...............................................................................................262
7-4 Deployment Process .......................................................................................................................262
7-5 High-Level Component Parts ......................................................................................................263
7-6 Very High-Level Component Parts ...........................................................................................264
7-7 Data Plane Overview and Automation ....................................................................................265
7-8 Lesson 2: Control Plane .................................................................................................................266
7-9 NSX Advanced Load Balancer Controller (1).........................................................................266
7-10 NSX Advanced Load Balancer Controller (2)........................................................................ 267
7-11 Controller Clustering ........................................................................................................................268
7-12 Controller High Availability (1) ......................................................................................................269
7-13 Controller High Availability (2) .................................................................................................... 270
7-14 Recover a Nonoperational Cluster .............................................................................................. 271
7-15 Controller Cluster: One Versus Multiple ................................................................................... 272
7-16 Controller Process Sharding (1) ................................................................................................... 273
7-17 Controller Process Sharding (2) .................................................................................................. 273
7-18 Controller Sizing (1) ..........................................................................................................................274
7-19 Controller Sizing (2) .........................................................................................................................274
7-20 Controller Operations ..................................................................................................................... 275
7-21 Lesson 3: Data Plane .......................................................................................................................276
7-22 Service Engine ...................................................................................................................................276
7-23 Service Engine NIC Architecture (1) .......................................................................................... 277
7-24 Service Engine NIC Architecture (2) ......................................................................................... 278
7-25 Service Engine NIC Architecture (3) ......................................................................................... 279
7-26 Service Engine CPU Architecture (1) ....................................................................................... 280
7-27 Service Engine CPU Architecture (2) ...................................................................................... 280
7-28 Service Engine CPU Architecture (3) ........................................................................................281
xii
VMware Confidential - Internal Only
7-29 Lesson 4: Tenants Overview .......................................................................................................282
7-30 About Tenants (1) .............................................................................................................................282
7-31 About Tenants (2) ............................................................................................................................283
7-32 Lesson 5: Service Engine Group Settings ...............................................................................284
7-33 Introduction to Service Engine Groups ....................................................................................284
7-34 Configuring Service Engine Groups: VMware Cloud ...........................................................286
7-35 Lesson 6: High Availability Modes .............................................................................................. 287
7-36 High Availability Modes (1)............................................................................................................. 287
7-37 High Availability Modes (2)............................................................................................................ 287
7-38 High Availability Modes (3) ...........................................................................................................288
7-39 High Availability Modes (4) ...........................................................................................................289
7-40 High Availability Modes (5) .......................................................................................................... 290
7-41 High Availability Modes (6) ............................................................................................................291
7-42 Elastic HA N+M Mode .....................................................................................................................292
7-43 Elastic HA N+M Example ...............................................................................................................293
7-44 Elastic HA N+M Mode Recommendation ................................................................................294
7-45 Elastic HA Active-Active Mode (1) ............................................................................................294
7-46 Elastic HA Active-Active Mode (2) ...........................................................................................294
7-47 Elastic HA Active-Active Example.............................................................................................295
7-48 Elastic HA Active-Active Mode Recommendation..............................................................296
7-49 Legacy HA Active-Standby Mode ............................................................................................296
7-50 Compact Versus Distributed Placement .................................................................................. 297
7-51 Impact of Placement on N+M Mode..........................................................................................298
7-52 Impact of Placement on Active-Active Mode .......................................................................299
7-53 Use Cases ............................................................................................................................................299
7-54 Determining SEs That Are Handling Traffic for a VS (1) ................................................... 300
7-55 Determining SEs That Are Handling Traffic for a VS (2) .................................................. 300
7-56 Lesson 7: Service Engine Failure and Self-Healing............................................................... 301
7-57 Controller to SE Failure Detection Method ............................................................................ 301
7-58 Failure Detection Algorithm (1) ................................................................................................... 302
7-59 Failure Detection Algorithm (2) .................................................................................................. 302
7-60 Elastic HA N+M Failure .................................................................................................................. 303
7-61 Elastic HA Active-Active Failure ............................................................................................... 304
7-62 Legacy HA Active-Standby Failure .......................................................................................... 305
7-63 Lesson 8: Service Engine Groups: Networks and Routing............................................... 306
7-64 SE Group Network Settings ........................................................................................................ 306

VMware Confidential - Internal Only xiii

7-65 SE Group Static Route .................................................................................................................. 307
7-66 Auto Gateway .................................................................................................................................. 308
7-67 Lesson 9: Service Engine As a Router .................................................................................... 309
7-68 Service Engine As a Router (1) ................................................................................................... 309
7-69 Service Engine As a Router (2) .................................................................................................. 309
7-70 Default Gateway: SE As a Router Example ........................................................................... 310
7-71 Service Engine As a Router (1) ...................................................................................................... 311
7-72 Service Engine As a Router (2) ..................................................................................................... 311
7-73 Lesson 10: Scale-Out Mechanisms: L2 Versus L3 ................................................................. 312
7-74 L2 SE Native Scaling ........................................................................................................................ 312
7-75 Native Scale-out.................................................................................................................................313
7-76 Network/SDN-Based Scale-Out .................................................................................................314
7-77 L3 SE ECMP Scaling .........................................................................................................................315
7-78 BGP Support for Elastic HA (1) ....................................................................................................316
7-79 BGP Support for Elastic HA (2) ...................................................................................................316
7-80 BGP Support for Elastic HA (3) ................................................................................................... 317
7-81 DNS-Based Scale-Out .....................................................................................................................318
7-82 Lesson 11: Upgrade 2.0 ....................................................................................................................319
7-83 Upgrade Preparation ........................................................................................................................319
7-84 Upgrade............................................................................................................................................... 320
7-85 Controller Upgrade ........................................................................................................................... 321
7-86 Service Engine Upgrade (Before 18.2.6).................................................................................. 322
7-87 Service Engine Upgrade (After 18.2.6)..................................................................................... 323
7-88 Service Disruption ............................................................................................................................324
7-89 Rollback (Before 18.2.6) ................................................................................................................. 325
7-90 Rollback (After 18.2.6) .................................................................................................................... 325
7-91 Patch Versus Upgrade....................................................................................................................326

Module 8 Introduction to Cloud Connector .......................................................... 327

8-2 Cloud Connector Configuration Concepts .............................................................................. 327
8-3 About Clouds (1) ...............................................................................................................................328
8-4 About Clouds (2) ..............................................................................................................................328
8-5 Additional Clouds ..............................................................................................................................329
8-6 Choosing the Right Cloud ............................................................................................................. 330
8-7 SE Image Types (1) ...........................................................................................................................331
8-8 SE Image Types (2) .........................................................................................................................332
8-9 Public Cloud (1) ..................................................................................................................................333
xiv
VMware Confidential - Internal Only
8-10 Public Cloud (2) .................................................................................................................................334
8-11 Public Cloud (3) .................................................................................................................................335
8-12 Public Cloud (4) .................................................................................................................................336
8-13 Public Cloud (5) .................................................................................................................................336
8-14 Cloud Connector Configuration Concepts (1) ........................................................................ 337
8-15 Cloud Connector Configuration Concepts (2) ....................................................................... 337
8-16 Cloud Connector Configuration Concepts (3) .......................................................................339
8-17 References ......................................................................................................................................... 340

Module 9 Installing, Configuring, and Managing NSX Advanced Load Balancer

in No-Access Clouds ......................................................................................................... 341
9-2 Module Lessons..................................................................................................................................341
9-3 Lesson 1: Cloud Configuration......................................................................................................342
9-4 No-Access Cloud ..............................................................................................................................342
9-5 Lesson 2: No-Access Service Engines .....................................................................................343
9-6 Linux Bare Metal (1) ..........................................................................................................................343
9-7 Hardware Requirements ............................................................................................................... 344
9-8 Linux Bare Metal (2) .........................................................................................................................345
9-9 No Access (1) .....................................................................................................................................346
9-10 No Access (2) ....................................................................................................................................347
9-11 No-Access Cloud ..............................................................................................................................348
9-12 Lesson 3: Linux Server Cloud ......................................................................................................349
9-13 Linux Server Cloud (1) .....................................................................................................................349
9-14 Linux Server Cloud (2) ................................................................................................................... 350
9-15 Linux Server Cloud (3) ..................................................................................................................... 351
9-16 Linux Server Cloud (4).................................................................................................................... 352
9-17 Linux Server Cloud (5) .................................................................................................................... 353
9-18 Lesson 4: VMware No-Access ....................................................................................................354
9-19 No Access (1) .....................................................................................................................................354
9-20 No Access (2) ....................................................................................................................................355
9-21 No Access (3) ....................................................................................................................................356
9-22 No Access (4) .................................................................................................................................... 357
9-23 No Access (5) ....................................................................................................................................358
9-24 Lesson 5: Advanced Features.....................................................................................................359
9-25 Advanced Features (1) ...................................................................................................................359
9-26 TSO, GRO, and RSS (1) ................................................................................................................. 360
9-27 TSO, GRO, and RSS (2) ..................................................................................................................361

VMware Confidential - Internal Only xv

9-28 TSO, GRO, and RSS (3) .................................................................................................................362
9-29 Multiple Dispatchers .........................................................................................................................363
9-30 Denylisting (1) .....................................................................................................................................364
9-31 Denylisting (2) ....................................................................................................................................364
9-32 VLAN Interfaces ...............................................................................................................................365
9-33 Port Channels (1) ...............................................................................................................................366
9-34 Port Channels (2) ..............................................................................................................................366
9-35 MAC Masquerade (1) .......................................................................................................................368
9-36 MAC Masquerade (2) ......................................................................................................................368

Module 10 Installing, Configuring, and Managing NSX Advanced Load Balancer

in VMware Environments: Cloud Configuration ..................................................... 369
10-2 Module Lessons.................................................................................................................................369
10-3 Lesson 1: Overview of vSphere ................................................................................................. 370
10-4 VMware Features (1) ........................................................................................................................ 371
10-5 VMware Features (2) ...................................................................................................................... 372
10-6 VMware No-Access Cloud............................................................................................................ 373
10-7 Lesson 2: VMware Write Cloud ..................................................................................................374
10-8 VMware Write Cloud: vCenter Server Integration ..............................................................374
10-9 vCenter Server Integration: NSX Advanced Load Balancer Controller (1) ................ 375
10-10 vCenter Server Integration: NSX Advanced Load Balancer Controller (2) ............... 375
10-11 vCenter Server Integration: NSX Advanced Load Balancer Controller (3) ............... 375
10-12 vCenter Server Integration: Service Engine ........................................................................... 376
10-13 Lesson 3: Deployment Prerequisites ........................................................................................ 377
10-14 VMware Write Cloud....................................................................................................................... 377
10-15 Deployment Prerequisites: VM Requirements ....................................................................... 378
10-16 Deployment Prerequisites: IP Address Requirements ....................................................... 378
10-17 Deployment Prerequisites: vCenter Server Account Requirements ............................ 379
10-18 Lesson 4: Deployment................................................................................................................... 380
10-19 Deployment (1) ................................................................................................................................. 380
10-20 Deployment (2) ..................................................................................................................................381
10-21 Deploying the Controller ................................................................................................................382
10-22 VMware Write Cloud (1).................................................................................................................383
10-23 VMware Write Cloud (2)................................................................................................................384
10-24 VMware Write Cloud (3)................................................................................................................385
10-25 VMware Write Cloud (4) ...............................................................................................................386
10-26 VMware Write Cloud (5)................................................................................................................ 387
xvi
VMware Confidential - Internal Only
10-27 Lesson 5: Configuring the Service Engine Group .................................................................388
10-28 Configuring the Service Engine Group (1) ...............................................................................388
10-29 Configuring the Service Engine Group (2) ..............................................................................389
10-30 Configuring the Service Engine Group (3) ............................................................................. 390
10-31 Lesson 6: Networks ..........................................................................................................................391
10-32 VMware Write Cloud (1)..................................................................................................................391
10-33 VMware Write Cloud (2)................................................................................................................392
10-34 VMware Write Cloud (3)................................................................................................................393
10-35 VMware Write Cloud (4) ...............................................................................................................394
10-36 Lesson 7: Placement .......................................................................................................................395
10-37 Virtual Service Placement .............................................................................................................395
10-38 One Arm Versus Two Arms .........................................................................................................396
10-39 Virtual Service Placement in Write Access Cloud ............................................................... 397
10-40 Prefer Static Routes Versus Directly Connected Networks (Server) ..........................398
10-41 Use Static Routes for Resolution of Network VIPs (Virtual Service) ...........................399
10-42 VMware Write Cloud......................................................................................................................400
10-43 Best Practices: VMware Write Cloud ....................................................................................... 401
10-44 Lesson 8: NSX Data Center for vSphere SDN Integration: Cloud Configuration.... 402
10-45 NSX Data Center for vSphere (1) .............................................................................................. 402
10-46 NSX Data Center for vSphere (2) ............................................................................................. 403
10-47 VMware Write Access with NSX Data Center for vSphere (1) ..................................... 404
10-48 VMware Write Access with NSX Data Center for vSphere (2) .................................... 405
10-49 VMware Write Access with NSX Data Center for vSphere (3) .................................... 406
10-50 VMware Write Access with NSX Data Center for vSphere (4) .................................... 407
10-51 NSX Data Center for vSphere Integration: Virtual Service Distributed Firewall Rule408
10-52 Lesson 9: NSX-T Data Center Integration: Cloud Configuration ................................... 409
10-53 NSX-T Data Center ........................................................................................................................ 409
10-54 Configuration Steps ......................................................................................................................... 410
10-55 Typical NSX Advanced Load Balancer VMware NSX-T Data Center Design ............411
10-56 Lesson 10: VMware Cloud Connector Options Summary: Cloud Configuration........412
10-57 VMware Cloud Connector Options Summary ........................................................................412

Module 11 AWS Cloud Configuration........................................................................ 413

11-2 AWS Cloud Connector (1) .............................................................................................................413
11-3 AWS Cloud Connector (2) ........................................................................................................... 414
11-4 AWS Cloud (1) ....................................................................................................................................416
11-5 AWS Cloud (2) ...................................................................................................................................417

VMware Confidential - Internal Only xvii

11-6 AWS Cloud (3) ...................................................................................................................................418
11-7 AWS Cloud (4) ...................................................................................................................................419
11-8 AWS Cloud (5) ................................................................................................................................. 420
11-9 AWS Cloud (6) ...................................................................................................................................421
11-10 AWS Cloud (7) ..................................................................................................................................422
11-11 AWS Cloud (8) ..................................................................................................................................423
11-12 AWS Cloud (9) ................................................................................................................................. 424

Module 12 DNS Foundation ......................................................................................... 425

12-2 Module Lessons.................................................................................................................................425
12-3 Lesson 1: DNS Summary ................................................................................................................426
12-4 Issue .......................................................................................................................................................426
12-5 About DNS..........................................................................................................................................427
12-6 DNS Transaction ...............................................................................................................................428
12-7 DNS Resolution .................................................................................................................................429
12-8 DNS Query ......................................................................................................................................... 430
12-9 Lesson 2: NSX Advanced Load Balancer DNS Virtual Service........................................431
12-10 NSX Advanced Load Balancer DNS Virtual Service (1) ......................................................431
12-11 NSX Advanced Load Balancer DNS Virtual Service Deployment .................................432
12-12 NSX Advanced Load Balancer DNS Application Profile ...................................................433
12-13 NSX Advanced Load Balancer DNS Virtual Service (2) ................................................... 434
12-14 NSX Advanced Load Balancer DNS Virtual Service (3) ....................................................435
12-15 NSX Advanced Load Balancer DNS Virtual Service (4) ....................................................436
12-16 Lesson 3: NSX Advanced Load Balancer IPAM and DNS Profile .................................. 437
12-17 NSX Advanced Load Balancer IPAM and DNS Profile (1) ................................................437
12-18 NSX Advanced Load Balancer IPAM and DNS Profile (2) ...............................................438
12-19 NSX Advanced Load Balancer IPAM and DNS Profile (3) ...............................................439
12-20 NSX Advanced Load Balancer IPAM and DNS Profile (4) .............................................. 440
12-21 NSX Advanced Load Balancer IPAM and DNS Profile (5) .............................................. 440

Module 13 Global Server Load Balancer ................................................................. 441

13-2 Module Lessons................................................................................................................................. 441
13-3 Lesson 1: NSX Global Server Load Balancer Introduction ............................................... 442
13-4 About GSLB ...................................................................................................................................... 442
13-5 GSLB Benefits ................................................................................................................................... 443
13-6 GSLB High-Level Functionality ................................................................................................... 444
13-7 Internet-Based DNS Query.......................................................................................................... 445

xviii
VMware Confidential - Internal Only
13-8 GSLB Use Cases (1) ........................................................................................................................ 446
13-9 GSLB Use Cases (2) ....................................................................................................................... 446
13-10 Lesson 2: NSX Advanced Load Balancer Architecture and Objects........................... 447
13-11 Generic GSLB Deployment Architecture ............................................................................... 447
13-12 GSLB Object...................................................................................................................................... 448
13-13 GSLB Object: GSLB Site ............................................................................................................... 449
13-14 GSLB Object: GSLB Site Classification (1) ................................................................................451
13-15 GSLB Object: GSLB Site Classification (2) ..............................................................................452
13-16 GSLB Object: GSLB Site Classification (3) ..............................................................................453
13-17 GSLB Object: GSLB Site Classification (4) ............................................................................. 454
13-18 GSLB Object: GSLB SE Group ....................................................................................................455
13-19 GSLB Object: GSLB Virtual Service...........................................................................................456
13-20 GSLB Object: GSLB Service.........................................................................................................457
13-21 GSLB Object: GSLB Pools (1) ......................................................................................................458
13-22 GSLB Object: GSLB Pools (2) .....................................................................................................459
13-23 GSLB Object: GSLB Pool Members ......................................................................................... 460
13-24 GSLB Object: Weights ....................................................................................................................461
13-25 GSLB Object: Load-Balancing Algorithms ..............................................................................462
13-26 GSLB Object: GSLB Health Monitor ..........................................................................................463
13-27 Lesson 3: Configuring GSLB Infrastructure............................................................................ 464
13-28 Configuration of GSLB Sites (1).................................................................................................. 464
13-29 Configuration of GSLB Sites (2)................................................................................................. 464
13-30 Setup of Individual Controller Clusters .....................................................................................465
13-31 Configuration of GSLB Sites (3)................................................................................................. 466
13-32 Configuration of GSLB Sites (4) ................................................................................................ 466
13-33 Configuration of GSLB Sites (5)..................................................................................................467
13-34 Configuration of GSLB Sites (6)..................................................................................................467
13-35 Configuration of GSLB Sites (7) ................................................................................................. 468
13-36 Configuration of GSLB Sites (8)................................................................................................. 468
13-37 Configuration of GSLB Sites (9)................................................................................................. 469
13-38 Configuration of GSLB Sites (10)............................................................................................... 470
13-39 Configuration of GSLB Sites (11) ................................................................................................ 470
13-40 Configuring a GSLB Service (1) ....................................................................................................471
13-41 Configuring a GSLB Service (2) ...................................................................................................471
13-42 Configuring a GSLB Service (3) ..................................................................................................472
13-43 Configuring a GSLB Service (4) ..................................................................................................473

VMware Confidential - Internal Only xix

13-44 Configuring a GSLB Service (5) ................................................................................................. 474
13-45 Configuring a GSLB Service (6) ..................................................................................................475
13-46 Configuring a GSLB Service (7) ..................................................................................................476
13-47 Configuring a GSLB Service (8) ..................................................................................................477
13-48 Configuring a GSLB Service (9) ..................................................................................................478

Module 14 VMware NSX Advanced Load Balancer: Troubleshooting ...... 479

14-2 Module Lessons.................................................................................................................................479
14-3 Lesson 1: Data Plane Versus Control Plane Troubleshooting ......................................... 480
14-4 Data Plane Versus Control Plane Troubleshooting ............................................................ 480
14-5 Lesson 2: Data Plane Troubleshooting Overview.................................................................481
14-6 Significant and Non-Significant Logs (1) ....................................................................................481
14-7 Significant and Non-Significant Logs (2) ..................................................................................482
14-8 Analytics Profile (1) ..........................................................................................................................483
14-9 Analytics Profile (2) .........................................................................................................................483
14-10 Data Plane Troubleshooting Tools............................................................................................ 484
14-11 Client Logs ..........................................................................................................................................485
14-12 Client Logs: Significant Logs (1) ...................................................................................................485
14-13 Client Logs: Significant Logs (2) ................................................................................................. 486
14-14 Client Logs: Significant Logs (3)..................................................................................................487
14-15 Client Logs: Full Clients (Non-Significant) Logs (1) .............................................................. 488
14-16 Client Logs: Full Clients (Non-Significant) Logs (2) ............................................................. 489
14-17 Client Logs: Navigating Logs Using Search ........................................................................... 490
14-18 Client Logs: Log All Request/Response Headers (1) ...........................................................491
14-19 Client Logs: Log All Request/Response Headers (2) ..........................................................491
14-20 Client Logs: Log All Request/Response Headers (3) .........................................................492
14-21 Virtual Service Log Analytics (1) .................................................................................................493
14-22 Virtual Service Log Analytics (2) ................................................................................................493
14-23 Packet Capture (1) .......................................................................................................................... 494
14-24 Packet Capture (2) ..........................................................................................................................495
14-25 Packet Capture (3) ......................................................................................................................... 496
14-26 Advanced Topics: Traffic Cloning ..............................................................................................497
14-27 Data Plane Troubleshooting with CLI (1) ................................................................................ 498
14-28 Data Plane Troubleshooting with CLI (2) ............................................................................... 498
14-29 Data Plane Troubleshooting with CLI (3) ............................................................................... 499
14-30 Data Plane Troubleshooting with CLI (4) ............................................................................... 500
14-31 Data Plane Troubleshooting with CLI (5) ................................................................................ 501
xx
VMware Confidential - Internal Only
14-32 Data Plane Troubleshooting with CLI (6) ............................................................................... 502
14-33 Data Plane Troubleshooting with CLI (7) ............................................................................... 503
14-34 Service Engine Logs ....................................................................................................................... 504
14-35 Health Monitor Troubleshooting (1) .......................................................................................... 505
14-36 Health Monitor Troubleshooting (2) ......................................................................................... 506
14-37 Health Monitor Troubleshooting (3) ......................................................................................... 507
14-38 BGP Session Troubleshooting .................................................................................................... 508
14-39 BGP Session Troubleshooting: Quagga Shell (1) ................................................................. 509
14-40 BGP Session Troubleshooting: Quagga Shell (2) ................................................................. 510
14-41 BGP Session Troubleshooting: Quagga Shell (3) ................................................................... 511
14-42 Lesson 3: Control Plane Troubleshooting Overview ........................................................... 512
14-43 Control Plane Troubleshooting Overview................................................................................ 512
14-44 Events/Audit Trail (1) .......................................................................................................................513
14-45 Events/Audit Trail (2) ......................................................................................................................513
14-46 System Logs .......................................................................................................................................514
14-47 Cluster Issues ......................................................................................................................................515
14-48 Cloud Connector Issues (1) ............................................................................................................516
14-49 Cloud Connector Issues (2) ...........................................................................................................516
14-50 Other Control Plane Debug Logs ................................................................................................ 517

Module 15 Events and Alerts ....................................................................................... 519

15-2 Module Lessons..................................................................................................................................519
15-3 Lesson 1: Overview of Events .................................................................................................... 520
15-4 Introduction to Events (1) ............................................................................................................. 520
15-5 Introduction to Events (2) ............................................................................................................ 520
15-6 Events Flow ......................................................................................................................................... 521
15-7 Events Analytics: Object/EventID ............................................................................................. 522
15-8 Events Analytics: Timestamp ....................................................................................................... 523
15-9 Events Analytics: Searching..........................................................................................................524
15-10 Events Analytics: Use Case for Virtual Service DOWN..................................................... 525
15-11 Events Analytics: Use Case for Configuration Change ...................................................... 526
15-12 Lesson 2: Overview of Alerts ...................................................................................................... 527
15-13 About Alerts (1) ................................................................................................................................. 527
15-14 About Alerts (2) ................................................................................................................................528
15-15 Alerts View Scope ...........................................................................................................................529
15-16 Alert Notifications ............................................................................................................................ 530
15-17 Alerts: Syslog Notifications Configuration................................................................................ 531

VMware Confidential - Internal Only xxi

15-18 Alerts: Email Notifications Configuration .................................................................................. 532
15-19 Alerts: SNMP Notifications Configuration................................................................................ 533
15-20 Alerts Actions ....................................................................................................................................534
15-21 Alerts Actions: Configuration ....................................................................................................... 535
15-22 Alert Config ........................................................................................................................................536
15-23 Alert Config: Configuration (1) ..................................................................................................... 537
15-24 Alert Config: Configuration (2) ....................................................................................................538
15-25 Alert Config: Use Case ...................................................................................................................539
15-26 Lesson 3: Overview of SNMP .................................................................................................... 540
15-27 SNMP: Flow ....................................................................................................................................... 540
15-28 SNMP: Configuration v2 ..................................................................................................................541
15-29 SNMP: Configuration v3 .................................................................................................................542

Module 16 Introduction to Avi REST API............................................................... 543

16-2 Module Lessons.................................................................................................................................543
16-3 Lesson 1: Avi API Overview ........................................................................................................ 544
16-4 Avi API Overview ........................................................................................................................... 544
16-5 Avi API Objects (1) ...........................................................................................................................545
16-6 Avi API Objects (2) ..........................................................................................................................545
16-7 Avi API Object References (1) ....................................................................................................546
16-8 Avi API Object References (2) ...................................................................................................546
16-9 Avi API Tenancy and API Version .............................................................................................547
16-10 Lesson 2: Accessing Avi REST API ...........................................................................................548
16-11 Accessing Avi REST API (1) .........................................................................................................548
16-12 Accessing Avi REST API (2) ........................................................................................................548
16-13 Accessing Avi REST API (3) ........................................................................................................549
16-14 Understanding API Usage (1) ...................................................................................................... 550
16-15 Understanding API Usage (2) ....................................................................................................... 551
16-16 Understanding API Usage (3) ...................................................................................................... 552
16-17 Understanding API Usage (4) ...................................................................................................... 553
16-18 Understanding API Usage (5) ......................................................................................................554
16-19 Understanding API Usage (6) ...................................................................................................... 555
16-20 Using PATCH API (1) ......................................................................................................................556
16-21 Using PATCH API (2) .....................................................................................................................556
16-22 Using Macro API (1).......................................................................................................................... 557
16-23 Using Macro API (2)......................................................................................................................... 557
16-24 Using Macro API (3).........................................................................................................................558
xxii
VMware Confidential - Internal Only
16-25 Lesson 3: Swagger-Based API Documentation....................................................................559
16-26 Swagger-Based API Documentation (1) ..................................................................................559
16-27 Swagger-Based API Documentation (2) ................................................................................ 560
16-28 Swagger-Based API Documentation (3) ..................................................................................561
16-29 Swagger-Based API Documentation (4) .................................................................................562
16-30 Lesson 4: SDK at a Glance ............................................................................................................563
16-31 Software Development Kit ...........................................................................................................563
16-32 Ansible Modules/Roles ...................................................................................................................564
16-33 Ansible Role avi_config Example ...............................................................................................565
16-34 Terraform ............................................................................................................................................566

VMware Confidential - Internal Only xxiii

xxiv
VMware Confidential - Internal Only
Module 1
Course Introduction

1-2 Course Introduction

1-3 Importance
NSX Advanced Load Balancer course provides comprehensive training on how to install,
configure, and manage a VMware NSX Advanced Load Balancer (Avi Networks) solution. This
course covers key NSX Advanced Load Balancer (Avi Networks) features and functionality
offered in the NSX Advanced Load Balancer 18.2 release.

The features covered include the overall infrastructure, virtual services, and application
components, global server load balancing, various cloud connectors. Also covered are
application troubleshooting and solution monitoring. Access to a software-defined data center
environment is provided through hands-on labs to reinforce the skills and concepts presented in
the course.

VMware Confidential - Internal Only 1

1-4 Learner Objectives (1)
• Describe the NSX Advanced Load Balancer architecture

• Describe the NSX Advanced Load Balancer components and main functions

• Explain the NSX Advanced Load Balancer key features and benefits

• Deploy and configure NSX Advanced Load Balancer infrastructure within Private or Public
Clouds by using Write and No Access Cloud Connectors

• Explain, deploy, and configure Service Engines

• Explain and configure Local Load-Balancing constructors such as Virtual Services, Pools,
Health Monitors, and related components

• Understand and modify application behavior by using Profiles, Policies, and DataScripts

1-5 Learner Objectives (2)

• Configure advanced services such as Global Server Load Balancing

• Describe and use NSX Advanced Load Balancer REST API interfaces and related
automation capabilities

• Describe and configure NSX Advanced Load Balancer application and infrastructure
monitoring

• Gather relevant information and perform basic troubleshooting of applications by using built-
in NSX Advanced Load Balancer tooling

2
VMware Confidential - Internal Only
1-6 Course Outline
1. Course Introduction

2. Introduction to NSX Advanced Load Balancer (Avi Networks)

3. Virtual Services Configuration Concepts

4. Profiles and Policies

5. Pools Configuration Concepts

6. Modifying Application Behavior

7. NSX Advanced Load Balancer Infrastructure Architecture

8. Introduction to Cloud Connector

9. Installing, Configuring, and Managing NSX Advanced Load Balancer in No-Access Clouds

10. Installing, Configuring, and Managing NSX Advanced Load Balancer in VMware
Environments: Cloud Configuration

11. AWS Cloud Configuration

12. DNS Foundation

13. Global Server Load Balancer

14. VMware NSX Advanced Load Balancer: Troubleshooting

15. Events and Alerts

16. Introduction to Avi REST API

VMware Confidential - Internal Only 3

1-7 Typographical Conventions
The following typographical conventions are used in this course.

Conventions Use and Examples

Monospace Identifies command names, command options, parameters, code fragments,

error message, filenames, folder names, directory names, and path names:

• Run the esxtop command.

• ... found in the /var/log/message file.

Monospace Identifies user inputs:

Bold
• Enter ipconfig /release.

Boldface Identifies user interface controls:

• Click the Configuration tab.

Italic Identifies book titles:

• vSphere Virtual Machine Administration

<> Indicates placeholder variables:

• <ESXi_host_name>

• ... the Settings/<Your_Name>.txt file

4
VMware Confidential - Internal Only
1-8 Reference (1)
Title Location

NSX Advanced Load Balancer: [Link]

Architectural Overview overview/

NSX Advanced Load Balancer: Avi [Link]

Controller Sizing sizing/

NSX Advanced Load Balancer: Create [Link]

Virtual Service overview/applications/virtual-services/create-virtual-
service/

NSX Advanced Load Balancer: [Link]

Application Profile

NSX Advanced Load Balancer: Server [Link]

Pools guide/applications/pools/

1-9 Reference (2)

Title Location

NSX Advanced Load Balancer: Pool [Link]

Groups

NSX Advanced Load Balancer: Virtual [Link]

Service Policies guide/applications/vs-policies/

NSX Advanced Load Balancer: Elastic [Link]

HA for Avi Service Engines service-engines/

NSX Advanced Load Balancer: Tenants [Link]

VMware Confidential - Internal Only 5

1-10 Reference (3)
Title Location

NSX Advanced Load Balancer: Virtual [Link]

Service Scaling guide/applications/vs-scaling/

NSX Advanced Load Balancer: Flexible [Link]

Upgardes

NSX Advanced Load Balancer: Installing [Link]

Avi Vantage for VMware vCenter vantage-for-vmware-vcenter/

NSX Advanced Load Balancer: Sizing [Link]

Service Engines engines/

NSX Advanced Load Balancer: Linux [Link]

Server Cloud

1-11 Reference (4)

Title Location

NSX Advanced Load Balancer: Avi DNS [Link]

Architecture and Features architecture/

NSX Advanced Load Balancer: Avi [Link]

GSLB Overview overview/

NSX Advanced Load Balancer: [Link]

Collecting TS logs support-logs/

NSX Advanced Load Balancer: Alerts [Link]

Overview

NSX Advanced Load Balancer: API [Link]

Guide

6
VMware Confidential - Internal Only
1-12 VMware Online Resources (1)
• Questions after class:

— avi-education@[Link]

• Documentation for NSX Advanced Load Balancer:

— [Link]

— [Link]

• Ways to contact Avi Support:

— [Link]

1-13 VMware Online Resources (2)

VMware Communities: [Link]

• Start a discussion.

• Access the knowledge base.

• Access documentation, technical papers, and compatibility guides.

• Access communities.

• Access user groups.

VMware Support: [Link]

VMware Hands-on Labs: [Link]

VMware Education: [Link]

• Access course catalog and worldwide course schedule.

VMware Confidential - Internal Only 7

1-14 Environment Considerations
• Screenshots might vary:

— From release to release

— If some standard features have been disabled in your environment

• Administrative steps might vary slightly with environment. Example: VMware, AWS, and so
on

8
VMware Confidential - Internal Only
1-15 VMware Education Overview
Your instructor will introduce other Education Services offerings available to you:

• VMware Learning Paths:

— Help you find the course that you need based on the product, your role, and your level
of experience

— Can be accessed at [Link]

• VMware Learning Zone, which is the official source of digital training, includes the following
options:

— On Demand Courses: Self-paced learning that combines lecture modules with hands-on
practice labs

— VMware Lab Connect: Self-paced, technical lab environment that lets you practice skills
learned during instructor-led training

— Certification Exam Prep: Comprehensive video-based reviews of exam topics and

objectives to help you take your certification exam

• For more information, see [Link]

VMware Confidential - Internal Only 9

1-16 VMware Certification Overview
VMware certifications validate your expertise and recognize your technical knowledge and skills
with VMware technology.

VMware certification sets the standards for IT professionals who work with VMware technology.
Certifications are grouped into technology tracks. Each track offers one or more levels of
certification (up to five levels).

For the complete list of certifications and details about how to attain these certifications, see
[Link]

10
VMware Confidential - Internal Only
Module 2
Introduction to NSX Advanced Load Balancer
(Avi Networks)

2-2 Module Lessons

1. Introduction to NSX Advanced Load Balancer (Avi Networks)

2. NSX Advanced Load Balancer (Avi Networks) Architecture and Components

3. NSX Advanced Load Balancer (Avi Networks) Controller

4. NSX Advanced Load Balancer (Avi Networks) Service Engine

5. NSX Advanced Load Balancer (Avi Networks) Common Terminologies

6. NSX Advanced Load Balancer (Avi Networks) Service Elements

7. NSX Advanced Load Balancer (Avi Networks) Clouds

8. Overview of Tenants

9. NSX Advanced Load Balancer (Avi Networks) Logs

10. NSX Advanced Load Balancer (Avi Networks) Service Health

VMware Confidential - Internal Only 11

2-3 Lesson 1: Introduction to NSX
Advanced Load Balancer (Avi
Networks)

2-4 Unique Features of NSX Advanced Load

Balancer

12
VMware Confidential - Internal Only
2-5 Lesson 2: NSX Advanced Load Balancer
(Avi Networks) Architecture and
Components

2-6 Modern Distributed Architecture

Architectural Overview:

• The Avi Vantage platform is built on software-defined principles, enabling a next generation
architecture to deliver the flexibility and simplicity expected by IT and lines of business. The
Avi Vantage architecture separates the data and control planes to deliver application
services beyond load balancing, such as application analytics, predictive autoscaling, micro-
segmentation, and self-service for app owners in both on-premises or cloud environments.
The platform provides a centrally managed, dynamic pool of load balancing resources on
commodity x86 servers, VMs, or containers, to deliver granular services close to individual
applications. This allows network services to scale near infinitely without the added
complexity of managing hundreds of disparate appliances

• The Avi Controller cluster uses big data analytics to analyze the data and present actionable
insights to administrators on intuitive dashboards on the Avi Admin Console

VMware Confidential - Internal Only 13

• Avi Vantage provides built-in integrations for on-premises or cloud deployments. These
integrations with private cloud frameworks, SDN controllers, container orchestration
platforms, virtualized environments, and public clouds enable turnkey application services
and automation

• Software-based distributed application delivery controller (“ADC” >> load balancing)

— ADC duties are load balancing, HA, security

— Application acceleration, app monitoring, and analytics

— Distributed to support both physical and virtual apps

— Manual or automatic configuration and control

— Traffic stats rolled up, time-synchronized, and analyzed

• Integrations: OpenStack, OpenShift, Kubernetes, Cisco APIC, Mesos, AWS, VMware, and so
on

• Control plane tasks:

— GUI/CLI for administrative users

— Set and control all ADC functions

— Serve as ADC troubleshooting focal point

• Data plane tasks:

— Expose virtual services to clients

— Process all virtual service traffic

— Proxy and load-balance the back-end servers

• Which is more important for the business?

• Who gets priority when data plane is flooded?

• Avi Vantage separates control plane from data plane

• Administrators command the Avi Controller cluster over the control plane

• End-user clients send requests / receive responses over data plane

• Independent scaling

• No bottlenecks

• Flexible placement

14
VMware Confidential - Internal Only
2-7 NSX Advanced Load Balancer (Avi
Networks) Components
The NSX Advanced Load Balancer platform includes the following core components:

• Controller cluster

• Service Engines

• Admin Console

VMware Confidential - Internal Only 15

• The NSX Advanced Load Balancer (Avi Networks) Platform has three core components :

— NSX Advanced Load Balancer (Avi Networks) Controller cluster. NSX Advanced Load
Balancer (Avi Networks) Service Engines. NSX Advanced Load Balancer (Avi
Networks) Admin Console

• NSX Advanced Load Balancer (Avi Networks) Controller

— The NSX Advanced Load Balancer (Avi Networks) Controller is the single point of
management and control. It serves as the brain of the entire Avi Vantage system, and
for high availability, it is typically deployed as a three-node cluster. As its name implies,
the Controller implements the control plane

• NSX Advanced Load Balancer (Avi Networks) Service Engine

— NSX Advanced Load Balancer (Avi Networks) Service Engines (SEs) handle all data
plane operations within Avi Vantage by receiving and executing instructions from the
Controller. The SEs perform load balancing and all client- and server-facing network
interactions. It collects real-time application telemetry from application traffic flows

• NSX Advanced Load Balancer (Avi Networks) Admin Console

— The NSX Advanced Load Balancer (Avi Networks) admin Console is a modern web-
based user interface that provides role-based access to control, manage, and monitor
applications. Its capabilities are likewise available through the Avi CLI. All services
provided by the platform are available as REST API calls to enable IT automation,
developer self-service, and various third-party integrations

16
VMware Confidential - Internal Only
2-8 Lesson 3: NSX Advanced Load Balancer
(Avi Networks) Controller

2-9 NSX Advanced Load Balancer (Avi

Networks) Controller Details
Centralized management:

• Single point of management

• Administered by using GUI, CLI, REST API

• Configuration repository and source of truth

• Logs and metrics repository

• License enforcement

Orchestration:

• Manage life cycle of Service Engines:

— Create, Read, Update, Delete (CRUD)

— High availability or failover decisions

• Northbound API interface to automation tooling (Ansible, Terraform, and so on)

• Southbound API interface to cloud orchestrators (vCenter Server, AWS, Kubernetes, and
so on)

Centralized Management

• The Avi Controller is the single point of management and control for Service Engines,
regardless of the number of applications being load balanced or the number of SEs required

• The Controller’s REST API provides visibility into all applications (virtual services) configured.
Controller can be managed using its web interface, CLI, or REST API

• The Avi Controller stores and manages all policies related to services and management.
Config of all applications being load balanced on SE is stored on Controlled
• The health of servers, client connection statistics, and client-request logs collected by the
SEs are regularly offloaded to the Controllers, which share the work of processing the logs
and aggregating analytics

VMware Confidential - Internal Only 17

Orchestration

• For a higher degree of automation, in write-access mode deployments, Controllers work

with the underlying orchestrator to launch new SEs as needed Be the SEs automatically or
manually created (as they would be in read or no access mode deployments), it is the
Controller’s duty to place virtual services on SEs to load balance new applications or
increase the capacity of running applications

• Integrates seamlessly with API driven automation and orchestration tools such as Ansible
Tower, Terraform, and so on

• The NSX Advanced Load Balancer (Avi Networks) Controller also provides a management
center for other cloud infrastructures, with the ability to manage resources in multiple
infrastructures simultaneously. For example, the Avi Controller can be configured to
communicate with both a VMware vCenter server and an OpenStack controller, to manage
resources in each type of cloud

18
VMware Confidential - Internal Only
2-10 NSX Advanced Load Balancer (Avi
Networks) Controller Services (1)
Process Supervisor:

• Manages the coordination, startup, and shutdown of services

Configuration Database:

• Stores configuration objects, alerts, and minimal runtime information for restartability

Metrics Database:

• Stores metrics tables

Task queues and RPC:

• Critical infrastructure for restartability and intercommunication

Runtime datastore:

• Object model and for serialization or deserilization of objects

2-11 NSX Advanced Load Balancer (Avi

Networks) Controller Services (2)
SE Manager:

• Manages the life cycle and configuration of SE (VNICs, IP address, VRF, routes) and runs
Heartbeat across the SEs to detect failures.

VS Manager:

• VS, pool, and associated objects life cycle for SE scale-out, scale-in, migrate, and so on

Resource Monitor:

• Rolling upgrade of SE, autoscale of SEs

VIManager and vCenter Manager:

• Manage the interactions with vCenter Server for discovery and provisioning

Cloud Connector:

• Cloud orchestration for all clouds except vCenter Server with Azure, Openstack, AWS,
Mesos, Kubernetes, and so on

VMware Confidential - Internal Only 19

2-12 NSX Advanced Load Balancer (Avi
Networks) Controller Services (3)
Health Score Manager:

• Periodic computation of Health Score for VS, Pool, and SE

AutoScale Manager:

• Manages autoscale of back-end server

Log Indexer:

• Collects logs through rsync from SE and manages fair disk allocation across VS (ADF, UDF,
NF)

Log Orchestrator:

• Manages mapping between VS and Metrics or Log manager process on the given controller

20
VMware Confidential - Internal Only
2-13 Controller High Availability: Cluster
Controller Redundancy:

• Controller can be deployed as a standalone or redundant three-node cluster.

• A Zookeeper-like model of a three-node cluster maintains quorum.

• All controllers are active, sharding-specific workloads.

• Management can be performed from any controller in the cluster without knowledge of
which cluster is the leader.

• All infrastructure communication is encrypted.

VMware Confidential - Internal Only 21

2-14 NSX Advanced Load Balancer (Avi
Networks) Controller Sizing
Controller sizing are the following requirements:

• The amount of resources allocated to a Controller has a direct impact on its performance.

• The following system resources must be defined:

— CPUs

— Memory (RAM)

— Disk: Based on configuration requirements

CPU/Memory 8 CPUs/24 GB 16 CPUs/32 GB 24 CPUs/48 GB

Base processes 15 GB 20 GB 24 GB

Log analytics 9 GB 13 GB 24 GB

Virtual Service Scale 0 through 200 200 through 1,000 1,000 through 5,000

Service Engine Scale 0 through 100 100 through 200 200 through 250

Reservation for CPU and memory is recommended if applicable. Modifying resource settings on
VMs, such as CPU cores or RAM, requires a restart.

Refer students to sizing KB

22
VMware Confidential - Internal Only
2-15 Lesson 4: NSX Advanced Load
Balancer (Avi Networks) Service Engine

2-16 NSX Advanced Load Balancer (Avi

Networks) Service Engine Details (1)
Service Engines (SEs) handle all the data plane operations.

Data plane processing:

• Local load balancing

• Global load balancing

• Web application firewall

• Application analytics and logging

Configuration perspective:

• Configuration (including SSL cert/keys) stored in memory only

• Consumes license which is managed at the Controller

Data plane availability:

• Standalone

• Active-standby

• Active-active

2-17 NSX Advanced Load Balancer (Avi

Networks) Service Engine Details (2)
• Lightweight data plane engine

• Proxies the servers behind it

• Distributes connections, based on LB algorithms or (HTTP/S) headers

VMware Confidential - Internal Only 23

2-18 NSX Advanced Load Balancer (Avi
Networks) Service Engine Details (3)
• Executes all data plane ADC operations, not just load balancing:

— Monitors health and measures performance of servers

— Persists requests (when appropriate) to previously used servers

— Caches response content for potential reuse

— Protects against security threats, for example, DoS, suspicious client IPs

— Delivers high-performance web security with iWAF

— Offloads SSL decryption from back-end servers, re-encrypts if necessary

24
VMware Confidential - Internal Only
2-19 NSX Advanced Load Balancer (Avi
Networks) Service Engine Details (4)
• SE runs on dedicated virtual machines, containers, or bare-metal servers.

• SE specs can be customized based on requirements.

VMware Confidential - Internal Only 25

2-20 NSX Advanced Load Balancer (Avi
Networks) Service Engine Group (1)
A collection of identically configured SEs provide ADC functionality to one or more virtual
services (VS).

Templates:

• SE Groups contain sizing, scaling, placement, and HA properties

• A SE is created from the SE Group properties.

• SE Group options vary based on the cloud or ecosystem.

Service Engine groups are collection of identically configured SEs providing ADC functionality to
one or more virtual services (VS).

26
VMware Confidential - Internal Only
Service Engine group is a template with the collection of SE configuration properties:

• Service Engines are created within a group, which contains the definition of how the SEs
should be sized, placed, and made highly available

• A new SE will be created from the SE Group properties

• Each cloud will have at least one SE group. The options within an SE group may vary based
on the type of cloud within which they exist and its settings, such as no access versus write
access mode

2-21 NSX Advanced Load Balancer (Avi

Networks) Service Engine Group (2)
Features:

• An SE is always a member of the group that it was created in.

• An SE group isolates the data plane.

• A VS is placed on one or more SEs residing in exactly one group.

• Apps might gracefully migrate, scale, or fail over across SEs in the group. No cross-group
scaling, failover, or migration occurs.

• Client session data automatically replicated to other SEs in the group per VS:

— Persistence tables

— SSL session/tickets

— DataScript variables

• Multiple SE groups may exist within a cloud.

VMware Confidential - Internal Only 27

Features of SE group:

• SEs may only exist within one group. Each group acts as an isolation domain. SE resources
within an SE group may be moved around to accommodate virtual services, but SE
resources are never shared between SE groups

• SE groups provide data plane isolation. A VS is “placed” onto some number of SEs residing
in exactly one group

• Apps may gracefully migrate, scale, or failover across SEs in the group. No cross-group
scaling, failover, or migration

• Client session data automatically replicated to other SEs in the group

— Persistence tables

— SSL session/tickets

— DataScript variables

• Multiple SE groups may exist within a cloud

28
VMware Confidential - Internal Only
2-22 SE High Availability Modes

NSX Advanced Load Balancer (Avi Networks) supports following High Availability Modes

Elastic HA

Elastic HA modes combine scale-out performance as well as high availability

• Active/Active

— In active/active mode, NSX Advanced Load Balancer (Avi Networks) Vantage places
each virtual service on more than one NSX Advanced Load Balancer (Avi Networks)
SE

— Upgrades are non-disruptive. If an SE in the group fails, the service in not interrupted,
only degraded until a new service engine is spun up and the VS is placed on it. If NSX
Advanced Load Balancer (Avi Networks) Vantage's orchestrator is in write mode. A
new SE will be spun up automatically to bring SE group back to its original capacity

• N+M

— In this mode, each virtual service is typically placed on just one SE, VSs can be scaled
out to two or more service engines

— Number of SE required to place virtual service is the minimum number (The “N” in
“N+M”) of SEs required to place virtual services in the SE group and the number(The
“M”of “N+M”) of additional Buffer Service Engines

— NSX Advanced Load Balancer (Avi Networks) Vantage ensures that enough buffer
capacity exists in aggregate to handle one (M=1) SE failure

— Upgrades are non-disruptive for VSs that are scaled out to two or more service engines

• Legacy HA

— Legacy HA mode, which enables a smooth migration from legacy appliance-based load
balancers

VMware Confidential - Internal Only 29

2-23 NSX Advanced Load Balancer (Avi
Networks) Service Engine Sizing
Multiple performance vectors or features might affect the performance of Service Engine.

Example: To achieve 1 Gbps of SSL throughput and 1,000 TPS of SSL with EC certificates, two
cores are required.

1 vCPU

L4 Throughput 5 Gbps

L7 Throughput 3 Gbps

Connections/s 40k

SSL Throughput 1 Gbps

SSL TPS (RSA2K) Approximately 1,000

SSL TPS (ECC) 2,500

Reservation for CPU and memory is recommended if applicable.

30
VMware Confidential - Internal Only
2-24 Lesson 5: NSX Advanced Load Balancer
(Avi Networks) Common Terminologies

2-25 NSX Advanced Load Balancer

Terminology and System Components
• Virtual Services

• NSX ALB SEs

• Pools

• Back-end Servers

VMware Confidential - Internal Only 31

2-26 Terminology Comparison
NSX Advanced Load F5 Citrix
Balancer

Controller F5 Control Plane (BigD, Big3D, NetScaler Control Plane

MCPD)

Service Engine F5 Data Plane Netscaler Data Plane

Virtual Service Virtual Server (Virtual IP) vServers (Virtual Servers)

Pool Pool Server Group

Server Member Server

Application Profile Services Profile Profile

TCP / UDP Profile Protocol Profile Profile

Datascript iRules Policy

Connection Multiplex OneConnect Connection Multiplex

REST Api iControl NITRO API

32
VMware Confidential - Internal Only
2-27 Lesson 6: NSX Advanced Load Balancer
(Avi Networks) Service Elements

2-28 About NSX Advanced Load Balancer

(Avi Networks) Service Elements
Virtual Service:

• Defines client-facing configuration

• Listener: Virtual IP address and port

Pool:

• Defines server-facing configuration

• Persistence, load balancing algorithms

• List of servers to be load balanced

Service Engines

Backend Servers

Virtual Service:

• Virtual services are the core of the NSX Advanced Load Balancer (Avi Networks) Vantage
load-balancing and proxy functionality. Virtual Service is an application listener hosted on
Service Engine, listening on specific IP and port for client traffic. Vantage allows a virtual
service to listen to multiple service ports or network protocols

• In a normal TCP/HTTP configuration, when a client connects to the virtual service address,
Vantage will process the client connection or request against a list of settings, policies and
profiles, then send valid client traffic to a back-end server that is listed as a member of the
virtual services pool

VMware Confidential - Internal Only 33

• Virtual Service fully supports termination of client SSL- and TLS-encrypted HTTPS traffic

• Multiple profiles are assigned to Virtual Service to perform specific functionality like

— Application profile: Application profiles determine the behavior of virtual services,

based on application type. Redirects, content switching, rewriting server responses to
client requests

— TCP/UDP profile: Define the network protocol VS will use

— Persistence profile: Governs the settings that will force a client to stay connected to
the same server for a specified duration of time

— Health monitor profile: Monitoring the health of backend application server

Pool:

• Pools maintain the list of servers assigned to them and perform health monitoring, load
balancing, persistence, and functions that involve Avi-Vantage-to-server interaction

• The pool connects to the server network

• To send traffic to destination servers, the virtual service internally passes the traffic to the
pool corresponding to that virtual service. A virtual service normally uses a single pool,
though an advanced configuration using policies or DataScripts can perform content
switching across multiple pools

Avi SE:

Backend Servers:

34
VMware Confidential - Internal Only
2-29 Lesson 7: NSX Advanced Load Balancer
(Avi Networks) Clouds

2-30 About NSX Advanced Load Balancer

(Avi Networks) Clouds
Abstraction:

• A cloud is a container for Avi service elements

• SE Groups (and therefore SEs) are always scoped within a cloud

Cloud APIs:

• An API connection between the Controller and the orchestrator for that cloud

Write Access Cloud:

• Controllers manage the full life cycle of Service Engines

No Access Cloud:

• Administrators are responsible for the life cycle of Service Engines

VMware Confidential - Internal Only 35

2-31 Deployment Types

36
VMware Confidential - Internal Only
2-32 SE Image Types (1)

VMware Confidential - Internal Only 37

2-33 SE Image Types (2)
• SE images are created by the Controller.

• First time image creation for new cloud may take 8 to 15 minutes.

• SE image types are agnostic of the Controller cloud location.

• SE images contain the Controller IP and a time based, one-time use secure token.

Cloud SE Image SE Type

Bare Metal tqz Container

Kubernetes tqz Container

OpenShift tqz Container

Mesos tqz Container

VMware ova VM

OpenStack qcow2 VM

GCP qcow2 VM

Azure vhd VM

AWS ami VM

38
VMware Confidential - Internal Only
2-34 SE Image Types (3)

VMware Confidential - Internal Only 39

2-35 Lesson 8: Overview of Tenants

2-36 About Tenants

Tenants provide management isolation: Tenant A cannot see Tenant B configuration.

Tenants might span multiple clouds.

A user might have access to one or more tenants and might have a different role in each.

40
VMware Confidential - Internal Only
2-37 Lesson 9: NSX Advanced Load Balancer
(Avi Networks) Logs

2-38 About Logs (1)

Types and scoping:

• Logs are available and scoped at VS and pool layer.

• GUI shows approximately 30 out of 100+ data points that are logged.

• Significant logs:

— Network, app errors

— Apdex

— Defined within Analytics profile: Client or Server RTT beyond usual

Virtual Service Logs

• Virtual services and pools are able to log client-to-application interactions for TCP
connections and HTTP requests/responses. These logs can be indexed, viewed, and filtered
locally within the Avi Controller. Logs can be useful for troubleshooting and surfacing
insights about the end-user experience and success of the application

Significant Logs

• Avi Vantage automatically logs common network and application errors under the umbrella
of significant logs. These significant logs may also include entries for lesser issues, such as
transactions that completed successfully but took an abnormally long time
Errors may include any of the following:

— HTTP errors, such as server or Vantage-originated 4xx and 5xx errors

— Network errors, such as aborted connections, abnormal latency, or out of order

packets

— See Log Events for a list of error events that may trigger a significant Log

VMware Confidential - Internal Only 41

2-39 About Logs (2)
• User Defined Filter logs (UDF):

— Policy

— DataScript

— DataScript Troubleshooting

— VS > Analytics filters

• Non-Significant logs:

— Everything else, aka ‘good traffic’ that is not an error

42
VMware Confidential - Internal Only
2-40 NSX Advanced Load Balancer (Avi
Networks) Logs
• Raw logs are generated by the SEs:

— May be kept locally unless requested by Controller

— May be streamed to Controller (Change in behavior between 16.x, 17.1, and 17.2)

— May be streamed to 3rd party log service (ELK, Splunk, SumoLogic, and so on)

— Logs are indexed on Controller:

— Indexed logs are about 6x larger (disk) than raw logs

— Controller may index about 3k logs per second hardware dependent

— SSD much faster than disk

• Throttling is a necessity for scale:

— SE Group: Throttle logs generated per SE core, per second for all VS on the SE

— Virtual Service: Throttle logs generated per SE, per second for the VS

— External log server: Throttle logs per second sent to log server per SE

VMware Confidential - Internal Only 43

2-41 Lesson 10: NSX Advanced Load
Balancer (Avi Networks) Service Health

2-42 Health Score (1)

Performance:

• End-user experience:

— Is the site fast or slow?

• Based on apdex, which defines a conn/req as satisfied, tolerated, or frustrated

— Is the site seeing errors?

• Some apps require customization of definition of an error, for instance:

• Sharepoint sends out 403 “authentication” errors, which means the user needs
to log in first

• Websphere sends out TCP RST to close connections rather than graceful FIN

• Defined within the Application Profile

44
VMware Confidential - Internal Only
2-43 Health Score (2)
Resource Penalty:

• CPU, memory, disk issues, either on Avi or back end servers

Anomaly Penalty:

• Unexpected traffic patterns

• Hundreds of metrics monitored for automated baselining

Security Penalty:

• DDoS, WAF, or other active attack defense triggered

• Configuration warning, such as SSL using insecure ciphers

VMware Confidential - Internal Only 45

2-44 Lesson 11: Overview of the GUI

2-45 Main Elements

46
VMware Confidential - Internal Only
2-46 Table Views

VMware Confidential - Internal Only 47

2-47 Analytics and Informational Views

48
VMware Confidential - Internal Only
2-48 Configuration Editing Views

VMware Confidential - Internal Only 49

2-49 Activity: GUI Orientation
To familiarize yourself with the GUI, after you have logged in try to:

• Display an overview of the Virtual Service components

• Remove the App Domain Name from the list view of the Virtual Services

• View the current analytics values over the last 30 minutes of a Virtual Service

• View the Config Audit Trail

• Can you find:

— The Packet Capture tool?

— Licensing information?

— Where you would back up the system?

50
VMware Confidential - Internal Only
2-50 References
Load Balancing Needs a New Architecture
Building Application Services for Containers

Impact of a Controller Failure

Controller to Service-Engine Communication

Clustering Controllers From Different Networks

Protocol and Ports Used by Avi for Management Communication

Sizing Service Engines

Installing Avi Vantage in Amazon Web Services

VMware Confidential - Internal Only 51

VMware Confidential - Internal Only
Module 3
Virtual Services Configuration Concepts

3-2 Module Lessons

1. Virtual Services Configuration Concepts

2. Understanding Virtual Services

3. Virtual Service Creation

VMware Confidential - Internal Only 53

3-3 Lesson 1: Virtual Services Configuration
Concepts

3-4 About Virtual Service

A Virtual Service is the primary software construct that:

• Groups all the software components needed to load balance client traffic

• Provide reporting functionality

54
VMware Confidential - Internal Only
3-5 Components of a Virtual Service
The minimum components required to create a Virtual Service are:

• A name for the Virtual Service

• At least one Virtual IP address and listening port

• An application type or Profile

— System-HTTP or System-Secure-HTTP for example

• At least one network type or Profile

— System-TCP or System-UDP for example

• A Virtual Service is normally associated with a Pool however this is not a requirement

VMware Confidential - Internal Only 55

3-6 About a Pool
A Pool is a software construct that:

• Groups all the software components needed to understand where to direct client requests
to

• Maintains an understanding of the configured servers

56
VMware Confidential - Internal Only
3-7 Components of a Pool
The minimum components required to create a Pool are:

• A name for the Pool

• A load-balancing algorithm

— Least Connections or Round Robin for example

• A default port on which to send traffic to the servers

A Pool is normally associated with at least a:

• Single server

• Application port

However, this is not a requirement.

VMware Confidential - Internal Only 57

3-8 Default Configuration
If the Basic creation wizard is followed:

The Virtual Service default configuration will be:

• System-HTTP listening on TCP port 80

The Pool default configuration will be:

• Least Connections load-balancing algorithm

• A Passive health monitor

• Expect the real servers to be available on TCP port 80

Performing a basic creation wizard demo

58
VMware Confidential - Internal Only
3-9 Lesson 2: Understanding Virtual
Services

3-10 Listening Services (1)

The listening services are the Virtual IP address and port associated with a Virtual Service.

Why may I want to expand the client side listening services?

Some examples:

• Internal and external client facing Virtual IP addresses

• A Virtual Service listening on both HTTP and HTTPS ports

3-11 Listening Services (2)

Two distinct software components define the listening services:

• The Virtual IP address

• The Service Port

VMware Confidential - Internal Only 59

3-12 Application Types
Why may I want to change the application type?

An example:

• Enable HTTPS instead of HTTP

Profiles can be used as configuration elements

Multiple predefined profiles configure the application type.

Some example profiles are:

• System-HTTP

• System-Secure-HTTP

SSL Off loading

Refer back to previous section around listeners

60
VMware Confidential - Internal Only
3-13 Virtual Service Types (1)
The following types of virtual services are available:

• HTTP:

— Load-balancing unencrypted web traffic on client side

— Inspect traffic at the application layer

• HTTPS:

— Load-balancing encrypted web traffic on client side

— Relieves web servers of the processing burden of encrypting and decrypting traffic

— Inspect traffic at the application layer

[Link]
services/create-virtual-service

VMware Confidential - Internal Only 61

3-14 Virtual Service Types (2)
• L4 VS:

— Inspecting and load-balancing traffic based on Layer 4 configuration

— Encrypted traffic ends on back-end server rather than on SE

• L4 SSL/TLS:

— Decrypting on SE and forwarding clear-text traffic to the server

— Not covered in detail within this module

[Link]
services/create-virtual-service

62
VMware Confidential - Internal Only
3-15 Application Types
A common configuration looks like:

• Listen on both HTTP and HTTPS ports

• This can be implemented as two distinct Virtual Services, one for each application type

VMware Confidential - Internal Only 63

3-16 Back-End Communications: Introduction
to Pools
As previously mentioned:

• Grouping of servers hosting the same application

• Identified by their individual IP address/FQDN and port number

• Load-balanced by a specified algorithm and persistence mechanism

• All pool members are monitored by each configured Health Monitor

64
VMware Confidential - Internal Only
3-17 Back-End Communications: Load-
Balancing Algorithms
Determines the method and prioritization for distributing connections

Multiple algorithms are available:

• Least Connections (Default)

• Round Robin

• Least Load

• Fewest Servers

• Consistent Hash

• Fastest Response

• Core Affinity

Talk about two or three only but make sure to talk about Passive monitor

VMware Confidential - Internal Only 65

3-18 Back-End Communications: Session
Persistence
Ensures that subsequent connections from the same client will be load balanced to the same
server

Critical for servers that maintain client session information locally

The following built-in persistent templates are available:

• Client-IP

• HTTP-Cookie

• App-Cookie

• Custom HTTP Header

• TLS

Does the choice of persistence affect the load-balancing algorithm choice?

Again, just talk about two or three put make sure to discuss how choices have repercussions.

66
VMware Confidential - Internal Only
3-19 Handling Failure: Health Monitors
Validates the health of the servers to make forwarding decisions

• Health Monitors are attached to Pools

• Supports both Active and Passive monitoring methods

• Inactive when the pool is not utilized by a Virtual Service

• Each SE makes individual health decisions, therefore each SE will need to probe the back-
end servers

VMware Confidential - Internal Only 67

3-20 Handling Failure: Passive Health Monitor
Enabled by default on L7 Virtual Services

• Monitors end-user interaction with the servers

• Assumes server in healthy state when detecting valid responses such as 2XX code

• Assumes server with error when detecting TCP Reset or 5XX code

• Will not mark a server down when a back-end server fails, but will reduce request to the
server

• Best practice is to enable both a passive and an active health monitors to each pool

68
VMware Confidential - Internal Only
3-21 Handling Failure: Active Health Monitors
Proactively send queries to servers, synthetically mimicking a client

• Numerous configurable Active Health Monitor types and built in templates

• Timeout intervals are tunable

• Each SE probes for the servers for which it is responsible

• Multiple Active Health Monitors can be assigned to a Pool in addition to the Passive Health
Monitor

— However, this will consume more CPU

• All active health monitors must be successful for the server to be marked up (logical AND)

VMware Confidential - Internal Only 69

3-22 Lesson 2: Virtual Service Creation

3-23 Virtual Service Dashboard

The Dashboard is the default landing page of Avi Vantage.

The Dashboard displays existing Virtual Services in the List or Tree view.

The List view shows the Virtual Services and Health Score.

The Tree view displays the Virtual Services, associated Service Engines, Server Pool, and Back-
end Servers.

70
VMware Confidential - Internal Only
3-24 Virtual Services Tab
• Detailed view of all the Virtual Services and its configured parameters and statistics

• Metric value computation and time ranges can be adjusted

• Columns can be added or removed

• Delete/Enable/Disable/Edit an existing Virtual Service

VMware Confidential - Internal Only 71

3-25 Virtual Service Creation
• A new virtual service may be created by using either the basic or advanced mode

• In basic mode, many features are not exposed during the initial setup

• After the virtual service has been created by using the basic mode, the options shown while
editing are the same as advanced mode, regardless which mode was initially used

• While basic mode may have been used to create the virtual service, it does not preclude
access to any advanced features

72
VMware Confidential - Internal Only
3-26 Virtual Service Creation: Basic Mode (1)
• Basic Mode:

— This mode is strongly recommended for most normal use cases

— It requires minimal user input and relies on pre-defined configurations for the virtual
service that should be applicable for most applications

— Creating a virtual service in basic mode can be accomplished within a single pop-up
window

— While basic mode might have been used to create the virtual service, it does not
preclude access to any advanced features

[Link]
services/create-virtual-service/

VMware Confidential - Internal Only 73

3-27 Virtual Service Creation: Basic Mode (2)

Name: Provide a unique name for new Application Type: Select from the

Service: Display default for the

VIP Address: Enter either the DNS

Add Servers: 1. Add by Server IP

Imply instructor to perform demo.

[Link]
services/create-virtual-service/

Add Servers:

• Add by Server IP address:

— Enter the IP address for the server to be included in the Address field, and then click
the Add Server button, a range of IP addresses may be entered using a dash character,
such as [Link]–[Link]

• Add by DNS Resolvable Server Name:

— Enter the name of the server to be included in the Address field. If the server name
successfully resolves, the IP address will be shown and the Add Server button will
change to green

74
VMware Confidential - Internal Only
Service:

• Accept the displayed default for the application type or key in a service port to listen for
connections

• Basic mode supports configuring only one port. For multiple service ports or ranges, edit the
virtual service after creation

VMware Confidential - Internal Only 75

VMware Confidential - Internal Only
Module 4
Profiles and Policies

4-2 Module Lessons

1. Overview of Profiles

2. Configuring and Using Profiles

3. TCP/UDP Profiles

4. Application Profiles

5. SSL Profile and Certificate Overview

6. SSL Re-Encryption

7. Acceleration Technologies

8. HTTP Compression

9. HTTP Caching

10. Connection Multiplexing

11. Policies Overview

VMware Confidential - Internal Only 77

4-3 Lesson 1: Introduction to Profiles (1)

4-4 Introduction to Profiles (2)

• Collections of settings into reusable objects

• Tenant-aware

• Profiles defined within admin tenant can be reused across all other tenants, providing
globally common configuration

• Profiles may only be deleted if it is not currently assigned to an object (such as Virtual
Service)

• The default system templates can be edited, but not deleted

78
VMware Confidential - Internal Only
4-5 Advanced Virtual Service Creation
• Advanced Mode:

— This mode requires additional user input and is recommended when requiring access to
less common features, such as policy rules or customized analytics settings

— This mode may also be used to configure a virtual service for multiple ports or network
protocols

— The Create Virtual Service pop-up and Edit Virtual Service pop-up share advanced
mode interface, which has the following tabs: Settings, Policies, Analytics and Advanced

[Link]
services/create-virtual-service/

VMware Confidential - Internal Only 79

4-6 Advanced Virtual Service Creation:
Settings (1)
• Enabled:

— The toggle icon enables (green) and disables (red) the virtual service. When disabled
(red icon), the virtual service will not accept any new connections, existing concurrent
connections will be terminated, and the virtual service will be not associated from all
SEs. No health monitoring is performed for disabled virtual services

• VIP Address:

— Enter either the DNS resolvable name or an IP address for the virtual service, if the
name cannot be resolved, it will appear in red

• Application Type:

— Select from the common app types

[Link]
services/create-virtual-service/

When adding both HTTP and HTTPS on the same virtual service, you may also configure the
virtual service to perform HTTP to HTTPS redirection.

This can be done in several ways, such as using a secure-HTTP application profile, which enables
the SSL Everywhere feature’s automatic redirects.

Alternatively, redirects may be configured by using an HTTP request policy.

80
VMware Confidential - Internal Only
4-7 Advanced Virtual Service Creation:
Settings (2)
• Services:

— Services Basic

• Enter a single service port number, such as 80 for HTTP, multiple service ports
may be added by clicking the green plus icon

— Service Advanced

• In advanced mode, services may be added individually, or as a range, such as 5000

to 10000. This can be useful for older apps such as CORBA, DNS, RADIUS, or
Syslog, the virtual service default TCP/UDP profile may be overridden on a per-
service port basis

[Link]
services/create-virtual-service/

When adding both HTTP and HTTPS on the same virtual service, you may also configure the
virtual service to perform HTTP to HTTPS redirection.

This can be done in several ways, such as using a secure-HTTP application profile, which enables
the SSL Everywhere feature’s automatic redirects.

Alternatively, redirects may be configured by using an HTTP request policy.

VMware Confidential - Internal Only 81

4-8 Introduction to Profiles
• Profiles are used to group collections of settings into a single, reusable object

• A profile may be used by multiple objects such as Virtual Services referencing the same
TCP Profile

• Profile groups:

— Application Profile

— TCP/UDP Profile

— Persistence Profile

— Health Monitor Profile

— Analytics Profile
— IPAM/DNS Profiles

— Traffic Clone Profiles

82
VMware Confidential - Internal Only
4-9 Introduction to Profiles: Application and
Network Profiles
Protocol Profile Types and Settings

• Common application profile types:

— DNS

— L4

— Syslog

— HTTP

• Main TCP/UDP Profile types are:

— TCP Proxy

— TCP Fast Path

— UDP Fast Path

VMware Confidential - Internal Only 83

• Type: Type of application profile, which will be either:

— DNS: Default for processing DNS traffic

— HTTP: Default for processing Layer 7 HTTP traffic

— L4: Catch-all for any virtual service that is not using an application-specific profile

— Syslog: Default for processing Syslog traffic

• Type of TCP/UDP profile, which can be one of the following:

— TCP Proxy: This profile terminates client connections to the Virtual Service and then
open a new TCP connection to the destination server. Each connection will negotiate
the optimal TCP settings for the connecting device. For example, a client may connect
with a 1400-byte MTU while the server can still send data to Avi Vantage in 1500-byte
MTUs. In this case, Avi Vantage will buffer the server’s responses and send them back
to the client separately. If the client connection drops a packet, then Avi Vantage will
handle retransmission, as the server may have already finished the transmission and
moved on to handling the next client request

— TCP Fast Path: Upon receiving a TCP SYN from the client, Avi Vantage will make a
load-balancing decision and forward the SYN and all subsequent packets directly the
server. The client’s source IP address will still be translated to the Service Engine’s IP
address for the server network to ensure return path routing. The client to server
communication occurs over a single TCP connection, using the parameters negotiated
between client and server

— UDP Fast Path: UDP is connection-less, meaning that packets are directly forwarded to
the load balanced server. The load-balancing decision is made on the first packet from
the client, and the source IP address is still changed to the Service Engine’s IP address

— Auto Learn: While in the default Auto Learn mode, the TCP/UDP profile will
dynamically adjust settings based on the application type assigned to the Virtual
Service. Disabling Auto Learn uses the parameters statically defined within the profile

84
VMware Confidential - Internal Only
4-10 Introduction to Profiles: Persistence
Profiles
• A persistence profile governs the settings that will force a client to stay connected to the
same server for a specified duration of time

• Persistence is an optional profile that is attached to a pool

• The default system profiles can be edited, but not deleted

Type: The available persistence types are:

• App Cookie: Rather than have Avi insert a new cookie for persistence, Avi will use an
existing cookie that has been inserted by the server. If the cookie does not exist, Avi will
look for a URI query of the same name and will persist on that value. Typically this
persistence will be performed on a ASP or Java session ID. * Client IP Address: Avi
Vantage will record the client’s source IP address in a table during the Persistence Timeout
for this profile. While the IP remains in the table, any new connection by the user will be sent
to the same server. The Client IP Address persistence table is stored in memory on the
Service Engine, and is automatically mirrored to the Controller and all other Service Engines
that support this virtual service.

— Persistence Timeout: Enter the number of minutes to preserve the client’s IP address in
the Persistence Timeout field. Entering 0 disables persistence and allows new
connections to be load balanced to a new server immediately. The timeout begins
counting down when all connections from the same source IP address to the virtual
service are closed. This field is applicable to Client IP persistence only

• Custom HTTP Header: This method allows an HTTP header to be specified for persistence.
The Service Engine will inspect the value of the defined header, and will match the value
against a statically assigned header field for each server in the pool. If there is a match, the
client will be persisted. The server’s header field is configured in the Pool’s edit server page,
where new servers are added

VMware Confidential - Internal Only 85

 — Header Name: Specify the HTTP header whose value will be used for the persistence
lookup.

• GSLB Site: This permits a given client of a global application to persist to the first site to
which it is directed. Refer to GSLB Site Cookie Persistence for more information

• HTTP Cookie: Applicable to virtual services with an attached HTTP application profile. Avi
Vantage inserts a cookie into outgoing HTTP responses and reads the cookie on incoming
requests. The cookie is session-based, meaning that the cookie persistence remains valid as
long as the client keeps their browser open. Closing the browser removes the cookie stored
by the client browser, thereby tearing down both the connections and the persistence.
Cookies uniquely identify each client, which is useful if multiple users are accessing the virtual
service from the same IP address. Clients store the persistence information, so it does not
consume memory on the Service Engine. The cookie is transparent to clients and will be
named AVI_COOKIE by default

— HTTP Cookie Name: By default, the cookie name is AVI_COOKIE unless this field is
populated with an alternate name

• TLS: Applicable to virtual services that are terminating SSL or TLS. This method embeds
user persistence information within the ticket field of a TLS session. Clients negotiating with
older SSL v3 will use a variation that inserts the persistence information into the SSL Session
ID. Avi Vantage does not allow clients to renegotiate the session, which is more secure and
also ensures that Avi Vantage can maintain the persistence as it controls if and when the
Session ID is renegotiated and recreated

• Select New Server When Persistent Server Down: Determine how this profile will handle a
condition when the Health Monitor marks a server as down while Avi Vantage is still
persisting clients to it. Immediate: Avi Vantage will immediately select a new server to
replace the one that has gone down and switch the persistence entry to the new server

• Never: No replacement server will be selected. Persistent entries will be required to expire
normally based on the persistence type

86
VMware Confidential - Internal Only
4-11 Introduction to Profiles: Health Monitor
Profile
• Only active health monitors may be defined

• The passive monitor has no settings

• Type of Health Monitor, which can be one of the following:

— DNS: Validate the health of responses from DNS servers

— External: Use a custom script to validate the health of a diverse array of applications

— HTTP: Validate the health of HTTP web servers

— HTTPS: Validate the health of HTTPS web servers when the connection between Avi
Vantage and the server is SSL/TLS encrypted

— Ping: An ICMP ping can be used to monitor any server that responds to pings. This is a
lightweight monitor, but it does not validate application health

— TCP: Validate TCP applications via simple TCP request/response data

— UDP: Validate UDP applications via simple UDP request/response data

• Send Interval: Frequency at which the Health Monitor initiates a server check, in seconds

• Receive Timeout: Maximum amount of time before the server must return a valid response
to the Health Monitor, in seconds

• Successful Checks: Number of consecutive health checks that must succeed before Avi
Vantage marks a down server as being back up
• Failed Checks: Number of consecutive health checks that must fail before Avi Vantage
marks an up server as being down

VMware Confidential - Internal Only 87

4-12 Introduction to Profiles: Analytics Profile
• Analytics Profile defines the analytics configuration throughout the system to determine the
health of applications based on expectations of what a typical user experience should be

88
VMware Confidential - Internal Only
4-13 Lesson 2: Configuring and Using
Profiles

4-14 Configuring and Using Profiles (1)

• Application Profiles tab includes the following functions:

— Search: Search against the name of the profile

— Create: Opens the Create Application Profile pop-up

— Edit: Opens the Edit Application Profile pop-up

— Delete: Removes an application profile if

VMware Confidential - Internal Only 89

4-15 Configuring and Using Profiles (2)
• To Configure or Edit Application Profile the initial settings are similar regardless of the type
of profile chosen:

— Name: Enter a unique name for the profile

— Description: Enter an optional description for the profile

— Type: Click the appropriate type button to select the application for this profile.
Select L4 for none

90
VMware Confidential - Internal Only
4-16 Configuring and Using Profiles:
Application Profile
Application Profile Creation

• Create application profile under Template -> Profiles -> Application

• Applying L7 Application Profile allows the Service Engine to decode application protocols-

— L4

— L4 SSL/TLS

— DNS

— Syslog

— HTTP/HTTPS

— SIP

VMware Confidential - Internal Only 91

4-17 Configuring and Using Profiles: TCP/UDP
(1)
• TCP/UDP Profiles tab includes the following functions:

— Search: Search across the list of objects.

— Create: Opens the New TCP/UDP Profile pop-up.

— Edit: Opens the Edit TCP/UDP Profile pop-up.

— Delete: A TCP/UDP profile may only be deleted if it is not currently assigned to a

Virtual Service.

92
VMware Confidential - Internal Only
4-18 Configuring and Using Profiles: TCP/UDP
(2)
• The default system Profiles cannot be deleted

• To Configure or Edit TCP/UDP Profile the initial settings are similar regardless of the type of
profile chosen:

— Name: Enter a unique name for the profile

— Description: Enter an optional description for the profile

— Type: Select the type of TCP/UDP Profile:

VMware Confidential - Internal Only 93

4-19 Configuring and Using Profiles:
Persistence Profile (1)
• New Persistence Profiles tab includes the following functions:

— Name: Enter a unique name for the Persistence Profile in the Name field

— Select New Server When Persistent Server Down

• Immediate

• Never

• Select New Server When Persistent Server Down

• Immediate: Avi Vantage will immediately select a new server to replace the one that has
gone down and switch the persistence entry to the new server

• Never: No replacement server will be selected. Persistent entries will be required to expire
normally based on the persistence type

94
VMware Confidential - Internal Only
4-20 Configuring and Using Profiles:
Persistence Profile (2)
• Type: The types of persistence can be one of the following:

• App Cookie

• Client IP Address

• Custom HTTP Header

• GSLB Site

• HTTP Cookie

• TLS

VMware Confidential - Internal Only 95

4-21 Lesson 3: TCP/UDP Profiles

4-22 L4: Proxy

• LB decision is made upon completion of the 3 way handshake

• Client performs TCP handshake with the SE

• SE to server TCP connection is independent of the client connection or TCP parameters.

For instance, an MSS on the client side may be 1400 bytes, but the server may be 1500
bytes. The SE provides buffering between the two connections.

• Maintaining two connections is more work for the SE (CPU and memory), but it results in
finer control over the connection for clients and servers

Voice Track:

TCP Proxy - Auto Learn (default): While in the default Auto Learn mode, the TCP/UDP profile
will dynamically adjust settings based on the application type assigned to the Virtual Service.
Disabling Auto Learn uses the parameters statically defined within the profile. Use “custom” for
testing so the parameter can be adjusted.

96
VMware Confidential - Internal Only
4-23 TCP/UDP Profile Creation (1)
• To create TCP/UDP Profile under Template - Select Templates > Profiles > TCP/UDP

• Then Select a Profile type:

— TCP Proxy

— TCP Fast Path

— UDP Fast

— UDP Proxy

Voice Track:

TCP Proxy - Auto Learn (default): While in the default Auto Learn mode, the TCP/UDP profile
will dynamically adjust settings based on the application type assigned to the Virtual Service.
Disabling Auto Learn uses the parameters statically defined within the profile. Use “custom” for
testing so the parameter can be adjusted.

VMware Confidential - Internal Only 97

4-24 Network Profile: TCP Proxy Versus TCP
Fast Path Flow (1)

98
VMware Confidential - Internal Only
4-25 Network Profile: TCP Proxy Versus TCP
Fast Path Flow (2)

VMware Confidential - Internal Only 99

4-26 Network Profile: L4 UDP Fast and UDP
Per Packet
UDP Fast Path:

— Flow-based packet processing

— Multiple packets on a 5-tuple are exchanged on a flow

— Flow entries are created for client and server-side flow

— Flow cleanup happens after configurable idle timeout

— Receiving a packet resets the idle timer for the flow

— Data script execution for all packets in both directions

UDP Per Packet:

— A packet from client side is a new flow

— Flow entry is created only on server side

— Receiving a packet from server will clean up the connection

— Data script execution for all packets in both directions

— Commonly used for DNS type of applications

UDP Per Packet is the common UDP implementation

100
VMware Confidential - Internal Only
4-27 TCP/UDP Profile Creation (2)

• TCP Proxy Settings can be modified by selecting Custom tab

• In the default Auto Learn mode, the TCP/UDP profile will dynamically adjust settings based
on the application type assigned to the Virtual Service

4-28 TCP/UDP Profile Creation (3)

• Select TCP Fast Path Tab to modify the TCP Fast Path Settings

• Select UDP Fast Path Tab to modify UDP Fast Path Settings

VMware Confidential - Internal Only 101

• Time has elapsed and the packet is sent half-full

• TCP Fast Path

• A TCP Fast Path profile does not proxy TCP connections. Rather, it directly connects
clients to the destination server and translates the client’s destination Virtual Service
address with the chosen destination server’s IP address. The client’s source IP address is still
translated to the Service Engine address to ensure that server response traffic returns
symmetrically

• This profile type has the following settings:

• Enable SYN Protection: When disabled, Avi Vantage performs load balancing based on the
initial client SYN packet. The SYN is forwarded on to the server and Avi Vantage merely
forwards the packets between client and server, which leaves servers vulnerable to SYN
flood attacks from spoofed IP addresses. When enabled, Avi Vantage will proxy the initial
TCP three-way handshake with the client to validate the client is not a spoofed source IP
address. After the three-way handshake has been established, Avi Vantage will replay the
handshake on the server side. After the client and server are connected, Avi Vantage will
drop back into pass through (fast path) mode. This process is sometimes referred to as
delayed binding. > Note: Consider using TCP Proxy mode for maximum TCP security

• Session Idle Timeout: Idle flows will terminate (time out) after the specified time period. Avi
Vantage will issue a TCP reset to both the client and the server

• UDP Fast Path

• The UDP Fast Path profile allows a Virtual Service to support UDP. Avi Vantage will
translate the client’s destination Virtual Service address to the destination server and rewrite
the client’s source IP address to the Service Engine’s address when forwarding the packet
to the server. This ensures that server response traffic traverses symmetrically through the
original SE

• This profile type uses the following settings:

• NAT Client IP Address: By default, Avi Vantage will translate the client’s source IP address
to an IP address of the Avi Service Engine. This may be disabled for connectionless
protocols which do not require server response traffic to traverse back through the same
Service Engine. For example, a syslog server will silently accept packets without responding.
Therefore, there is no need to ensure response packets route through the same SE. When
SNAT is disabled, it is recommended to ensure that the Session Idle Timeout is kept to a
lower value

102
VMware Confidential - Internal Only
• Per Packet Load Balancing: By default, Avi Vantage treats a stream of UDP packets from
the same client IP: Port as a session, making a single load-balancing decision and sending
subsequent packets to the same destination server. For some application protocols, each
packet should be treated as a separate session that can be uniquely load balanced to a
different server. DNS is one example where enabling Per Packet Load Balancing causes Avi
Vantage to treat each packet as an individual session or request

• Session Idle Timeout: Idle UDP flows will terminate (time out) after the specified time
period. Subsequent UDP packets could be load balanced to a new server unless a
persistence profile is applied

VMware Confidential - Internal Only 103

4-29 Advanced Topics: Server Network
Profile
• By default, a new virtual service will use the same TCP/UDP profile specified in the Settings
tab for both the client and the server side of a TCP proxied connection

• Can be override this setting for the connection between the Service Engine and the server
by specifying a different TCP proxy profile

• While the same TCP stack should negotiate correctly and independently with both clients
and servers, it may be desirable to have a TCP stack tuned to servers, such as disabling
Nagle’s algorithm on the server side but enabling it on the client side

• This option only applies to virtual services set for TCP proxy

[Link]
services/create-virtual-service/

4-30 TCP/UDP Profile Creation

TCP/UDP Profile Use Case

• Application needs a custom TCP window size for testing long RTT

104
VMware Confidential - Internal Only
4-31 Lesson 4: Application Profiles

4-32 L4 and L4 SSL/TLS Application Profiles

• The L4 Profile is used for any virtual service that does not require application-layer proxying

• L4 SSL/TLS for any virtual service that is SSL-encrypted and not using an application-
specific profile

Voice Track:

TCP Proxy - Auto Learn (default): While in the default Auto Learn mode, the TCP/UDP profile
will dynamically adjust settings based on the application type assigned to the Virtual Service.
Disabling Auto Learn uses the parameters statically defined within the profile. Use “custom” for
testing so the parameter can be adjusted.

VMware Confidential - Internal Only 105

4-33 DNS Application Profile
A DNS application profile specifies settings dictating Avi’s request-response handling

• Number of IPs returned by DNS server

• Process EDNS Extensions / Subnet prefix length

• TTL / Negative TTL

• (Options for) Invalid DNS Query processing

• Rate Limit Connections from a Client

• Preserve Client IP Address

• Valid subdomains

• Authoritative Domain Names

Voice Track:

TCP Proxy - Auto Learn (default): While in the default Auto Learn mode, the TCP/UDP profile
will dynamically adjust settings based on the application type assigned to the Virtual Service.
Disabling Auto Learn uses the parameters statically defined within the profile. Use “custom” for
testing so the parameter can be adjusted.

106
VMware Confidential - Internal Only
4-34 Introduction to HTTP Application Profile
• Enable Reverse HTTP Proxy functionality

• Exposes HTTP specific features for Virtual Services such as redirects, content switching,
and rewriting server responses

VMware Confidential - Internal Only 107

4-35 Introduction to HTTP Profile: General Tab
• Contains the basic HTTP settings:

— Connection Multiplex

— X-Forwarded-For (Header)

— Preserve Client IP Address (Source IP)

— WebSockets Proxy

108
VMware Confidential - Internal Only
4-36 Introduction to HTTP Profile: Security
Tab
• Controls the security settings for HTTP applications that are associated with the profile:

— SSL Everywhere

— HTTP-to-HTTPS Redirect

— Rewrite Server Redirects to HTTPS

— Secure Cookies

— HTTP Strict Transport Security

— HTTP-only Cookies

— X-Forwarded-Proto

— Client SSL Certificate Validation

• “SSL Everywhere” is the difference between the pre-defined HTTP and HTTPS profiles

Verify with application owner to ensure the backend servers support these security settings

VMware Confidential - Internal Only 109

4-37 Introduction to HTTP Profile:
Compression
• Enables HTTP Gzip compression for responses from the Service Engine to the client:

— Compression Mode

— Remove Accept Encoding Header

— Compressible Content Types

110
VMware Confidential - Internal Only
4-38 Introduction to HTTP Profile: Caching
• Caching HTTP content locally to reduce workloads for both servers and Service Engine:

— X-Cache

— Age Header

— Date Header

— Aggressive

— Cache Expire Time

— Heuristic Expire

— Cacheable Object Size

— Cache URI with Query Arguments

— Cacheable and Non-Cacheable MIME Types

VMware Confidential - Internal Only 111

4-39 Introduction to HTTP Profile: DDoS
• DDoS mitigation controls for HTTP and the underlying protocols

— HTTP Limit

— Rate Limit HTTP and TCP

112
VMware Confidential - Internal Only
4-40 HTTP Profile Use Cases
HTTP Profile Use Case

• HTTP to HTTPS redirect

• Application team needs a custom header of up to 16 KB size and default application profile is
reporting intermittent errors (Need to create custom application profile with larger header
size)

VMware Confidential - Internal Only 113

4-41 Lesson 5: SSL Profile and Certificate
Overview

4-42 Introduction to SSL Profile and

Certificate
• Supports the ability to terminate SSL connections

• SSL Profile defines the accepted SSL versions and ciphers

• SSL Certificate contains the cert and key to enable secure communications

• Termination of SSL connections requires both an SSL profile and an SSL certificate to be
associated with the Virtual Service

• Re-encryption between Service Engines and back-end servers requires an SSL profile to be
associated with the Pool

• The default System-SSL Profile is automatically assigned to a Virtual Service created by

using the basic mode

114
VMware Confidential - Internal Only
4-43 SSL Profile Overview (1)
• Create Profile under Template -> Security -> SSL/TLS Profile

• SSL Profile contains supported SSL/TLS ciphers and versions

• It will prioritize ECC over RSA by default because of faster performance with handshake
negotiation, if both ciphers are supported by the client

— ECC consumes less CPU than RSA on average

VMware Confidential - Internal Only 115

4-44 SSL Profile Overview (2)
• Default SSL Profile is optimized for security rather than prioritizing the fastest ciphers

• Increasing accepted ciphers and SSL versions also increases the compatibility with clients
and potentially lowering security

• Provides Security, Performance, and Compatibility rankings based on selections

• Configurable options include:

— Versions

— Enable SSL Session Reuse

— Ciphers

— SSL Session Timeout

— Send “close notify” alert

— Prefer client cipher ordering

116
VMware Confidential - Internal Only
4-45 SSL Certificate Overview (1)
• Create certificate under Template -> Security -> SSL/TLS Certificates

• Comes with four self-signed certificates for Applications and the Controller portal

• SSL certificates cannot be shared across tenants by default, however the behavior can be
changed

VMware Confidential - Internal Only 117

4-46 SSL Certificate Overview (2)
• Provides the interface to create/import a certificate

— Import a cert/key pair, generate a CSR or a self-signed certificate

— Supports PEM and PKCS #12 formatted certificates

4-47 SSL Certificate Use Cases

SSL Certificate Use Cases

• Security team has requested TLS1.1 to be disabled for all their existing applications

118
VMware Confidential - Internal Only
4-48 Lesson 6: SSL Re-Encryption

4-49 SSL Re-Encryption Overview

• To encrypt the traffic between Service Engines and back-end servers

• Server-side encryption has no dependency on client-side encryption

VMware Confidential - Internal Only 119

4-50 Configuring SSL Re-Encryption
• Available under Pool creation’s setting

• Mandatory to provide an SSL Profile to determine supported ciphers and versions

• Optional configuration parameters:

— Server SSL Certificate Validation PKI Profile

— Service Engine Client Certificate

— Common Name Check

— TLS SNI

— TLS SNI Server Name

— Rewrite Host Header to SNI Name

120
VMware Confidential - Internal Only
4-51 SSL Re-Encryption Use Case
SSL Re-Encryption Use Case

• Organizational Security policy requires connections between pool servers and Avi Service
Engines to be encrypted

VMware Confidential - Internal Only 121

4-52 Lesson 7: Acceleration Technologies

4-53 Introduction Acceleration Technologies

• Acceleration technologies reduce the access time by using following technologies:

— HTTP Compression

— HTTP Caching

— Connection Multiplexing

122
VMware Confidential - Internal Only
4-54 Lesson 8: HTTP Compression

4-55 HTTP Compression Tab

• Check the Enable Compression check box to enable HTTP Gzip compression, then enter
the appropriate compression settings

• Enables HTTP Gzip compression for responses from the Service Engine to the client:

— Compression Mode

— Remove Accept Encoding Header

— Compressible Content Types

VMware Confidential - Internal Only 123

4-56 HTTP Compression Mode and Options
• Auto mode enables Vantage to determine the optimal settings

• Custom mode allows creation of custom filters that provide more granular control over who
should receive what level of compression

• There are six default Compressible Content Types

• Compressible Content Types determine which HTTP Content-Types are eligible to be

compressed. This field points to a String Group which contains the compressible type list

• Remove Accept Encoding Header removes the Accept Encoding header, which is sent by
HTTP 1.1 clients

Auto is recommended, to dynamically tune the settings based on clients and available Service
Engine CPU resources.

124
VMware Confidential - Internal Only
4-57 HTTP Compression: Custom Setting
• Default setting for “System-Re writable-Content-Type”

• Default setting for “System-Compressible-Content-Types”

• Default setting for “System-Devices-Mobile”

VMware Confidential - Internal Only 125

4-58 HTTP Compression Filter and Options
• Click “Compression Filter” to create a custom filter and provide desired value to required
fields:

— Filter Name:

— Matching Rules (Client IP Address and User Agent)

— Compression Level

• To create a custom compression filter:

• Click Add New Filter to create a custom filter

• Enter the following:

— Filter Name: Provide a unique name for the filter (optional)

— Matching Rules determine if the client (through Client IP or User Agent string) is eligible
to be compressed by using the associated Action. If both Client IP and User Agent rules
are populated, then both must be true for the compression action to fire

• Client IP Address allows you to use an IP Group to specify eligible client IP

addresses. For example, an IP Group called Intranet that contains a list of all internal
IP address ranges. Clearing the Is In button reverses this logic, meaning that any
client that is not coming from an internal IP network will match the filter

• User Agent matches the client's User Agent string against an eligible list contained
within a String Group. The User Agent is a header presented by clients indicating

126
VMware Confidential - Internal Only
 the type of browser or device they may be using. The System-Devices-Mobile
Group contains a list of HTTP User Agent strings for common mobile browsers

• The Action section determines what will happen to clients or requests that meet the Match
criteria, specifically the level of HTTP compression that will be used

— Aggressive compression uses Gzip level 6, which will compress text content by about
80% while requiring more CPU resources from both Avi Vantage and the client

— Normal compression uses Gzip level 1, which will compress text content by about 75%,
which provides a good mix between compression ratio and the CPU resources
consumed by both Avi Vantage and the client

— No Compression disables compression. For clients coming from very fast, high
bandwidth and low latency connections, such as within the same data center,
compression may actually slow down the transmission time and consume unnecessary
CPU resources

VMware Confidential - Internal Only 127

4-59 Lesson 9: HTTP Caching

4-60 HTTP Caching (1)

• It can cache HTTP content, thereby enabling faster page load times for clients and reduced
workloads for both servers and Avi Vantage

• Cache Configuration Options are:

— X-Cache

— Age Header

— Date Header

— Aggressive

— Cache Expire Time

— Heuristic Expire

— Cacheable Object Size

— Cache URI with Query Arguments

— Cacheable and Non-Cacheable MIME Types

The following parameters all are optional:

• X-Cache: Vantage will add an HTTP header labeled X-Cache for any response sent to the
client that was served from the cache. This header is informational only, and will indicate that
the object was served from an intermediary cache

• Age Header: Vantage will add a header to the content served from cache that indicates to
the client the number of seconds that the object has been in an intermediate cache. For
example, if the originating server declared that the object should expire after 10 minutes and
it has been in the Vantage cache for 5 minutes, then the client will know that it should only
cache the object locally for 5 more minutes

• Date Header: If a date header was not added by the server, then Vantage will add a date
header to the object served from its HTTP cache. This header indicates to the client when
the object was originally sent by the server to the HTTP cache in Vantage
• Cacheable Object Size: The minimum and maximum size of an object (image, script, and so
on) that can be stored in the Vantage HTTP cache, in bytes. Most objects smaller than 100
bytes are web beacons and should not be cached despite being image objects

• Cache Expire Time: An intermediate cache must be able to guarantee that it is not serving
stale content. If the server sends headers indicating how long the content can be cached
128
VMware Confidential - Internal Only
 (such as cache control), then Vantage will use those values. If the server does not send
expiration timeouts and Vantage is unable to make a strong determination of freshness,
then Vantage will store the object for no longer than the duration of time specified by the
Cache Expire Time

• Heuristic Expire: If a response object from the server does not include the Cache-Control
header but does include an If-Modified-Since header, then Vantage will use this time to
calculate the cache-control expiration, which will supersede the Cache Expire Time setting
for this object

• Cache URL with Query Arguments: This option allows caching of objects whose URI
includes a query argument. Disabling this option prevents caching these objects. When
enabled, the request must match the URI query to be considered a hit. Below are two
examples of URIs that include queries. The first example may be a legitimate use case for
caching a generic search, while the second may be a unique request posing a security
liability to the cache

— [Link]/[Link]?search=caching

— [Link]/[Link]?loginID=User

• Cacheable Mime Types: Statically defines a list of cacheable objects. This may be a String
Group, such as System-Cacheable-Resource-Types, or a custom comma-separated list of
Mime types that Vantage should cache. If no Mime Types are listed in this field, then
Vantage will by default assume that any object is eligible for caching

• Non-Cacheable Mime Types: Statically define a list of objects that are not cacheable. This
creates a denylist that is the opposite of the cacheable list

4-61 HTTP Caching (2)

• When caching is enabled, SE caches HTTP objects for the following types of responses:

— HTTP/HTTPS

— GET, HEAD methods

— 200 status code

VMware Confidential - Internal Only 129

4-62 HTTP Caching (3)
• SE never caches HTTP objects for the following types of responses:

— Put / Post / Delete methods

— Request Headers:

— Cache-Control: no-store

— Authorization

— Response Headers:

— Cache-Control: no-cache

— Expires header’s date is already expired

— Warning, Set-Cookie, Vary: *

— Cache-Control: private, no-store

— Both etag and Last-Modified headers do not exist and either:

— GET/HEAD method includes a Query

— No expires/max-age header exists

— Non-200 status codes

130
VMware Confidential - Internal Only
4-63 HTTP Caching Tab and Configuration

• To enable caching, check the Caching checkbox

The following parameters all are optional:

• X-Cache: Vantage will add an HTTP header labeled X-Cache for any response sent to the
client that was served from the cache. This header is informational only, and will indicate that
the object was served from an intermediary cache

• Age Header: Vantage will add a header to the content served from cache that indicates to
the client the number of seconds that the object has been in an intermediate cache. For
example, if the originating server declared that the object should expire after 10 minutes and
it has been in the Vantage cache for 5 minutes, then the client will know that it should only
cache the object locally for 5 more minutes

• Date Header: If a date header was not added by the server, then Vantage will add a date
header to the object served from its HTTP cache. This header indicates to the client when
the object was originally sent by the server to the HTTP cache in Vantage

• Cacheable Object Size: The minimum and maximum size of an object (image, script, and so
on) that can be stored in the Vantage HTTP cache, in bytes. Most objects smaller than 100
bytes are web beacons and should not be cached despite being image objects

• Cache Expire Time: An intermediate cache must be able to guarantee that it is not serving
stale content. If the server sends headers indicating how long the content can be cached
(such as cache control), then Vantage will use those values. If the server does not send
expiration timeouts and Vantage is unable to make a strong determination of freshness,
then Vantage will store the object for no longer than the duration of time specified by the
Cache Expire Time

• Heuristic Expire: If a response object from the server does not include the Cache-Control
header but does include an If-Modified-Since header, then Vantage will use this time to
calculate the cache-control expiration, which will supersede the Cache Expire Time setting
for this object

• Cache URL with Query Arguments: This option allows caching of objects whose URI
includes a query argument. Disabling this option prevents caching these objects. When

VMware Confidential - Internal Only 131

 enabled, the request must match the URI query to be considered a hit. Below are two
examples of URIs that include queries. The first example may be a legitimate use case for
caching a generic search, while the second may be a unique request posing a security
liability to the cache

— [Link]/[Link]?search=caching

— [Link]/[Link]?loginID=User

• Cacheable Mime Types: Statically defines a list of cacheable objects. This may be a String
Group, such as System-Cacheable-Resource-Types, or a custom comma-separated list of
Mime types that Vantage should cache. If no Mime Types are listed in this field, then
Vantage will by default assume that any object is eligible for caching

• Non-Cacheable Mime Types: Statically define a list of objects that are not cacheable. This
creates a denylist that is the opposite of the cacheable list

132
VMware Confidential - Internal Only
4-64 HTTP Caching CLI Commands
• To show Cached Objects from the Pool’s Cache

— : > show pool prod-l7-pool httpcache

• To show Cache Statistics

— : > show pool prod-l7-pool httpcachestats

• Clear Cache Objects

— : > clear pool prod‐l7‐pool httpcache

• Clear Cache Statistics

— : > clear pool prod‐l7‐pool httpcachestats

• Filter Cache Objects

— : > show pool prod-l7-pool httpcache filter resource_type html

VMware Confidential - Internal Only 133

4-65 Lesson 10: Connection Multiplexing

4-66 Introduction to Connection Multiplexing

(1)
• This option controls the behavior of HTTP request switching and server TCP connection
reuse. This allows SE to reduce the number of open connections maintained by servers and
better distribute requests across idle servers, thus reducing server overloading and
improving performance for end users

• The exact reduction of connections to servers will depend on:

— how long lived the client connections are

— HTTP version

— how frequently request/responses are utilizing the connection

4-67 Introduction to Connection Multiplexing

(2)
Connection Multiplexing behavior changes with server persistence enabled:

• Multiplex enabled, Persistence disabled: Requests are load-balanced across the servers in
the pool using either new or pre-existing connections to those servers. The connections to
the servers may be shared by requests from any clients

• Multiplex enabled, Persistence enabled: Client connections and their requests are sent to a
single server. These requests may share connections with other clients who are persisted to
the same server. Load balancing of HTTP requests is not performed

134
VMware Confidential - Internal Only
• Multiplex disabled, Persistence enabled: Vantage opens a new TCP connection to the
server for each connection received from the client. Connections are not shared with other
clients. The number of client connections will be the same as the number of server
connections

• Multiplex disabled, Persistence disabled: Connections between the client and server are
one-to-one. Requests remain on the same connection they began on. Multiple connections
from the same client may be distributed among the available servers

VMware Confidential - Internal Only 135

4-68 Lesson 11: Policies Overview

4-69 Virtual Service Policies: Overview

• Virtual Service Policies consist of one or more rules that control the flow of requests
through the virtual service

• Virtual Service policies are used to perform following action on client request and server
response:

— Security: Network layer security, HTTP security

— Insert dynamic data: Modify Header of HTTP Request and HTTP Response

• Policies are not shared by default. They are defined on a per virtual service basis

• Starting with NSX ALB release 18.2.5, network security policies can be shared across virtual
services

136
VMware Confidential - Internal Only
4-70 Virtual Service Policies: Rules
• Policies are comprised of one or more rules

• A Rule is match/action pairs, which are similar in concept to “if / then” logic

• Any connection or request matches the rule’s match criteria can generate logs

• Rule Match

— Match options will vary depending on the context defined by the policy type (HTTP,
DNS, L4)

— If a rule has multiple matching conditions configured, then all conditions must be true for
the action to be executed

— If a rule is not given a match, then all connections or requests will be considered true or
matched

• Rule Action

— One or more actions can be taken when the matches are true

— The available options will vary depending on the type of rule

VMware Confidential - Internal Only 137

4-71 HTTP Virtual Service Policies (1)
• HTTP policies are only available on Virtual Services with HTTP related Application Profiles

• Allow advanced customization of network layer security, HTTP security, HTTP requests,
and HTTP responses

• To control security, client http request header attributes, or server http response header
attributes

— Allow or deny client based on Client IP

— HTTP redirect from HTTP to HTTPS

— Content switching

— Insert or modify HTTP header

• HTTP policies can be referenced by Virtual Service which has an application profile of type
HTTP/HTTPS application profile

4-72 HTTP Virtual Service Policies (2)

• HTTP Virtual Service policies rule options

138
VMware Confidential - Internal Only
4-73 HTTP Virtual Service Policies: Network
Security Rule (1)
• Explicitly allow or block traffic of user-defined types onto the network

Select Policies Tab

Select Network Security

Click + to Add rule

Matching Rule Section

Click dropdown button to select Match

Criteria

VMware Confidential - Internal Only 139

4-74 HTTP Virtual Service Policies: Network
Security Rule (2)

140
VMware Confidential - Internal Only
4-75 HTTP Virtual Service Policies: HTTP
Security Rule (1)
• Inspects HTTP Request and Response headers

• Perform defined actions such as allow/deny, redirect to HTTPS, or respond with a static
page

Click ‘+’ to Add a rule

VMware Confidential - Internal Only 141

4-76 HTTP Virtual Service Policies: HTTP
Security Rule (2)

142
VMware Confidential - Internal Only
4-77 HTTP Virtual Service Policies: HTTP
Security Rule (3)

VMware Confidential - Internal Only 143

4-78 HTTP Virtual Service Policies: HTTP
Security Rule (4)

Select Match Criteria

Value

144
VMware Confidential - Internal Only
4-79 HTTP Virtual Service Policies: HTTP
Security Rule (5)

Select Action to be taken

after Criteria Match

VMware Confidential - Internal Only 145

4-80 HTTP Virtual Service Policies: HTTP
Security Rule (6)

146
VMware Confidential - Internal Only
4-81 HTTP Virtual Service Policies: HTTP
Request Rule (1)
• Allow manipulation of HTTP Requests like capture data from one location and apply it to
another location, insert new HTTP header

• Content switching based on client HTTP Request header key and value pair

VMware Confidential - Internal Only 147

4-82 HTTP Virtual Service Policies: HTTP
Request Rule (2)

148
VMware Confidential - Internal Only
4-83 HTTP Virtual Service Policies: HTTP
Request Rule (3)

VMware Confidential - Internal Only 149

4-84 HTTP Virtual Service Policies: HTTP
Request Rule (4)

150
VMware Confidential - Internal Only
4-85 HTTP Virtual Service Policies: HTTP
Request Rule (5)

VMware Confidential - Internal Only 151

4-86 HTTP Virtual Service Policies: HTTP
Request Rule (6)

152
VMware Confidential - Internal Only
4-87 HTTP Virtual Service Policies: HTTP
Request Rule (7)

VMware Confidential - Internal Only 153

4-88 HTTP Virtual Service Policies: HTTP
Response Rule (1)
• Used to modify the server’s response headers, example : rewriting a website’s name

• Perform defined actions such as allow/deny, redirect to HTTPS, or respond with a static
page

154
VMware Confidential - Internal Only
4-89 HTTP Virtual Service Policies: HTTP
Response Rule (2)

Click dropdown button to

select match criteria

VMware Confidential - Internal Only 155

4-90 HTTP Virtual Service Policies: HTTP
Response Rule (3)

156
VMware Confidential - Internal Only
4-91 HTTP Virtual Service Policies: HTTP
Response Rule (4)

VMware Confidential - Internal Only 157

4-92 HTTP Virtual Service Policies: HTTP
Response Rule (5)

158
VMware Confidential - Internal Only
4-93 HTTP Virtual Service Policies: HTTP
Response Rule (6)

VMware Confidential - Internal Only 159

4-94 DNS Virtual Service Policies (1)
• Match DNS request, such as query type, query domain name, DNS transport protocol used,
client IP originating the request

• Allow or deny client DNS request based on configured set of IP addresses

• Routing of requests to back-end DNS servers

• Override the usual GSLB-algorithm-based response

• Redirect DNS queries over UDP to instead come over TCP

• DNS policies can be referenced by Virtual Service which has an application profile of type
DNS

4-95 DNS Virtual Service Policies (2)

• DNS Virtual Service policies rule options

160
VMware Confidential - Internal Only
4-96 DNS Virtual Service Policies: DNS Rule (1)

VMware Confidential - Internal Only 161

4-97 DNS Virtual Service Policies: DNS Rule
(2)

162
VMware Confidential - Internal Only
4-98 DNS Virtual Service Policies: DNS Rule
(3)

Enter Rule name or

continue with default

Click dropdown button

to select match criteria

VMware Confidential - Internal Only 163

4-99 DNS Virtual Service Policies: DNS Rule
(4)

Select match criteria from

164
VMware Confidential - Internal Only
4-100 DNS Virtual Service Policies: DNS Rule
(5)

VMware Confidential - Internal Only 165

4-101 DNS Virtual Service Policies: DNS Rule
(6)

Select Action to be taken on the

166
VMware Confidential - Internal Only
4-102 DNS Virtual Service Policies: DNS Rule
(7)

VMware Confidential - Internal Only 167

4-103 References

• Create Virtual Service

• Difference Between Virtual Service and Virtual IP

• Disable a Virtual Service

• Layer 4 SSL Support

• Server Pool

• Non-Uniform Round-Robin Traffic Distribution

• Overview of Server Persistence

• HTTP Cookie Persistence

• Client IP Persistence

• Custom HTTP Header Persistence

• TLS Persistence

• App Cookie Persistence

• Pool Group

• Health Monitor

168
VMware Confidential - Internal Only
Module 5
Pools Configuration Concepts

5-2 Module Overview

1. Pools Overview

2. Pools Configuration

3. Pool Groups

VMware Confidential - Internal Only 169

5-3 Lesson 1: Pools Overview

5-4 About Pools (1)

• A pool is the list of servers hosting the same application

• Consists of individual IPs (FQDNs) and port numbers

• Load-balanced by an algorithm

Pools maintain the list of servers assigned to them and perform health monitoring, load
balancing, persistence, and functions that involve Avi-Vantage-to-server interaction.

A typical virtual service will point to one pool. However, more advanced configurations may
have a virtual service content switching across multiple pools via HTTP request policies or
DataScripts.

5-5 About Pools (2)

• Health Monitors monitor pool members.

• Client connections are persisted by an algorithm.

• May encrypt back-end connections.

• Pools can be shared across Virtual Services.

170
VMware Confidential - Internal Only
5-6 Lesson 2: Pool Configuration

5-7 Creating a Pool: Settings

• Navigate to Application > Pool > Create Pool

• Pools are Tenant, Cloud, and VRF specific

VMware Confidential - Internal Only 171

5-8 Creating a Pool: Servers
• Servers can be added by IP address, IP range, or DNS name.

• By selecting the server check box you can remove, enable, or disable it.

The servers added to a pool may be specified one of three ways:

IP address, IP range, or DNS name:

Add Server

Select Server By Network: available if Avi has read or write access to the cloud orchestrator.
Provides significantly richer information regarding the server, like CPU, memory, and disk
utilization.

IP group: Server IP addresses are stored in separate comma-separated IP Group that configured
elsewhere

Auto-scaling groups: defined by public cloud (AWS/Azure/GCP)

Changing Server Status: By selecting server checkbox you can Remove, Enable or Disable it

• Changing the DNS Refresh Interval

• The default DNS refresh time is 60 minutes. This may be changed using the CLI:

• : > configure controller properties dns_refresh_period 50 : > save

172
VMware Confidential - Internal Only
5-9 Creating a Pool: Advanced
• The Advanced tab of the Pool Create/Edit pop-up specifies optional settings for the pool.

Placement Settings

In some scenarios, a server may exist in multiple networks. Similarly, a network may have
multiple IP subnets or a single subnet may exist in multiple networks. For example, VMware
servers may have multiple port groups assigned to a single subnet, or a single port group is
assigned to multiple subnets. Normally, Avi Vantage will try to determine the network for the
servers. However in scenarios where it cannot determine which network to use, an administrator
may be required to manually select it as follows.

Server Network: Click the pull-down menu icon to reveal potential networks for server
placement. The list might look as shown below. Click the desired network.

Candidate networks

Subnet: Once the network has been chosen, a subnets will be displayed. Choose one or enter
one using the syntax [Link]/24.

Select subnet

Pool Full Settings

VMware Confidential - Internal Only 173

This section configures HTTP request queuing, which causes Avi Vantage to queue requests
that are received after a back-end server has reached its maximum allowed number of
concurrent connections. Queuing HTTP requests provides time for new connections to become
available on the server, thus avoiding the configured pool-down action. For complete details,
refer to the HTTP Request Queueing article.

Pool Failure Settings

Fail Action: Three fail actions are defined.

Close Connection: If all servers in a pool are down, the default behavior of the virtual service is
to close new client connection attempts by issuing TCP resets or dropping UDP packets.
Existing connections are not terminated, even though their server is marked down. The
assumption is the server may be slow but may still be able to continue processing the existing
client connection.

HTTP Local Response: Returns a simple webpage. Specify a status code of 200 or 503. If a
custom HTML file has not been uploaded to Avi Vantage, it will return a basic page with the
error code.

HTTP local response

Status Code: Select 200 or 503 from the pull-down.

Upload File: click the button to navigate to and select an HTML page to be returned as the SE-
local response.

HTTP Redirect: Returns a redirect HTTP response code, including a specified URL.
HTTP redirect

Status Code: Choose 301, 302, or 307 from the pull-down.

HTTP/HTTPS: By default Avi Vantage will redirect via HTTPS unless HTTP is clicked instead.

URL: Enter a URL of the format [Link]/path/file?query=bbb.

Note: A virtual service is marked up when at least one of the pools associated with the virtual
service is up, or if there are redirect policies that do not need pools. The mere presence of pool-
down action by itself does not mark the virtual service up.

Starting with Avi Vantage release 18.2.1, you can specify the minimum threshold parameters for
a pool to make it serviceable. For more information, view Parameters to Mark a Virtual Service
or Pool up.

If a virtual service is marked down, and any of the following situations apply, then the pool-down
action is not triggered:

“Remove Listening Port when VS Down” setting on the VS - When this is set, and the VS is
down, the dispatcher drops the packets, and the pool-down action is not triggered.

BGP cases: When the virtual service is marked down, BGP withdraws this VS from the peer.
Consequently, the peer does not send any traffic to this SE.

174
VMware Confidential - Internal Only
ECMP case (without route summarization) - When the virtual service is marked down, the
Controller withdraws the VS from the router. As a result, the router does not send any traffic to
this SE.

Other Settings

Disable Port Translation: This feature is for virtual services that are listening on multiple service
ports, such as Microsoft Lync, which has multiple listener ports. Instead of having all connections
directed to a single port on the server (defined by the pool’s default server port or the server’s
optional port field), they will be sent to the same port that they were received on the virtual
service.

Description: Enter an optional description of up to 256 characters in this field. This field is for user
convenience only.

Connection Ramp: Enabling this option by entering a number larger than 0 results in a graceful
increase in the number of new connections sent to a server over the specified time period. For
example, assume that the load-balancing algorithm is set to least connections and a pool has two
servers with 100 connections each. Adding a third server would immediately overwhelm that
third server by immediately sending the next 100 consecutive connections to it. Setting a
connection ramp adds traffic to a new server in a manner similar to using a ratio. Over the
specified period of time, the new server will receive an ever-increasing ratio of traffic in relation
to its peers. For instance, setting the ramp to 4 seconds means that the new server will receive
1/4 of the traffic it would normally be given for the first second. By the second second, the
server will be receiving 1/2 the traffic it might otherwise have been given. After the 4-second
ramp time has elapsed, the server will receive the normal amount of traffic as determined by the
load-balancing algorithm.

Max Connections per Server: Specify the maximum number of concurrent connections allowed
for a server. If all servers in the pool reach this maximum the virtual service will send a reset for
TCP connections or silently discard new UDP streams unless otherwise specified in the Pool
Down Action, described above. As soon as an existing connection to the server is closed, that
server is eligible to receive the next client connection. A value of 0 disables the connection limit.

HTTP Server Reselect: This option retries an HTTP request that fails or returns one of a set of
user-specified error codes from the back-end server. Normally, Avi Vantage forwards these
error messages back to the client. For more information, refer to this article.

VMware Confidential - Internal Only 175

5-10 Creating a Pool: Review
• The Review tab displays a summary of the information entered in the previous pool creation
tabs.

Placement Settings

In some scenarios, a server may exist in multiple networks. Similarly, a network may have
multiple IP subnets or a single subnet may exist in multiple networks. For example, VMware
servers may have multiple port groups assigned to a single subnet, or a single port group is
assigned to multiple subnets. Normally, Avi Vantage will try to determine the network for the
servers. However in scenarios where it cannot determine which network to use, an administrator
may be required to manually select it as follows.

Server Network — click the pull-down menu icon to reveal potential networks for server
placement. The list might look as shown below. Click the desired network.
Candidate networks

Subnet — Once the network has been chosen, a subnet(s) will be displayed. Choose one or
enter one using the syntax [Link]/24.

Select subnet

Pool Full Settings

This section configures HTTP request queuing, which causes Avi Vantage to queue requests
that are received after a back-end server has reached its maximum allowed number of
concurrent connections. Queuing HTTP requests provides time for new connections to become
available on the server, thus avoiding the configured pool-down action. For complete details,
refer to the HTTP Request Queueing article.

Pool Failure Settings

Fail Action — Three fail actions are defined.

176
VMware Confidential - Internal Only
Close Connection — If all servers in a pool are down, the default behavior of the virtual service is
to close new client connection attempts by issuing TCP resets or dropping UDP packets.
Existing connections are not terminated, even though their server is marked down. The
assumption is the server may be slow but may still be able to continue processing the existing
client connection.

HTTP Local Response — Returns a simple webpage. Specify a status code of 200 or 503. If a
custom HTML file has not been uploaded to Avi Vantage, it will return a basic page with the
error code.

HTTP local response

Status Code — Select 200 or 503 from the pull-down.

Upload File — click the button to navigate to and select an HTML page to be returned as the SE-
local response.

HTTP Redirect — Returns a redirect HTTP response code, including a specified URL.

HTTP redirect

Status Code — Choose 301, 302, or 307 from the pull-down.

HTTP/HTTPS — By default Avi Vantage will redirect via HTTPS unless HTTP is clicked instead.

URL — Enter a URL of the format [Link]/path/file?query=bbb.

Note: A virtual service is marked up when at least one of the pools associated with the virtual
service is up, or if there are redirect policies that do not need pools. The mere presence of pool-
down action by itself does not mark the virtual service up.

Starting with Avi Vantage release 18.2.1, you can specify the minimum threshold parameters for
a pool to make it serviceable. For more information, view Parameters to Mark a Virtual Service
or Pool up.

If a virtual service is marked down, and any of the following situations apply, then the pool-down
action is not triggered:

BGP cases - When the virtual service is marked down, BGP withdraws this VS from the peer.
Consequently, the peer does not send any traffic to this SE.

ECMP case (without route summarization) - When the virtual service is marked down, the
Controller withdraws the VS from the router. As a result, the router does not send any traffic to
this SE.

Other Settings

Disable Port Translation — This feature is for virtual services that are listening on multiple service
ports, such as Microsoft Lync, which has multiple listener ports. Instead of having all connections
directed to a single port on the server (defined by the pool’s default server port or the server’s
optional port field), they will be sent to the same port that they were received on the virtual
service.

VMware Confidential - Internal Only 177

Description — Enter an optional description of up to 256 characters in this field. This field is for
user convenience only.

Connection Ramp — Enabling this option by entering a number larger than 0 results in a graceful
increase in the number of new connections sent to a server over the specified time period. For
example, assume that the load-balancing algorithm is set to least connections and a pool has two
servers with 100 connections each. Adding a third server would immediately overwhelm that
third server by immediately sending the next 100 consecutive connections to it. Setting a
connection ramp adds traffic to a new server in a manner similar to using a ratio. Over the
specified period of time, the new server will receive an ever-increasing ratio of traffic in relation
to its peers. For instance, setting the ramp to 4 seconds means that the new server will receive
1/4 of the traffic it would normally be given for the 1st second. By the 2nd second, the server will
be receiving 1/2 the traffic it might otherwise have been given. After the 4-second ramp time
has elapsed, the server will receive the normal amount of traffic as determined by the load-
balancing algorithm.

Max Connections per Server — Specify the maximum number of concurrent connections allowed
for a server. If all servers in the pool reach this maximum the virtual service will send a reset for
TCP connections or silently discard new UDP streams unless otherwise specified in the Pool
Down Action, described above. As soon as an existing connection to the server is closed, that
server is eligible to receive the next client connection. A value of 0 disables the connection limit.

HTTP Server Reselect — This option retries an HTTP request that fails or returns one of a set of
user-specified error codes from the back-end server. Normally, Avi Vantage forwards these
error messages back to the client. For more information, refer to this article.

178
VMware Confidential - Internal Only
5-11 Load-Balancing Algorithms (1)
• The heart of a load balancer

• Effectively distribute traffic across servers

Algorithms are:

• Round robin:

New connections are sent in a sequential order.

• Least connections (default):

New connections are sent to the server that has the least concurrent connections.

• Least load:

HTTP-specific new connections are sent to the server with the least HTTP load.

• Fewest servers:

Determine the fewest number of servers required. Excess servers will no longer receive
traffic. Cost effective in the cloud with AutoScale Policy (AWS/Azure/GCP).

VMware Confidential - Internal Only 179

5-12 Load-Balancing Algorithms (2)
• Consistent hash:

New connections are distributed across the servers using a hash

Hash keys available: Custom Header, Source IP, Source IP and Port, HTTP URI, Custom
String

Combined load-balancing algorithm and persistence method

• Fastest response:

New connections are sent to the back-end server that is currently providing the fastest
response time to new requests

• Core affinity:

Provides a mapping between servers and CPU cores, New connections established
according to that mapping

180
VMware Confidential - Internal Only
5-13 Persistence Algorithms (1)
• Force a client to stay connected to the same server

• Sometimes referred to as sticky connections

5-14 Persistence Algorithms (2)

Algorithms are:

— HTTP Cookie

Avi inserts a cookie into HTTP responses

— App Cookie

Avi reads existing server cookies or URI embedded data

— HTTP Header

Avi uses custom static mappings of header values to specific servers

— Client IP Address

The client’s IP is used

— TLS
Persistence information is in the client’s SSL/TLS ticket ID

— DataScript

Custom persistence may be built using DataScripts

VMware Confidential - Internal Only 181

5-15 Health Monitors
• NSX Advanced Load Balancer can actively send one or more health checks to detect
liveness

• Passive health monitor is used to form a view of client experience

• Health Monitors are attached to a Pool

• Inactive when the pool is not utilized by a Virtual Service or Disabled

• Health Monitoring is only performed for pools that are associated with active Virtual
Services

182
VMware Confidential - Internal Only
5-16 Passive Health Monitor
• Track the client-server interactions:

— Is the server quickly responding with valid responses (such as HTTP 200)?

— Is the server sending back errors (such as TCP resets or HTTP 5xx errors)?

Then server is in trouble!

— Enabled by default

— Available only for HTTP Virtual Services

• The passive health monitor cannot cause NSX ALB to mark that server as down

— Instead reduce the number of requests up to 75%

— Add a penalty to the application index score

— Errors are defined by the analytics profile

• Best practice is to enable both a passive and an active health monitor

VMware Confidential - Internal Only 183

5-17 Active Health Monitors
• Periodically send queries to pool servers

• Multiple Active Health Monitors can be assigned

• By default, all HMs must pass to mark server UP (logical AND)

• Service Engines perform all active monitoring

Configurable active health monitors:

— HTTP Monitors

— HTTPS Monitors

— DNS Monitors

— Ping Monitors

— TCP Monitors

— UDP Monitors

— External Monitors

— RADIUS Monitors

— SIP Monitors

184
VMware Confidential - Internal Only
5-18 DNS Monitor
Validates the health of DNS servers by sending a UDP DNS request and comparing the
response IP address

VMware Confidential - Internal Only 185

5-19 Ping Monitor
Service Engines will send an ICMP Request to the server

186
VMware Confidential - Internal Only
5-20 TCP Monitor
• By default, the monitor will wait for the TCP handshake connection establishment.

• Optionally a request string can be sent and response can be used to identify the state of
the server.

• Half Open option is supported, where monitor sends a SYN and upon receipt of an ACK,
Service Engine responds with an RST.

• Maintenance Server Response Data option, If the defined string is seen in the server
response, place the server in maintenance mode (health checks continue).

VMware Confidential - Internal Only 187

5-21 UDP Monitor
• Send a UDP datagram to the server, and match the server’s response against the expected
response data.

• Send the request string, and wait for the server to respond with the expected content.

• Maintenance Server Response Data option, If the defined string is seen in the server
response, place the server in maintenance mode (health checks continue).

188
VMware Confidential - Internal Only
5-22 HTTP Monitor (1)
• Used to validate the health of HTTP servers

• Client Request Data field holds HTTP request to the web server

Example: GET /[Link] HTTP/1.1\r\nHost:

[Link]\r\nConnection: Close\r\n\r\n\r\n
• Use Exact Request to use the exact http request string as specified by the user

• Client Request Data field holds HTTP response code to match successful. Example: HM
Page OK

• Maintenance Response Code and Response Data are support

• Response Code defines a successful HTTP code. Example: 200 OK

5-23 HTTP Monitor (2)

VMware Confidential - Internal Only 189

5-24 HTTP Monitor (3)
• Inherits HTTP monitor options

• SSL Attributes

— TLS SNI Server Name to be matched FQDN in SSL host header extension

— SSL Profile defines the ciphers and SSL versions to be used for the health monitor
traffic

— PKI Profile will be used as to validate the SSL certificate presented by the server

— SSL Key and Certificate will be used as to present this SSL certificate to the server

190
VMware Confidential - Internal Only
5-25 External Monitor (1)
• The external monitor type uses scripts to provide highly customized health checks

• Linux Shell, Python, or Perl (example: wget)

5-26 External Monitor (2)

• Settings:

— Script Variables are the custom variables, to allow simplified reusability (Example:
USER=admin)

— Example Script Code

#!/bin/bash
curl [Link] -I -L --ntlm -u
$USER:$PASS -I -L | grep "200 OK"
If a script exits with output, it is considered as success

— Script Parameters are passed in as arguments, such as $1 = server IP, $2 = server port

5-27 External Monitor (3)

VMware Confidential - Internal Only 191

5-28 SIP Monitor
• Server health can be monitored using the SIP request Code and SIP response.

• SIP Monitor Transport can be either UDP or TCP.

192
VMware Confidential - Internal Only
5-29 RADIUS Monitor
• Server health can be monitored by using the RADIUS request and response.

• The server status will be marked UP only if the RADIUS response is Access Accept or
Access Challenge.

VMware Confidential - Internal Only 193

5-30 Lesson 3: Pool Groups

5-31 Pool Groups Overview (1)

• A pool group is a list of server pools, accompanied by logic to select a server pool from the
list.

• Each pool has Priority and Ratio (Weight).

194
VMware Confidential - Internal Only
5-32 Pool Groups Overview (2)
• Persistence is defined and honored within the pool.

• Pool groups are associated in the same way to Virtual Services as pools and can be shared
across.

• Useful during server upgrades (Blue/Green Deployment, Canary Upgrades).

VMware Confidential - Internal Only 195

5-33 Pool Groups Configuration
• Navigate to Applications > Pool Groups > Create Pool Group.

• The highest priority pool is chosen as long as a server is available within it.

• Ratio governs the traffic percentage within the defined priority groups.

5-34 Pool Group Best Practices

You must keep the priorities spaced, and leave gaps.

For the pure priority use case, the ratio of the pool group is optional.

Setting the ratio to 0 for a pool results in sending no traffic to this pool.

196
VMware Confidential - Internal Only
5-35 Pool or Pool Groups
Pools and pool groups can be used interchangeably on a virtual service.

Pool Groups provide extra flexibility, without disruption to existing traffic.

The existing connections are not disrupted when pool group membership changes.

5-36 References
[Link]

[Link]

VMware Confidential - Internal Only 197

VMware Confidential - Internal Only
Module 6
Modifying Application Behavior

6-2 Module Lessons

1. Controlling Traffic with NSX Advanced Load Balancer

2. L7 Traffic Manipulation

3. L4 Traffic Manipulation

4. Customizing Application Delivery with DataScripts

5. Getting Started with DataScripts

6. DataScripts Constructs

7. Using DataScripts Examples

VMware Confidential - Internal Only 199

6-3 Lesson 1: Controlling Traffic with NSX
Advanced Load Balancer (1)
• Profiles: Used to group collections of settings into a single, reusable object

— Application

200
VMware Confidential - Internal Only
6-4 Controlling Traffic with NSX Advanced
Load Balancer (2)
• Profiles: Used to group collections of settings into a single, reusable object

— TCP/UDP

VMware Confidential - Internal Only 201

6-5 Controlling Traffic with NSX Advanced
Load Balancer (3)
• Profiles: Used to group collections of settings into a single, reusable object

— Persistence

6-6 Controlling Traffic with NSX Advanced

Load Balancer (4)
• Profiles: Used to group collections of settings into a single, reusable object

— And many more

• SSL

• Health Monitors

• PKI

• And more

202
VMware Confidential - Internal Only
6-7 Controlling Traffic with NSX Advanced
Load Balancer (5)
• Policies: Used to control security, client request, or server response attributes

— Network Security

— HTTP Security

— HTTP Request

— HTTP Response

6-8 Controlling Traffic with NSX Advanced

Load Balancer (6)
• Policies: Used to control security, client request, or server response attributes

VMware Confidential - Internal Only 203

6-9 Lesson 2: L7 Traffic Manipulation

6-10 App Teams Need to Know Who Is

Connecting

6-11 X-Forwarded-For (1)

• X-Forwarded-For inserts a header that contains the client IP address

• Is configured within an HTTP application profile

204
VMware Confidential - Internal Only
6-12 X-Forwarded-For (2)
• Can also be configured from an HTTP Request Policy

VMware Confidential - Internal Only 205

6-13 X-Forwarded-For (3)
• The client-side protocol can also be passed to the back end

206
VMware Confidential - Internal Only
6-14 Validating That Headers Are Inserted

6-15 Capturing and Examining Headers (1)

• With a L7 service, the headers can be captured on both the client and server side

• Edit the Virtual Service

• Click Analytics

• Select the Log all headers check box

VMware Confidential - Internal Only 207

6-16 Capturing and Examining Headers (2)
• Click your Virtual Service logs

6-17 Capturing and Examining Headers (3)

• Select the log you want and click on the plus sign to expand it

• Click View All Headers

208
VMware Confidential - Internal Only
6-18 Capturing and Examining Headers (4)
The X-Forwarded-For and X-Forwarded-Proto headers that Avi inserted.

VMware Confidential - Internal Only 209

6-19 Minimizing the Number of Connections to
Back-End Servers

210
VMware Confidential - Internal Only
6-20 Connection Multiplexing (1)
• Connection multiplexing controls the behavior of HTTP 1.0 and 1.1 request switching and
server TCP connection reuse

• Decouples the client TCP and HTTP connection from the server-side TCP and HTTP
connection

• Multiplexing provides three important benefits to improve the efficacy and performance of
servers

— Reduces the number of connections to the server that must be opened and closed in a
given period

— Reduces the number of concurrently open connections to the server relative to the
number of open client connections

— Allows load balancing and distribution of HTTP requests across servers using any open
server-side connection

6-21 Connection Multiplexing (2)

VMware Confidential - Internal Only 211

6-22 Traffic Needs to Be HTTPS but HTTP
Traffic Cannot Be Missed

6-23 HTTP to HTTPS Redirection

• For security, an industry best practice is to ensure that all HTTP traffic is SSL-encrypted as
HTTPS

• Because typical end users do not specify the HTTPS protocol, the initial requests arrive
over HTTP

• What to do?

• How to handle this?

• HTTP to HTTPS Redirection

212
VMware Confidential - Internal Only
6-24 HTTP to HTTPS Redirection: App Profile

VMware Confidential - Internal Only 213

6-25 HTTP to HTTPS Redirection: HTTP
Security Policy

214
VMware Confidential - Internal Only
6-26 Blocking Bad Actors

6-27 Using Network Security Policies to

Denylist (1)
• A client’s IP address may need to be prevented from accessing an application for any
number of reasons

• Using a Network Security Policy is a simple way to stop unauthorized access

VMware Confidential - Internal Only 215

6-28 Using Network Security Policies to
Denylist (2)
• As an alternative to specifying the IP addresses within the policy, you can use an IP group

216
VMware Confidential - Internal Only
6-29 Using Network Security Policies to
Denylist (3)
• Add the IP addresses, ranges, or subnets into the group

VMware Confidential - Internal Only 217

6-30 Using Network Security Policies to
Denylist (4)
• Alternatively, you can select by country code

218
VMware Confidential - Internal Only
6-31 Using Network Security Policies to
Denylist (5)
• Can use multiple IP groups

VMware Confidential - Internal Only 219

6-32 App Team Wants Certain Traffic to Be
Processed Apart from Their Main App

6-33 HTTP Redirects

• Redirects a client to a different protocol, port, host, or path

• By default the host, path, and query are not altered unless an administrator populates those
fields

220
VMware Confidential - Internal Only
6-34 Finding Other Options

6-35 Content Switching (1)

• Forward matched requests to a pool, or a specific server with that pool

• Requests can also be forwarded to a pool group

VMware Confidential - Internal Only 221

6-36 Content Switching (2)
• Alternatively, NSX Advanced Load Balancer can serve an HTTP response code, along with
either a default or custom response page

222
VMware Confidential - Internal Only
6-37 Options Other Than Policies

6-38 DataScripts
• DataScripts are a powerful mechanism for customizing the behavior of NSX Advanced Load
Balancer

• Configured on a per Virtual Service basis, but can be shared

• Written in Lua

• Can be executed against each client making a TCP connection, an HTTP request or
response, or other events within the data plane

VMware Confidential - Internal Only 223

6-39 Example DataScripts
Close connection on unsupported extension based on the path

224
VMware Confidential - Internal Only
6-40 DataScripts
• DataScripts are a powerful mechanism for customizing the behavior of NSX Advanced Load
Balancer

• Configured on a per Virtual Service basis, but can be shared

• Written in Lua

• Can be executed against each client making a TCP connection, an HTTP request or
response, or other events within the data plane

• Attached to each Virtual Service under Policies/Datascripts

VMware Confidential - Internal Only 225

6-41 App Owner Wants to Authenticate Client
Connections (1)

6-42 App Owner Wants to Authenticate Client

Connections (2)
• What was Arthur trying to do?

• What was Arthur’s justification?

• What was the purpose of the guard?

226
VMware Confidential - Internal Only
6-43 Client SSL Certificate Validation
• There will be times that you need to validate the host that is trying to connect to your
application

• One way is to validate the identity of the client is by using an issued client certificate

• The client can be requested or required to present the certificate during the SSL handshake

• The certificate will be checked against a certificate authority to ensure it is genuine

• Optionally, the certificate can also be checked against a certificate revocation list (CRL) to
ensure that it is still valid

6-44 Client SSL Certificate Validation:

Requirements (1)
• HTTP Application Profile

VMware Confidential - Internal Only 227

6-45 Client SSL Certificate Validation:
Requirements (2)
• PKI Profile

228
VMware Confidential - Internal Only
6-46 Client SSL Certificate Validation: PKI
Profile (1)
• PKI Profiles define a Certificate Authority to validate a presented certificate’s authenticity

• Optionally, a PKI profile can validate a presented certificate’s validity

6-47 Client SSL Certificate Validation: PKI

Profile (2)
• Click Add CA to add CA Certificates

• Multiple certs can be added

VMware Confidential - Internal Only 229

6-48 Client SSL Certificate Validation: PKI
Profile (3)

230
VMware Confidential - Internal Only
6-49 Client SSL Certificate Validation: PKI
Profile (4)
• To use a CRL, ensure that the Enable CRL box is selected

• Click Add CRL

• Either a static file, or a URL may be defined

VMware Confidential - Internal Only 231

6-50 Client SSL Certificate Validation: Putting
It All Together (1)
• Edit the application profile assigned to your VS

232
VMware Confidential - Internal Only
6-51 Client SSL Certificate Validation: Putting
It All Together (2)
• Click the Security tab

VMware Confidential - Internal Only 233

6-52 Client SSL Certificate Validation: Putting
It All Together (3)
• Under Client SSL Certificate Validation, click request or required, based on enforcement
requirements

234
VMware Confidential - Internal Only
6-53 Client SSL Certificate Validation: Putting
It All Together (4)
• In the PKI Profile drop-down menu, select the PKI profile that you created

VMware Confidential - Internal Only 235

6-54 Client SSL Certificate Validation: Putting
It All Together (5)
• Optionally, additional headers may be inserted and passed to the backend servers

236
VMware Confidential - Internal Only
6-55 Traffic Speed

6-56 Rate-Limiting Traffic

• What happens when I get too much traffic?

— Too many PPS

— Too many CPS

— Too many RPS

— Too many SSL TPS

• What can I do to slow it down?

VMware Confidential - Internal Only 237

6-57 Throttling: Application Profile (1)
• One way to throttle your application is through the Application Profile

• Throttle will be applied to every VS that is using this profile

238
VMware Confidential - Internal Only
6-58 Throttling: Application Profile (2)
• A second way to throttle is the Advanced tab on the Virtual Service

• With this method, all connections to the Virtual Service will be throttled

• Edit the Virtual Service

• Click the Advanced tab

• Check the Performance Limits box

VMware Confidential - Internal Only 239

6-59 Throttling: Virtual Service
• A third way to throttle is through a Virtual Service HTTP security policy

• Benefit of this is the throttle is applied whenever the policy is matched

• No match, no throttle

240
VMware Confidential - Internal Only
6-60 App Teams Complain That Posts Are
Too Big

6-61 App Team Needs to Limit POST

Message Bodies
• App team is complaining that POST messages are too big

• Their application is expecting POST bodies to be no more than 200k

• With the Application Profile, this is easily controlled

VMware Confidential - Internal Only 241

6-62 Client Post Body Size
• Edit the Virtual Service’s application profile

• Click the DDoS tab

• Set the value with the Client Post Body Size field

242
VMware Confidential - Internal Only
6-63 Lesson 3: L4 Traffic Manipulation

6-64 Handling Idle Connections

6-65 Idle Connection Behavior

• TCP settings are controlled via the TCP/UDP profile

• TCP settings are set either manually, or automatically

• Default behavior is all TCP settings will be negotiated automatically

• Avi has two methods for handling idle TCP connections

— TCP Keepalive

— Age out idle connections

VMware Confidential - Internal Only 243

6-66 Idle Connection Types (1)
• TCP Keepalive

— Sends a periodic keep-alive signal to the client that will keep the current connection
open

244
VMware Confidential - Internal Only
6-67 Idle Connection Types (2)
• Age out idle connections

— Terminate idle connections that have no keep-alive signal from the client

— Idle timeout duration is controlled by the value within the Idle Duration field

— Default idle timeout for the TCP Proxy profile is 600 seconds, TCP Fast Path is 300
seconds

VMware Confidential - Internal Only 245

6-68 Users Are Seeing 403s After Successful
Authentication

6-69 Session Persistence

• A persistence profile governs the settings that will force a client to stay connected to the
same server for a specified duration of time

• Ensures that the client will reconnect to the same server every time, for a desired duration
of time

• Persistent connections are critical for most servers that maintain client session information
locally

• Persistence is configured within a pool

• L7 Persistence Types

— System-Persistence-App-Cookie

— System-Persistence-Client-IP

— System-Persistence-Custom-HTTP-Header

— System-Persistence-HTTP-Cookie

— System-Persistence-TLS

• L4 Persistence Types

— System-Persistence-Client-IP

246
VMware Confidential - Internal Only
6-70 Lesson 4: Customizing Application
Delivery with DataScripts

6-71 Overview
• Getting Started with DataScripts

• DataScripts Constructs

• Using DataScripts Examples

• Troubleshooting and Error Handling

VMware Confidential - Internal Only 247

6-72 Lesson 5: Getting Started with
DataScripts

6-73 DataScripts
Data Plane Scripting for Real-time Traffic Steering and Manipulation

What it is?

• Data plane manipulation and traffic steering for HTTP headers

• Executed on Service Engines (data plane)

• Lua scripting language built on an embedded Lua interpreter, with additional Avi specific
libraries, called functions, added

• DataScripts are more advanced, flexible versions of the Policies

• DataScript will typically be in some form of if / then logic, similar to a Policy's match / action
logic

What it is not?

• DataScript is Lua, whereas Policies are C code.

• DataScript is not as fast and may consume more CPU for an equivalent operation.

• ControlScript, Avi’s control plane scripting, is based on Python and is executed on the
Controller.

• DataScript will typically be in some form of if / then logic, similar to a Policy's match / action
logic.

• Intent is to eliminate 75% of customer’s custom scripting with AVI native functionality. The
rest may still require DataScript.

248
VMware Confidential - Internal Only
6-74 DataScripts for the Data Plane (1)
When neither rules nor policies can satisfy a specific use case

• DataScript – Lua-based tool manipulates data plane traffic

• Conceptually like policies, though more complex and more powerful

• One or more can be attached to an HTTP(S)-based virtual service

• Shareable across multiple virtual services, each virtual service object can refer to any
number of DataScript

• To use a DataScript with an HTTP request event, the virtual service must be configured for
application type HTTP

• Configured via the Avi Controller; executed on Avi Service Engines

• Executed in order

6-75 DataScripts for the Data Plane (2)

• Executed when an event (possibly more) is (are) triggered:

— HTTP Request Event - HTTP_REQ

— HTTP Response Event - HTTP_RESP

— HTTP Response Failed - RESP_FAILED

VMware Confidential - Internal Only 249

6-76 Supported Events
• HTTP_REQ event:

— Event triggers when the request line and all the headers of the HTTP request have
been parsed successfully, but before any potential POST body has been received

• HTTP_RESP event:

— Event triggers when the response status line and all headers of the HTTP response
have been parsed successfully, but before the response body has been received

• RESP_FAILED event:

— Event triggers when any error/timeout happens before a valid response header can be
received from the server and forwarded to the client

— There are only 3 HTTP functions which can be invoked from the RESP_FAILED event:
[Link](), [Link](), [Link].internal_status()

— Examples: TCP/SSL connection/handshake to back-end server fails, Request

proxy/send to back-end server times out, No Response or partial Response Headers or
Bad Response Headers from server, Server resets connection while Avi is waiting for
the back-end server to respond

[Link]

250
VMware Confidential - Internal Only
6-77 Lesson 6: DataScripts Constructs

6-78 DataScripts Constructs: Logic and

Operators
• A DataScript can reference any number of function or method calls

• Inspect and act on traffic flowing through a virtual service

• DataScripts rely on the Lua for the syntax and supported usage of arithmetic, relational, and
logical operators

• When evaluating strings, DataScript is case-sensitive, so “a” does not equal “A”.

[Link]

VMware Confidential - Internal Only 251

6-79 DataScripts Constructs: Variables

6-80 DataScripts Constructs: Functions (1)

• Functions are the commands to be executed by the DataScript

• Internally these are API calls the Lua interpreter is making to Avi to execute an action

• Functions are defined by the library they are scoped within, using following syntax:

— string.<function_name>()

— [Link].<function_name>()

— [Link].<function_name>()

— [Link].<function_name>()

— [Link].<function_name>()

[Link]

252
VMware Confidential - Internal Only
6-81 DataScripts Constructs: Functions (2)
• A function will always have an () at the end, flags may be embedded within the () if
applicable

• Functions can be nested:

— If [Link]([Link].get_header(“Server”)) == “APACHE” then

• Other Lua libraries may also be used – see [Link]

[Link]

VMware Confidential - Internal Only 253

6-82 DataScripts Constructs: Functions (3)

Documentation available per function, example:

[Link]

254
VMware Confidential - Internal Only
6-83 DataScripts: Attaching Objects
• If DataScript uses Pools, Pool Groups, IP Groups or Strings Groups they have to be
associated with Datascript

VMware Confidential - Internal Only 255

6-84 Lesson 7: Using DataScripts Examples

6-85 DataScript Example

[Link]

256
VMware Confidential - Internal Only
6-86 Public DataScript Examples Library (1)
Availability

• HTTP Host Switching

• Error! Hyperlink reference not valid.

• Error! Hyperlink reference not valid.

• HTTP URI Switching - Simple

• HTTP URI Switching - Advanced

• HTTP IP Switching

• JSessionID Persistence

• Error! Hyperlink reference not valid.

• Fall back to secondary pool or custom maintenance page

• Load Balancing during Maintenance Window

• Header Persistence using Akamai True-Client-IP header

• Error! Hyperlink reference not valid.

• Rewrite HTTP redirect response from 301 to 302

• Ratio based Load Balancing

• Error! Hyperlink reference not valid.

Data Script Library: [Link]

VMware Confidential - Internal Only 257

6-87 Public DataScript Examples Library (2)
Acceleration

• Client Cache Control Behavior

Troubleshooting

• Log SSL Version

• Log HTTP Headers

Misc

• Pseudo-random hex character generator

• Random Letter generator

258
VMware Confidential - Internal Only
6-88 Public DataScript Examples Library (3)
Security
• Error! Hyperlink reference not valid.

• Client Cert check

• Block SSLv3.0, TLSv1.0 or cipher suites that don't provide encryption

• Close Connections without Host

• Error! Hyperlink reference not valid.

• Header Insertion for Content Security

• Remove X-* and Server Headers from Response

• X-Client-IP allow requests from range of IPs

• Blocklist Specific URIs using String Group

• X-Forwarded-For Header Insert

• Disable HTTP Processing For Selective HTTP Methods

• Mitigate Microsoft vulnerability MS15-034 and CVE-2015-1635

• Redirect Location header validator

• Computing HMAC

• Controlling Bots

• Cross Origin Resource Sharing Implementation

• Validate String Characters in Cookie

VMware Confidential - Internal Only 259

VMware Confidential - Internal Only
Module 7
NSX Advanced Load Balancer Infrastructure
Architecture

7-2 Module Lessons

1. Architecture Overview

2. Control Plane

3. Data Plane

4. Tenants Overview

5. Service Engine Group Settings

6. High Availability Modes

7. Service Engine Failure and Self-Healing

8. Service Engine Groups: Networks and Routing

9. Service Engine As a Router

10. Scale-Out Mechanisms: L2 Versus L3

11. Upgrade 2.0

VMware Confidential - Internal Only 261

7-3 Lesson 1: Architecture Overview

7-4 Deployment Process

Mainly to get across this is software only and that it requires somewhere to host it:

• To introduce the idea of automation - let the tools take the strain

NOTE: We are going to rely on the automation piece when we describe the cloud connector so
we talk generic.

262
VMware Confidential - Internal Only
7-5 High-Level Component Parts

Control Plane - contains Avi controllers - this is the point of admin / maintenance / reporting of
the Avi infrastructure. It requires a virtualization infrastructure to run, eg VMware, AWS etc.

Data Plane - contains Avi Service Engines - this is analogous for the ports in the physical
appliances and performs the data / packet processing. Also requires a virtualization
infrastructure to run. It may or may not be the same infrastructure as is hosting the controllers.
EG: We could run controllers in VMware and control Service Engines hosted in AWS etc.
depends on design / where the load is and so on.

NOTE: Bare metal operation still execute as a Docker container.

Introduce:
• The controller

• The service engine

VMware Confidential - Internal Only 263

7-6 Very High-Level Component Parts

264
VMware Confidential - Internal Only
7-7 Data Plane Overview and Automation
• Controller is configured to communicate with chosen virtualization API endpoint. This is a
Cloud Connector

• Via Cloud Connector, Controller instructs virtualization infrastructure to deploy the data
plane, the service engines

• A group of service engines is a Service Engine Group

• The service engines handle the application traffic

This slide hopes to do several things:

• Enforce software only with Control and Data plane separation

• Enforce automation idea, Avi implements this idea heavily

• Introduce the ‘Cloud Connector’ in a very generic way

• Introduce the ‘Service Engine Group’

NOTE: We have to be very careful with language here, cloud might be our term for an object
definition but it means something completely different to a user, I think ‘Cloud Connector’ fits
and is the term we should use in the training. Eg, Cloud == AWS && AWS != Actual Avi object.

VMware Confidential - Internal Only 265

7-8 Lesson 2: Control Plane

7-9 NSX Advanced Load Balancer Controller

(1)
Centralized Management:

• Only point of management for Service Engines

• Administered through GUI, CLI, REST API

• Configuration repository and source of truth

• License enforcement

Orchestration:

• Manage life cycle of Service Engines

• Create / delete / move

• HA / failover decisions

• Northbound API interface to automation tooling (Ansible, Terraform)

• Southbound API interface to cloud orchestrators (vCenter Server, AWS, Kubernetes)

Analytics Processing:

• Long-term storage and indexing of application logs

• Metrics aggregation from Service Engines

266
VMware Confidential - Internal Only
Main points:

• Make the controller something that can be related

• PoC / Prod deployments

• Just like all the other VMs

Make it a quiz, what image formats does VMware support / KVM etc?

7-10 NSX Advanced Load Balancer Controller

(2)
Delivered as software, depending on virtualization infrastructure

• OVA, QCOW2, AMI, and so on

Deployed using the tools available, although can be easily automated

• vSphere Client

• AWS Management Console

• OpenStack Heat template

Just like the appliances, NSX Advanced Load Balancer can be deployed with availability and
resilience in mind:

• POC deployments 1 controller, Production deployments 3 controllers.

Not what you are used to with appliances:

• Sizing matters

• Controllers are free (still require hosting though)

VMware Confidential - Internal Only 267

7-11 Controller Clustering
• Controller may be deployed as a standalone or redundant three-node cluster.

• High availability uses a Zookeeper-like model of a three-node cluster to maintain a quorum.

• All Controllers are active, sharding workloads.

• Management may be performed from any Controller in the cluster without knowledge of
which is leader.

• All intra-Avi communications are encrypted.

268
VMware Confidential - Internal Only
7-12 Controller High Availability (1)
Single Node Failure:

• No impact to data plane (the Service Engines) or management

Two Node Failure:

• The remaining Controller node will not take over as active without quorum (2 nodes)

• Mitigates split-brain issue with traditional A/A, such as if one Controller was not down but
merely lost connectivity to peers

VMware Confidential - Internal Only 269

7-13 Controller High Availability (2)
Three-Node Failure:

• No impact to data plane. Service Engines continue to run in headless mode until Controllers
are restored

• No configuration changes possible until Controllers are restored / redeployed

• Service Engines will buffer metrics and logs until Controllers are back. Buffer size depends
on disk allocation for SEs

270
VMware Confidential - Internal Only
7-14 Recover a Nonoperational Cluster
• When two of the three nodes within a cluster are permanently down and not-recoverable,
the other controller node in the cluster will be marked operationally down due to lack of a
cluster quorum

• Service Engines continue to operate in headless fashion

• To recover a cluster with configuration:

/opt/avi/scripts/recover_cluster.py
• To recover a cluster without configuration(essential factory reset):
/opt/avi/scripts/clean_cluster.py

VMware Confidential - Internal Only 271

7-15 Controller Cluster: One Versus Multiple
Considerations for Moving from One to Multiple Clusters

• More than 400 SEs

• More than 4000 VS

• Controller to Controller under 5 ms latency

• Controller to SE under 40 ms latency

— Heartbeat timing

• Need different authentication sources, such as OpenStack Keystone and LDAP

• Reduce blast radius, such as Lab versus Production

272
VMware Confidential - Internal Only
7-16 Controller Process Sharding (1)

7-17 Controller Process Sharding (2)

• Each virtual service is hashed to a Controller to divide the workload.

• Scales workloads well with large numbers of virtual services.

VMware Confidential - Internal Only 273

7-18 Controller Sizing (1)
General:

• Undersizing Controller resources is a common issue that unfolds as more applications are
brought online.

• The most common symptom of undersized Controllers is general slowness, through GUI,
API, and so on.

• If the slowness gets extreme, some internal watchdog timers may elapse, triggering
events/alerts.

Storage:

• SSD is preferred over disk, ~3x faster writes for logging

• Min 128 GB storage. More storage equals more logs

CPU or Memory:

• More CPU and memory allows greater number of SEs and VSs

• Recommended to have 2-3x the memory of the CPUs

Further Reading

• Controller Sizing

7-19 Controller Sizing (2)

CPU 8 16 24

Memory (GB) 24 32 48

VS Scale 0 through 200 200 through 1,000 1,000 through 4,000

SE Scale 0 through 100 100 through 200 200 through 600

Base processes (GB) 15 20 24

Log analytics (GB) 9 12 24

274
VMware Confidential - Internal Only
7-20 Controller Operations
General:

• RPC infrastructure built on top of a reliable message queue

• All communication secure and encrypted through SSH

• Typically communication performed through management network

Controller to SE Initiated:

• Heartbeat message to validate Service Engine status

• Push configuration changes

SE to Controller Initiated:

• Metrics and logs

• Object status change (such as app server marked down)

• Database backup (SSL, persistence, DataScript tables)

Further Reading
• Ports used for Avi Vantage Communication

• Controller to SE Trust Establishment

• Architectural Overview

VMware Confidential - Internal Only 275

7-21 Lesson 3: Data Plane

7-22 Service Engine

Data plane processing:

• Local load balancing

• Global load balancing

• Web application firewall

• Application analytics and logs

Controller-managed resource:

• All management is performed from Controllers

• Configuration (including SSL cert/keys) stored in memory only

• Consumes license which is managed at the Controller

High availability:

• Standalone

• Active-standby

• Active-active

276
VMware Confidential - Internal Only
7-23 Service Engine NIC Architecture (1)
Management NIC:

• A Service Engine always has one management NIC, which is used to communicate with the
Controller

• Management NIC is ‘owned’ by Linux

Data plane NICs:

• A Service Engine may have one or more data plane NICs

• Data plane NICs are ‘owned’ by the SE, not Linux

• TCPdump ran from Linux will not show traffic for data NICs

General:
• Management and data may be on the same network and same NIC (inband management).

• NICs may be bonded for high availability and throughput.

VMware Confidential - Internal Only 277

7-24 Service Engine NIC Architecture (2)
NIC Types

PCAP:

• Packets are copy/queued from physical NIC through Linux to Avi

• Used in Linux bare metal with non-DPDK compatible NICs

• Limited performance

Virtual NIC:

• VMware, OpenStack, Azure, AWS, GCP, and virtual machine deployments

• Performance limited on the PPS the hypervisor can transfer

Consider the following different use cases for SE group

• Separate SE-group for critical Apps vs non critical Apps

• Separate SE-group for different departments

• Separate SE-group for Production vs test Apps

• Dedicated resources for an App

278
VMware Confidential - Internal Only
7-25 Service Engine NIC Architecture (3)
NIC Types

SR-IOV:

• NSX ALB bypasses Docker and the host Linux to attach directly to physical NIC

• Cisco CSP, VMware, AWS, and Microsoft Azure with Enhanced Networking

• Best performance

DPDK:

• NSX ALB bypasses Docker and the host Linux to attach directly to physical NIC

• VMware, Linux bare metal

• Best performance

VMware Confidential - Internal Only 279

7-26 Service Engine CPU Architecture (1)
1 Core SE:

• The single CPU core of the SE picks up packets off the NIC and performs normal load-
balancing service

2 to 4 Core SE:

• A CPU core is designated as a shared dispatcher for a NIC

• The dispatcher distributes packets from the NIC to the memory queues of all cores in the
SE

• The dispatcher core handles some load-balancing duties

5+ Core SE:

• The dispatcher core is dedicated full time to dispatching

7-27 Service Engine CPU Architecture (2)

Scale:

• A dispatcher is assigned per data plane NIC

• A typical CPU core can dispatch about 1.5 m PPS

• With DPDK’s Receive Side Scaling (RSS), SE can support multiple dispatchers per NIC for
higher PPS

• Take dispatching into account when estimating SE performance

280
VMware Confidential - Internal Only
7-28 Service Engine CPU Architecture (3)
Hyperthreading:

• Scaling horizontally across CPU cores works well

• Hyperthreaded cores provide 50% to 75% the performance of physical cores.

• NSX ALB cannot determine a real from a hyperthreaded core.

• The license does not distinguish between real and HT core – they count the same.

• Hyperthreading options vary in every public cloud.

VMware Confidential - Internal Only 281

7-29 Lesson 4: Tenants Overview

7-30 About Tenants (1)

• Tenants provide management isolation: Tenant-A cannot see Tenant-B config

• Tenants may be mapped to SEs in either:

— Provider mode: SEs are shared across tenants

— Tenant mode: Data plane isolation with dedicated SEs per tenant

• Tenants may span multiple clouds

• A user may have access to one or more tenants, and may have a different role in each

• A Controller may manage 2000 tenants

• Best Practice: Tenants and VRFs should be determined at time of Avi deployment

In addition to Provider / Tenant option, mixed states are possible.

282
VMware Confidential - Internal Only
7-31 About Tenants (2)

In addition to Provider / Tenant option, mixed states are possible.

VMware Confidential - Internal Only 283

7-32 Lesson 5: Service Engine Group
Settings

7-33 Introduction to Service Engine Groups

Templates:

• SE Groups contain sizing, scaling, placement, and HA properties.

• A SE is created from the SE Group properties.

• SE Group options vary based on the cloud / ecosystem.

Folders:

• An SE is always a member of the group it was created within.

• Each SE group is an isolation domain.

• Apps may gracefully migrate, scale, or failover across SEs in the group.

• Client session data automatically replicated to other SEs in the group.

— Persistence tables

— SSL session/tickets

— DataScript variables

284
VMware Confidential - Internal Only
• SEs in write access mode deployments cannot be migrated to new SE groups. Instead, the
old SE is deleted and a new one is created

• SEs in read access and no access mode deployments can be migrated to new SE groups,
provided all VSs deployed on the SE are disabled

• An SE cannot scale out across or fail over to an SE which is in a different SE group, even if
both SEs share physical host or network properties. Different applications can thus receive
guaranteed data plane isolation when deployed on different SE groups

VMware Confidential - Internal Only 285

7-34 Configuring Service Engine Groups:
VMware Cloud
• High Availability Mode

• VS Placement across SEs

• Scale per Virtual Service (on Advanced Tab)

• Resource definition for each SE spun up in a particular SE Group (Varies based on

ecosystem)

Consider the following different use cases for SE group.

• Separate SE-group for critical Apps vs noncritical Apps

• Separate SE-group for different departments

• Separate SE-group for Production vs test Apps

• Dedicated resources for an App

286
VMware Confidential - Internal Only
7-35 Lesson 6: High Availability Modes

7-36 High Availability Modes (1)

• Vantage provides three high availability modes.

• Two elastic HA modes which combine scale out performance as well as high availability:
N+M mode(default) and active-active mode.

• A legacy HA active-standby mode.

• Elastic HA N+M mode is recommended for applications where the following conditions
apply: performance required by an application can be delivered by a fraction of one SE’s
capacity, hence each VS is typically placed on a single SE and applications can tolerate brief
outages amounting to the duration of placing the VS on the buffer SE and plumbing the
network

• Elastic HA active-active mode is more suitable for mission-critical applications which need to
run with no interruptions.

7-37 High Availability Modes (2)

VMware Confidential - Internal Only 287

7-38 High Availability Modes (3)
Legacy active-standby:

• VS is active on one SE, standby on another

• No VS scale-out support

• Primarily for default gateway

• Preserve client IP support

• Fastest fail over, but half of SE resources are idle

High Availability Mode A/S

SE failure detection 0

Controller determines SE to fail over to -

Controller creates new SE -

Copy VS configuration to new SE -

Configure vNIC on new SE -

Move VIP through GARP or cloud API 0

288
VMware Confidential - Internal Only
7-39 High Availability Modes (4)
Elastic N + M [default mode]:

• All SEs are active

• N = number of SEs a new VS is scaled across

• M = the buffer, or number of failures the group can sustain

• SE failover decision determined at time of failure

• Longer application recovery time

Elastic Active-Active [best practice]:

• All SEs are active

• VS must be scaled across at least 2 SEs

• SE failover decision pre-determined

• Faster failover, potentially greater SE resource requirement

High Availability Mode A/A N+M

SE failure detection 0 0

Controller determines SE to fail over to 0

Copy VS configuration to new SE 0

Move IP through GARP or cloud API 0 0

VMware Confidential - Internal Only 289

7-40 High Availability Modes (5)

290
VMware Confidential - Internal Only
7-41 High Availability Modes (6)
High Availability Mode A/A N+M

SE failure detection 0 0

Controller determines SE to fail over to 0

Copy VS configuration to new SE 0

Move IP through GARP or cloud API 0 0

VMware Confidential - Internal Only 291

7-42 Elastic HA N+M Mode
Default mode for Elastic HA. In this mode, each VS is typically placed on just one SE.

• N represents the maximum number of SEs required to place the VSs in the SE Group.

• M is the aggregate capacity of the SE Group to handle up to M SE failures.

292
VMware Confidential - Internal Only
7-43 Elastic HA N+M Example
• Twenty VSs are configured in an SE Group.

• Max VS per SE is set to 8.

• N = 20/8, that is, 3 SEs are required in the SE Group to accommodate the 20 VSs.

• With buffer M configured as 1, a total of 4 SEs are required in the group

• In N+M mode, Controller ensures enough aggregate capacity to handle ’M’ SE failures

VMware Confidential - Internal Only 293

7-44 Elastic HA N+M Mode Recommendation
• Elastic HA N+M mode is typically recommended under the following conditions:

— The SE performance required by any application can be delivered by a fraction of one

SE's capacity, hence each VS is typically placed on a single SE

— Applications can tolerate brief outages, albeit no longer than it takes to place a VS on
an existing SE and plumb its network connections

• Pre-existence of buffer SE capacity speeds replacement of virtual services affected by a

failure

7-45 Elastic HA Active-Active Mode (1)

• Controller places each VS on more than one SE, as specified under Minimum scale per
Virtual Service parameter

7-46 Elastic HA Active-Active Mode (2)

• Default minimum is 2

• Minimum: Setting to the minimum above 1 ensures that every VS is scaled out across
multiple SEs regardless of the capacity requirements. Increases potential capacity of the
application and minimizes impact after a failure

• Maximum: Sets the maximum number of SEs across which a VS may be scaled, max limit
being 4 for L2 scale-out

294
VMware Confidential - Internal Only
7-47 Elastic HA Active-Active Example
• Five VSs are configured in an SE Group

• Max VS per SE is set to 3, max number of SEs is set to 6

• Min scale per VS is set to 2 and max scale per VS is set to 4

VMware Confidential - Internal Only 295

7-48 Elastic HA Active-Active Mode
Recommendation
Elastic HA active-active mode is typically recommended for mission-critical applications where
the virtual services must continue without interruption during the recovery period

7-49 Legacy HA Active-Standby Mode

• Two SEs are configured. Active SE carries all the traffic for the VSs placed on it.

• The other SE in the pair acts as Standby for the VSs and does not carry any active traffic

• Standby SE during takeover also assumes ownership of floating addresses [VIPs/SNAT-IP]

• Useful for replicating a hardware appliance-based solution

296
VMware Confidential - Internal Only
7-50 Compact Versus Distributed Placement
• With this feature enabled, NSX ALB uses minimum number of SEs required to place the
configured VSs

• With compact placement off that is, distributed placement turned on, Vantage will use as
many SEs as required up to the limit allowed by Max Number of Service Engines configured

• By default, compact placement is ON for N+M mode and distributed placement is ON for
active-active mode

Discuss the use case for compact vs distributed.

When compact placement is ON (label G in figure 8b), Avi Vantage uses the minimum number of
SEs required. When distributed placement is ON, Avi Vantage will use as many SEs as required,
up to the limit allowed by Max Number of Service Engines (labeled E in the figures). By default,
compact placement is ON for elastic HA N+M mode while distributed placement is ON for elastic
HA active-active mode.

VMware Confidential - Internal Only 297

7-51 Impact of Placement on N+M Mode
• Max Number of Service Engines is set to 8 for the N+M SE Group

• Eight Virtual Services are created in sequence

• After VS1 is placed, SE2 is deployed because M=1 (buffer SE)

• When VS2 requires placement, controller assigns it to SE2 to make best use of all running
SEs

• From this point on, the behavior diverges

— Compact Placement ON: Subsequent placement of VS3 through VS8 is performed on

the existing SEs, SE1 and SE2.

— Distributed Placement ON: Subsequent placement of VS3 and VS4 result in scaling the
SE group to its maximum 4(illustrating the preference of performance over resources).
Having reached the maximum of 4 SEs, Virtual Services VS5 through VS8 are placed on
pre-existing least-loaded SEs

298
VMware Confidential - Internal Only
7-52 Impact of Placement on Active-Active
Mode
• Distributed Placement is ON by default for active-active mode

• When SE3 encountered a failure, Controller delayed the placement of VS2 and VS3 until a
new Service Engine SE7 was spun up

• None of the other existing SEs were chosen to place VS2 and VS2 to fulfill the distributed
placement logic that is, Vantage will use as many SEs as required up to the limit allowed by
Max Number of Service Engines configured

7-53 Use Cases

1. What is the HA mode of a Virtual Service?

2. What are the SEs handling traffic for a Virtual Service?

VMware Confidential - Internal Only 299

7-54 Determining SEs That Are Handling
Traffic for a VS (1)
• From the Virtual Service dashboard, navigate to the VS. Point to the VS name for the quick
info pop-up which displays the SEs hosting the VS

• Click one of the SEs to get to the Service Engine page

• Point to the SE name for the SE quick info pop-up

7-55 Determining SEs That Are Handling

Traffic for a VS (2)

300
VMware Confidential - Internal Only
7-56 Lesson 7: Service Engine Failure and
Self-Healing

7-57 Controller to SE Failure Detection

Method
• Controller periodically sends heartbeat messages to all SEs in all groups once every 10
seconds

• If no response is obtained from a specific SE for six consecutive heartbeat messages, the
controller concludes that the SE down and generates a SE_DOWN event

• When the SE is declared down, the Controller moves all Virtual Services from that SE to
new SEs.

VMware Confidential - Internal Only 301

7-58 Failure Detection Algorithm (1)
The following are the sequence of actions taken to detect a SE failure:

• Consider a SE group on which a Virtual Service has been scaled out

• Primary SE sends periodic heartbeat messages to the secondary SEs. Period is determined
by the mode used(Aggressive or Standard)

• A string of consecutive response failures to the heartbeats from a particular SE indicates

that a given SE could be down

• When the Primary SE suspects that a SE may be down, it informs the Controller which then
sends echo messages to the suspect SE

• Consecutive response failures to these echo messages from the SE indicates to the
Controller that the SE is down

• Timers used for Standard and Aggressive modes of failure detection are above

• Standard mode is recommended. Aggressive mode of failure detection can only be enabled
from the CLI

7-59 Failure Detection Algorithm (2)

Data-path (primary Avi- Control -path (Avi Total time to
SE-to-secondary-Avi-SE) Controller-to-suspect- declare Avi SE
failed heartbeats Avi-SE) failed echoes failed

Standard 10 consecutive every 100 4 consecutive every 2 sec9 seconds

ms

Aggressive 5 consecutive every 100 2 consecutive every 500 1.5 seconds

ms ms

302
VMware Confidential - Internal Only
7-60 Elastic HA N+M Failure

VMware Confidential - Internal Only 303

7-61 Elastic HA Active-Active Failure

304
VMware Confidential - Internal Only
7-62 Legacy HA Active-Standby Failure

VMware Confidential - Internal Only 305

7-63 Lesson 8: Service Engine Groups:
Networks and Routing

7-64 SE Group Network Settings

• Networks tab presents the list of discovered and manually configured networks within the
Cloud

• Individual networks can be configured for DHCP or for Static IP address allocation

• The Network Settings can be edited by clicking the pencil icon on the far right side of the
page

• Name: Name of the Network

• Discovered Subnets: Auto-discovered subnets via the Virtualization Orchestrator. This field
maybe None, Excluded or a list of one of more IP networks

• Configured Subnets: Subnets manually configured on Vantage, which are used to allocate IP
addresses to the SE vNICS

• Discuss this only if the ecosystem is VMware

• Network in VMware comes into the picture as follows: When VMware cloud is configured on
the controller, controller discovers all VMware objects like port-groups, ESXi hosts. Data
stores, VMs, and their associated IPs. A VM’s IP and the associated port-group combination
is what is referenced as a discovered subnet on Avi. In this discovered subnet, we can
configure specific IPs to be used for SE vNIC configuration for more granular control

306
VMware Confidential - Internal Only
7-65 SE Group Static Route
• Static Routes may be used to configure next hop path for routed traffic to the back-end
servers

• A static route can also be set as the default-gateway

• Available under Infrastructure > Routing > Static Route tab

• Prefix: Any egress traffic from the SEs matching this IP subnet will be sent to the IP address
of the next hop gateway. A Prefix set to Default Gateway indicates that all traffic not
matching any other static route prefix will be forwarded to the next hop for the Default
Gateway

• Next Hop: Next hop or gateway address to use when routing traffic to the IP subnet
specified by the prefix

This slide will be discussed if and when needed, based on the customer’s deployment. Only
static route is discussed here. Bgp is another option in a single-arm scenario for SE accessibility
to back-end servers. This is advanced and hence has been skipped here.

VMware Confidential - Internal Only 307

7-66 Auto Gateway
Auto Gateway: Return packets are sent to the source MAC address that is associated with the
connection instead of returning client data through the default gateway of Avi Vantage.

• If SE has the wrong default gateway, no configured gateway, or multiple gateways, client-
initiated return traffic will still flow correctly

• The SE default gateway will still be used for management and outbound-initiated traffic

[Link]
services/create-virtual-service/

308
VMware Confidential - Internal Only
7-67 Lesson 9: Service Engine As a Router

7-68 Service Engine As a Router (1)

• SE is used for routing back-end server traffic. Required with ’Preserve client IP’ enabled
under the Application Profile

• Back-end servers receive traffic with the source IP set to the IP of originating clients

• SE IP towards the backend needs to be configured as the default-gateway for the servers
to route the traffic back to the clients through the SE

• Works only with Legacy Active-Standby HA

• Back-end servers must be on the same subnet as the SE

7-69 Service Engine As a Router (2)

Avi SE can be used for routing the traffic of server networks.

If Preserve Client IP option is required.

• IP routing is supported on two-armed, no-access configurations of Linux server clouds and

VMware clouds, and conditionally supported on CSP. On CSP, it is supported when the
interfaces attached to the SE instances are configured in SR-IOV mode

• Avi Vantage supports IP routing for VMware Cloud (with the following conditions) back-end
servers receive traffic with the source IP set to the IP of originating clients

— One arm (in the two-arm mode deployment) must be placed in the back-end network.
For this network, SE acts as the default gateway

— The other arm is placed in the desired front-end network

• Only HA (active-standby)

IP routing is supported on the following:

• IP routing is enabled per VRF

• Only DPDK-based SEs

VMware Confidential - Internal Only 309

7-70 Default Gateway: SE As a Router
Example

310
VMware Confidential - Internal Only
7-71 Service Engine As a Router (1)

7-72 Service Engine As a Router (2)

Enable IP routing on all SEs in the SE group:

: > configure serviceenginegroup Default-Group

: serviceenginegroup> enable_rounting
Overwriting the previously entered value for enable_routing
: serviceenginegroup> save

VMware Confidential - Internal Only 311

7-73 Lesson 10: Scale-Out Mechanisms: L2
Versus L3

7-74 L2 SE Native Scaling

• The primary SE ARPs for the VIP address.

• Traffic increases beyond the capacity of a single SE.

• Controller brings new load balancers (SEs) online.

• The primary SE delegates some traffic to new SEs by forwarding some connections (L2
switched) to the MAC addresses of the other SEs.

• Each SE takes a portion of the load. With SNAT, servers return traffic to the source SE
[Link] forward response traffic directly back to clients.

312
VMware Confidential - Internal Only
7-75 Native Scale-out

VMware Confidential - Internal Only 313

7-76 Network/SDN-Based Scale-Out

ECMP functionality is part of the SDN’s distributed routing service

314
VMware Confidential - Internal Only
7-77 L3 SE ECMP Scaling
Scale Service Engines via Upstream Router:

• All SEs advertise the VIP to BGP via Route Health Injection

• Router hashes client flows across SEs

• ECMP mode enables scaling across 2 to 64 Service Engines

• With SNAT, servers return traffic to the source SE MAC address

• SEs send response traffic directly to clients

Failure Mitigation:

• BFD may be enabled to ensure faster detection of an SE failure

• Persistence and SSL connections are mirrored to ensure a graceful and automatic recovery
for a router hash redistribution

• SEs will forward incorrectly hashed flows to the proper SE using IPIP tunneling

1. Discuss the distributed DB across the SEs that are maintaining shared LB state for each VS -
- IP Persistence DB, SSL Session DB and son on. That is if any app state that is needed
across multiple TCP connections

2. Network based scale-out - If network has intelligence to spray traffic across the SEs

• BGP for on-prem use-cases

• GCP we program the GCP's SDN to do ECMP

• Azure we program their SDN to do ECMP

• AWS we use DNS based scale-out

• Openstack/Contrail - use SDN to do ECMP

VMware Confidential - Internal Only 315

7-78 BGP Support for Elastic HA (1)
• BGP is supported in legacy (active-standby) as well as elastic (active-active and N+M) high
availability modes.

• Using L3 scale-out, a VS can be placed on up to 64 SEs within a SE Group

• BGP is supported only with VMware and Linux Server Cloud, OpenShift, and Kubernetes

7-79 BGP Support for Elastic HA (2)

Navigate to Infrastructure > Routing and select the BGP Peering tab to configure BGP

316
VMware Confidential - Internal Only
7-80 BGP Support for Elastic HA (3)
Enable ’Advertise VIP via BGP’ under Advanced tab of the VS

VMware Confidential - Internal Only 317

7-81 DNS-Based Scale-Out

318
VMware Confidential - Internal Only
7-82 Lesson 11: Upgrade 2.0

7-83 Upgrade Preparation

Software Versioning:

• Software versioning: [year] . [major release] . [minor release / bug fixes] such as v18.2.5

• Avi generally has about two major releases a year, with ~eight minor releases in between

• Avi has an active beta program, generally tied to specific features / functionality

Upgrade Preparation:

• Make an external backup of the configuration

• Check disk capacity on Controllers

— Recommended to have at least 30% free on partitions

— Ask support if there is insufficient capacity, or delete large logs

• Upgrades across disparate versions may require intermediate upgrades

— v15.1 > v18.1 first requires upgrading to v16.x

• The upgrade file is about 2.5 GB: Download the file to the leader node prior to the activity

Further Reading

• Release Notes

VMware Confidential - Internal Only 319

7-84 Upgrade
General:

• The upgrade must be initiated from the leader node.

• The upgrade file only needs to be copied to the leader node.

• View the status of the upgrade via:

— GUI: Administration > System > Upgrade

— CLI: show upgrade status

— API: [Link]

• Choose an upgrade file:

— controller_docker.tgz (Linux bare metal)

— [Link] (all other Controller deployments)
— The upgrade file type depends on where the Controllers are deployed, not the SE in a
cloud.

— Upgrade files are available from [Link]/portal.

Further Reading

• Avi Customer Portal

• Upgrade Vantage Software

320
VMware Confidential - Internal Only
7-85 Controller Upgrade
• Controller makes a local snapshot of the running config

• Upgrade file is replicated to follower Controllers

• Configuration is locked

• All Controllers in the cluster initiate the upgrade simultaneously

• All Controllers in the cluster reboot

• After reboot, Controllers attempt to contact each other to validate successful upgrade

— If contact is not reestablished in 5 minutes, Controllers will timeout and reboot back into
prior version partition, effectively rolling back

• Once connection established, Controllers rebuild quorum

— This process involves resynching and validation of databases, configurations, and so on

— Time to establish quorum varies, but may take 5 minutes

• Controller upgrade complete

• Typical Controller 3 node cluster may take ~10-20 minutes

VMware Confidential - Internal Only 321

7-86 Service Engine Upgrade (Before 18.2.6)
Service Engine Upgrade:

• No configuration changes allowed

• SEs within an SE Group are upgraded one at a time

• Multiple SE Groups are upgraded in parallel

Details:

• For active-active VS, client traffic is gracefully migrated to alternate SEs

— By default, graceful client conn bleed off is set to 1 minute. Increasing this value
increases total time to upgrade

• Controller installs new SE image into a new partition on the same SE

— Controller to SE latency, plus SSD versus hard disk determines SE upgrade time

• SE boots into the new partition running the new version

• Typical SE upgrade takes ~3-5 minutes

322
VMware Confidential - Internal Only
7-87 Service Engine Upgrade (After 18.2.6)
Service Engine Upgrade:

• Configuration changes allowed

• SEs within an SE Group are upgraded one at a time

• SE groups can be selectively upgraded

Details:

• For active-active VS, client traffic is gracefully migrated to alternate SEs

— By default, graceful client conn bleed off is set to 1 minute. Increasing this value
increases total time to upgrade

• Controller installs new SE image into a new partition on the same SE

— Controller to SE latency, plus SSD versus hard disk determines SE upgrade time

• SE boots into the new partition running the new version

• Typical SE upgrade takes ~3-5 minutes

VMware Confidential - Internal Only 323

7-88 Service Disruption
• Upgrades are nondisruptive for virtual services running in:

— Elastic Active-Active mode

— Elastic N+M mode for VS scaled out to two or more SEs

— Legacy Active-Standby mode

• Upgrades are disruptive for virtual services running in:

— Elastic N+M mode for VS are not pre-scaled out

324
VMware Confidential - Internal Only
7-89 Rollback (Before 18.2.6)
Rollback:

• Should an issue occur during the Controller upgrade, the entire Avi deployment
automatically rolls back to the previous version

• Controllers and SEs will reboot into the previous partition

• Controllers will rebuild quorum, which may take several minutes

• Should an issue occur during SE upgrade, the Controller will retry the SE a second time. If
that fails, it will create SEs and leave the un-upgraded SEs in an unused state

• Rollback may always be initiated later by an admin after a successful upgrade

• Rollback is disruptive to existing client traffic, as all SEs are rebooted immediately.

7-90 Rollback (After 18.2.6)

Rollback:

• Controller Rollback will transition to the previous major version of software

• Selective ability to rollback the SE groups

• Rollback of the SE Group will be to the previous version

VMware Confidential - Internal Only 325

7-91 Patch Versus Upgrade
• Patching enables customer to upgrade portions of Avi, rather than entire deployment

— Controller

— SEs (specifically, SEs within an SE Group)

— GUI

• Typically for smaller bug fixes or security fixes

• Different SE Groups may run different patch levels (but must still run same version level)

• Rebooted SEs within non-upgraded SE Groups will automatically pick up the patch on boot

326
VMware Confidential - Internal Only
Module 8
Introduction to Cloud Connector

8-2 Cloud Connector Configuration Concepts

Cloud Connector Setup

A cloud connector is an object used to integrate with cloud providers such as AWS, Google,
Microsoft Azure, and VMware through their respective API endpoints.

• Avi Vantage cloud connector is used to talk to APIs of other cloud ecosystems

• Avi can integrate and operate with each ecosystem using many features that exist in each
ecosystem which start with the cloud connector

• As long as the avi controller has reachability to the ecosystem, the cloud connector can
send commands to the API of the cloud provider communication is destined for

• The amount of control Avi Vantage Controller has on your cloud is configured when
deploying the cloud

VMware Confidential - Internal Only 327

8-3 About Clouds (1)
Write Access Cloud:

• If no SEs have reachability, the Controller may modify the networking properties of an
existing SE.

• If no SEs have capacity, the Controller may spin up new SEs.

No Access Cloud:

• Administrators are responsible for the life cycle of Service Engines.

• Manually set networking properties or create SEs if conditions are not met.

• Controller may still place VS if an SE has capacity and reachability.

8-4 About Clouds (2)

Folder:

• A cloud is a container for objects that are deployed within the cloud.

• SE Groups are always scoped within a cloud.

• Controllers are not within clouds, regardless where they are installed.

APIs:

• An API connection between the Controller and the orchestrator for that cloud.

328
VMware Confidential - Internal Only
8-5 Additional Clouds
Additional Clouds with Limited Support:

• Cisco CSP

• Nutanix Acropolis

• Hyper-V

• IBM SoftLayer

• Oracle Cloud

• Rancher

Deprecated Support:

• Mesos

• Docker Swarm

• Docker UCP

• CloudStack

VMware Confidential - Internal Only 329

8-6 Choosing the Right Cloud
• NSX Advanced Load Balancer is agnostic to which cloud customers choose.

• Network admins often iterate through a few on-prem clouds.

• The illustration is a common journey that admins might go through to choose the “right”
environment.

330
VMware Confidential - Internal Only
8-7 SE Image Types (1)
1. The Controller creates SE images.

2. First-time image creation for a new cloud might take 8 mins to 15 mins.

3. SE image types are agnostic of the Controller cloud location.

VMware Confidential - Internal Only 331

8-8 SE Image Types (2)
Cloud SE Image SE Type

Bare Metal tqz Container

Kubernetes tqz Container

OpenShift tqz Container

Mesos tqz Container

VMware ova VM

OpenStack qcow2 VM

GCP qcow2 VM

Azure vhd VM

AWS ami VM

332
VMware Confidential - Internal Only
8-9 Public Cloud (1)
Inconsistency within a Cloud:

• You are renting gear from the cloud provider and have limited control.

• Public cloud providers are not always consistent across regions.

• Some regions only support 2 AZs, others support 3.

• Some regions use VMs with hyperthreading, others do not.

• VMs may be migrated with vSphere vMotion across hosts without notice:

— Potentially disruptive to the SE

• Latency is highly variable.

• Storage tends to be slow.

Requesting larger storage might stripe storage across multiple devices. Performance
improves up to 10x performance.

VMware Confidential - Internal Only 333

8-10 Public Cloud (2)
Unlimited Scale:

• Many customers choose public clouds for this infinite server capacity.

• NSX ALB dataplane scales elastically with demand.

• All public cloud SE scaling methods are effectively the same as on-prem BGP/ECMP.

• Packet per second varies greatly by VM flavor, plan on testing.

• Set realistic scaling restraints for NSX ALB dataplane:

— A DoS attack can result in many additional SEs spun up.

Cloud Scale Across SEs Pingable VIP Hyperthreading Multi-AZ

GCP Native ECMP Yes Yes, most regions No

Azure ALB (ECMP) No No Some Regions

AWS Elastic IP (L2) Yes Yes, can be disabled Yes

334
VMware Confidential - Internal Only
8-11 Public Cloud (3)
Networking:

• No access to the MAC layer

• L3 tunnel for SE to SE packet transfer

• AWS: Larger flavors may support more IP address (more VIPs)

• GCP / Azure: All IPs are /32, traffic is routed

Supports 1 NIC, with one routed self-IP address

VIPs are routed to the self-IP address

Further Reading:

• AWS IP Per Instance Flavor

AWS Flavor Max NIC IP/NIC

[Link] 2 4

[Link] 3 6

[Link] 2 10

[Link] 3 10

C5.4xlarge 8 30

1 NIC reserved for mgmt

1 IP / NIC reserved for self-IP

VMware Confidential - Internal Only 335

8-12 Public Cloud (4)
Sample Flavor Performance:

• AWS and Azure have both added enhanced networking, which is SR-IOV bypass of the
hypervisor.

This provides near 10x performance improvements on flavors support it.

AWS

SE Flavor HT Cores SSL Tput SSL TPS

[Link] 1 200 Mb 1.8k

[Link] 2 650 Mb 6.2k

[Link] 2 350 Mb 2.4k

[Link] 4 600 Mb 4.9k

[Link] 2 2.1 Gb 4k

[Link] 4 2.4 Gb 7.6k

8-13 Public Cloud (5)

AZURE

SE Flavor Cores SSL Tput SSL TPS

D2_V3 2 800 Mb 1.9k

F2s 2 900 Mb 3.3k

D4_V3 4 1.5 Gb 4.1k

F4s 4 140 Mb 5.6k

D8_V3 8 1.5 Gb 9.1k

F8s 8 270 Mb 6.6k

336
VMware Confidential - Internal Only
8-14 Cloud Connector Configuration Concepts
(1)
Cloud Display

• Default Cloud Is the default cloud created and cannot be deleted

• Ability to create multiple clouds and list will continue to grow just as shown above

• From this page can Create/Edit/Delete

• Depending on cloud, there are other options such as:

— rediscover environment: which will refresh the cloud connector process with the
specific cloud environment

— Convert to no access: converts the cloud to a No-Access cloud, similar to default-cloud

shown above

— Generate Token: this is to build SE’s manually under the specific cloud

8-15 Cloud Connector Configuration Concepts

(2)
Cloud Display

• Summary of Cloud configuration can be viewed from the Clouds tab.

VMware Confidential - Internal Only 337

• Useful if large number of clouds configured

• There is also a concept known as orchestration access modes:

— No-Access - Avi Vantage has no access to the ecosystem or components in it, so in

other words Service Engines and any configuration required is done manually (cloud
connector not used)

— Read-Only - Avi Vantage can acquire inforrmation about the ecosystem such as
networks and pull it in as objects bound to the cloud (Cloud connector in read only)

— Read/Write - Avi Vantage has access to deploy and configure service engines as per
traffic needs, and in some ecosystems leverage other features, for example, for AWS
when configuring a virtual service, configuring elastic IP will trigger Avi Vantage cloud
connector to request an EIP from AWS (Cloud Connector write mode)

• The example above shows the VMware cloud is configured with write access and as shown
a username is required, and if we were to edit the cloud a password is also needed, this
means a user with necessary permissions will need to be defined.

338
VMware Confidential - Internal Only
8-16 Cloud Connector Configuration Concepts
(3)
New Cloud Creation

• As of 18.2.5 building a new cloud will look like this but a list of what Avi Vantage supports
today can be found here: [Link]
ecosystem/

• Specifics on versions of OS for bare metal or of different cloud types can be found at the
link

• Avi Vantage has knobs that connect into various clouds and can auto populate details from
the ecosystem on Avi, and likewise Avi has the ability to deploy SEs or push configuration
to remote ecosystem that cloud-connector is plugged into

• Based on the type of cloud, further configuration will be exposed such as credentials, IP
assignments, as well as options for environment/ecosystem specific features
• Will go in more detail on the different types in the individual modules around the different
types

VMware Confidential - Internal Only 339

8-17 References
Supported Ecosystems: [Link]

Ports: [Link]
management-communication/

Access Modes: [Link]

340
VMware Confidential - Internal Only
Module 9
Installing, Configuring, and Managing NSX
Advanced Load Balancer in No-Access
Clouds

9-2 Module Lessons

1. Cloud Configuration

2. No-Access Service Engines

3. Linux Server Cloud

4. VMware No-Access

5. Advanced Features

VMware Confidential - Internal Only 341

9-3 Lesson 1: Cloud Configuration

9-4 No-Access Cloud

Service Engine Deployment:

• No SE life cycle automation provided by Controller:

— Deployment and life cycle of Service Engines are manual or through third-party
automation (Ansible, Terraform, and so on).

342
VMware Confidential - Internal Only
9-5 Lesson 2: No-Access Service Engines

9-6 Linux Bare Metal (1)

Scenario

• Potential for the best SE performance (when used with good hardware)

• Some additional complexity with the host Linux OS

• Reserve 1-2 CPU cores for Linux when using core-based license model

— Allocating all CPU for SE starves the host Linux and Docker, which still require CPU for
management data processing and other tasks

— Important when using in-band management (single NIC for mgmt. and data)

• Service Engines prefer the Overlay2 driver for Docker storage

• Support is limited to specific Linux flavors and kernels

VMware Confidential - Internal Only 343

9-7 Hardware Requirements
Dependencies / Recommendations

Software:

• Operating System Support is limited to specific versions of RHEL, CentOS, Ubuntu, and OEL

• Limitation also applies to specific kernel versions

• Docker 1.6.1 or greater

• Python or Ansible for deployment

Hardware:

• Specific NICs required for DPDK support

— Unsupported NICs might be non-DPDK libpcap

• CPU/RAM and HD requirements

Recommendations:

• Disable Hyperthreading in BIOS

• NTP Server on Host OS

344
VMware Confidential - Internal Only
9-8 Linux Bare Metal (2)
Scenario

Deployments

• 1 Host: Controller and SE may share the host Linux server

• 2 Hosts: One Controller on host1, one SE on host2

• 3 Hosts: Controller and SE per host

• Deploy Controllers using native environment image type when possible

— For example, using ova for a VMware deployment reduces the complexity with docker
and host Linux optimization, particularly for storage

• Avi provides Ansible playbook for the complete automation of the Linux cloud
• Avi has specific supported versions of Linux and kernels

— Disable automatic updates of host Linux

— Disable selinux

— Disable firewall

VMware Confidential - Internal Only 345

9-9 No Access (1)
Service Engine Docker Deploy

Step 1 – Download docker_install.[Link], transfer it to supported linux host with installed linux
software dependencies and extract contents

Step 2 – Execute avi_baremetal_setup.py script using options or interactive mode

Step 3 – NSX Advanced Load Balancer Service engine deployed and associated to NSX
Advanced Load Balancer Controller

346
VMware Confidential - Internal Only
9-10 No Access (2)
Service Engine Docker Deploy

• Linux versions and software dependencies configured

• Download docker_install.[Link] file for the specific version used and transfer to the server
endpoint

• Extract contents from docker_install.[Link] for manual service engine bare-metal

deployment

• Script to deploy service engines is avi_baremetal_setup.py

• Script will validate dependencies/requirements

• Can be run using options or in interactive mode

• Detailed information on the options can be viewed using –h (help) option

VMware Confidential - Internal Only 347

9-11 No-Access Cloud
Service Engine Docker Deploy

• Complete inputs as per interactive deployment

• Execute avi_baremetal.py script packaged with docker_install.tgz

• By default Service Engine built under Default-Cloud

348
VMware Confidential - Internal Only
9-12 Lesson 3: Linux Server Cloud

9-13 Linux Server Cloud (1)

Service Engine LSC Deployment

Step 1 – Linux host with installed Linux software dependencies to deploy Service Engines

Step 2 – SSH User Setup on target Linux host (username/password or SSH Key Value Pair)

Step 3 – Linux Server Cloud Connector configured

(Name, IPAM profile, DNS profile, static routing preferences, SSH User)

Step 4 – Service Engine deployed through Linux Server Cloud registration with Controllers,
ready for consumption

VMware Confidential - Internal Only 349

9-14 Linux Server Cloud (2)
Cloud Requirements

• SSH User required

— Password

— SSH Key

• User will need necessary permissions to deploy docker container

• User for permissions and sshkey or password configured to allow NSX Advanced Load
Balancer to log in and make necessary changes

350
VMware Confidential - Internal Only
9-15 Linux Server Cloud (3)
Cloud Deployment

• Ease of installation

• Administrator directs the Controller to manage Service Engines

• Docker containers for maximum efficiency

• Reduced resources consumption

VMware Confidential - Internal Only 351

9-16 Linux Server Cloud (4)
Service Engine Deployment

• Credentials to use to log into Linux Servers

• Can specify different partitions/paths for logs

• Server list to log into Linux Server and deploy SE

• DPDK can be enabled, but if not enabled libpcap will be used

• By default all cores and memory will be consumed, however it is recommended to have
certain number of cores and memory reserved for base Operating System

352
VMware Confidential - Internal Only
9-17 Linux Server Cloud (5)
Service Engine Deployment

Service Engine Deployment Validation

• NSX Advanced Load Balancer GUI via health and status

• Linux host done via Docker ps -a

VMware Confidential - Internal Only 353

9-18 Lesson 4: VMware No-Access

9-19 No Access (1)

Service Engine Deploy within VMware

Step 1 – No-Orchestrator Cloud Connector configured

Basic options (Name, IPAM profile, DNS profile, static routing preferences)

Step 2 – Download OVA

OVA template created/downloaded from Controller

Step 3 – Cloud secure tokens generated

Unique secure token for each Service Engine, one-hour expiration, tied to Cloud

Step 4 – Service Engine OVAs deployed through vCenter Server

Automatic registration with Controller, ready for consumption

354
VMware Confidential - Internal Only
9-20 No Access (2)
Service Engine Deploy within VMware

• Custom cloud created for deployment which can be referenced in token when using OVA
deploy method

VMware Confidential - Internal Only 355

9-21 No Access (3)
Service Engine Deploy within VMware

• OVA can be downloaded from Controller

• Token used to authenticate service engine to Controller

• Token identifies cloud to which service engine is to be associated

356
VMware Confidential - Internal Only
9-22 No Access (4)
Service Engine Deploy within VMware

• In vCenter Server, deploy OVF template

• Follow workflow for Service Engine deployment on VMware

• Destination Network of management is the interface for service engine to controller

communication

VMware Confidential - Internal Only 357

9-23 No Access (5)
Service Engine Deploy within VMware

• Custom attributes for service engines completed on the template page

• Authentication token acquired from controller

• Service Engine IP address and gateway configured for management network(if none
provided DHCP used)

• DNS Information also configured on this page

358
VMware Confidential - Internal Only
9-24 Lesson 5: Advanced Features

9-25 Advanced Features (1)

Advanced Features

• Usually Linux Server Cloud or Bare Metal deployments are driven by performance needs
and employ list of advanced features

— Certain data plane features are only available in DPDK mode:

— TSO

— GRO

— RSS

— Multiple Dispatchers

— Denylisting

— VLAN subinterfaces

— Port-Channels (to include support of VLAN subinterfaces)

— MAC Masquerade

— VRF

VMware Confidential - Internal Only 359

9-26 TSO, GRO, and RSS (1)
Advanced Features

TCP Segmentation Offload (TSO) – reduces CPU overhead of TCP/IP on egress of fast
networks

• Completes the segmentation of packets at the Network Interface Card(NIC) instead of at

software of the Operating System

Generic Receive Offload (GRO) – Technique for increasing ingress throughput on fast networks
for reduce CPU overhead

• Aggregation of multiple incoming packets of a single flow into a larger packet before
passing up the network stack

• Reduces the number of packets that need to be processed

• Features are enabled by default since 18.2.5 otherwise must be enabled manually

• NIC support required

• Once enabled Service Engines will have statistics for both features

360
VMware Confidential - Internal Only
Commands to Enable TSO and GRO Features on the Controller (Should be in Shell mode):

• configure serviceenginegroup Default-Group

• no disable_gro

• no disable_tso

• save

After Entering Save - You will have the output.

9-27 TSO, GRO, and RSS (2)

Dispatcher on Service Engine is responsible for fetching incoming packets on a NIC and sending
them to appropriate core for proxy and sending back the outgoing packets to the NIC.

• 1 Core SE – no dispatcher required

• 2-4 Core SE – 1 shared dispatcher per NIC

• 5+ Core SE – dedicated dispatcher per NIC

Problem: As packet per second(pps) increases having a single core dispatcher can turn into a
point of congestion

VMware Confidential - Internal Only 361

9-28 TSO, GRO, and RSS (3)
Receive Side Scaling (RSS) – Enables the use of multiple queues on a single physical NIC
(multiple dispatchers)

• Multiple dispatchers(queues) created across multiple cores for a single physical NIC

• The Physical NIC will pin traffic flows to specific queues

• This feature is enabled on the transmit and receiver side

• When distribute_queues option is set to true, Service Engine automatically sets number of
dispatchers, but can be manually configured as well

Enable RSS Features

362
VMware Confidential - Internal Only
9-29 Multiple Dispatchers
Advanced Features

• Dispatcher cores can be configured in powers of 2 up to a maximum of 16 (0,1,2,4,8,16)

• When num_dispatcher_cores set to 0, this will tell service engine to automatically calculate
ideal dispatcher

• Service engine requires a reboot if number of dispatchers updated

• When num_dispatchers_cpu is configured this will set num_queues to same value

VMware Confidential - Internal Only 363

9-30 Denylisting (1)
Advanced Features

The Blacklist feature is used when Linux Server Cloud is configured and NICs are NOT to be
used or claimed by Service Engine/DPDK.

• Denylist file location on Linux Host is /etc/blacklist

• If updates are completed when service engine is already running, service engine must be
restarted

• File format is PCI BDFs(domain:bus:[Link]) of the NIC

• Entries in denylist file are comma-separated with no spaces

9-31 Denylisting (2)

Denylist file example

root@[Link]:~# ethtool -i eth9

driver:vmxnet3
version: [Link]-k-NAPI
firmware-version:
bus-info:0000:1c:00.0 ---- PCI BDF
supports-statistics:yes
supports-tests: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flogs: no
root@10-1-1-1:~# cat /etc/blacklist
0000:1c:00.0 ---- Black list file format

364
VMware Confidential - Internal Only
9-32 VLAN Interfaces
Advanced Features

Service Engines can be configured with VLAN interfaces which in turn will support 802.1q-
tagging

• Supported in both bare metal and linux server cloud

• Each VLAN interface has its own IP Address

• Multiple VLAN interfaces can be configured per physical interface

VMware Confidential - Internal Only 365

9-33 Port Channels (1)
Advanced Features

Port-channels can be used to group multiple physical interfaces into a single logical interface on a
service engine

• Provides fault tolerance, bandwidth aggregation and traffic load balancing

• Can be configured VLAN trunking

• Up to eight physical interfaces can be grouped into a single logical interface

• IPv6 support introduced in 18.1.2+

• Bond Configuration completed on linux host

• Service Engine will display interface with name of the bond

9-34 Port Channels (2)

DEVICE=bond0
IPADDR=[Link]
NETMASK=[Link]
ONBOOT=yes
BOOTPROTO=none
USERCTL=np
NM_CONTROLLED=no
BONDING_OPS="mode=4 miimon=100 xmit_hash_policy-layer3+4
use_carrier=1"
DEVICE=ens1f0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
NM_CONTROLLED=no

366
VMware Confidential - Internal Only
DEVICE=ens1f1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
NM_CONTROLLED=no
/etc/sysconfig/network-scripts/ifcfg-bond0

/etc/sysconfig/network-scripts/ifcfg-ens1f0

/etc/sysconfig/network-scripts/ifcfg-ens1f1

VMware Confidential - Internal Only 367

9-35 MAC Masquerade (1)
Advanced Features

In a situation where Service Engines are deployed in an active/standby legacy setup, to support
accelerate HA Service Engine failover MAC Masquerade will need to be enabled. This is due to
the floating IP configured on the service engines interface.

9-36 MAC Masquerade (2)

MAC Masquerade is the means to allow a mac address to move between hosts along with the
floating IP

• This allows front-end and back-end servers to continue communication even after a failure
because the same IP and Mac addresses continue to work

• Used with NSX Advanced Load Balancer SE IP routing

• Enabled in the service engine group settings of the CLI

[admin:10-140-1-4]: > configure serviceenginegroup Default-

Group
[admin:10-140-1-4]: servuceenginegroup> enable_vmac
Overwriting the previously entered value for enable_vmac
[admin:10-140-1-4]: serviceenginegroup> save
[admin:10-140-1-4]: >
MAC Masquerade enable

368
VMware Confidential - Internal Only
Module 10
Installing, Configuring, and Managing NSX
Advanced Load Balancer in VMware
Environments: Cloud Configuration

10-2 Module Lessons

1. Overview of vSphere

2. VMware Write Cloud

3. Deployment Prerequisites

4. Deployment

5. Configuring the Service Engine Group

6. Networks

7. Placement

8. NSX Data Center for vSphere SDN Integration: Cloud Configuration

9. NSX-T Data Center Integration: Cloud Configuration

10. VMware Cloud Connector Options Summary: Cloud Configuration

VMware Confidential - Internal Only 369

10-3 Lesson 1: Overview of vSphere
Comprised of building blocks to provide a feature-rich stack for compute virtualization.

370
VMware Confidential - Internal Only
10-4 VMware Features (1)
• Deployment Building Blocks:

— ESXi Hosts: VMs deployed on top

— vCenter Server: Management Frontend

• Clusters

• Resource Pools

• Datastores

• Virtual Switches (vSphere standard switch and vSphere Distributed Switch) and
Port Groups

• Controller can exist inside VMware or outside VMware, location is not important, just
reachability to the networks the controller or cloud connector need to communicate with is
important

• vCenter Server should exist and inside there is typically a data center that contains host
objects that are clustered

VMware Confidential - Internal Only 371

10-5 VMware Features (2)
Deployment Methods

• VMware No Access Cloud

— Manual installation, no VMware infrastructure awareness from NSX Advanced Load

Balancer Controller

• Linux Server Cloud on VMs

— No VMware integration; Similar to No Access except customer-managed OS

• VMware Write

— Full life cycle automation for resources on VMware

• VMware Write with NSX Data Center for vSphere

— Full lifecycle automation for resources on VMware with NSX Data Center for vSphere

• VMware environments with NSX-T Data Center:

— VMware No Access Cloud

— VMware Write with NSX-T Data Center (TBD)

• VMware Cloud on AWS (TBD)

372
VMware Confidential - Internal Only
10-6 VMware No-Access Cloud
VMware No Access Final Points

Best Practices

• vSphere vMotion is not supported for SEs and Controllers

• Storage Recommendation: SSD

• Deploy SE with Thick Provision Lazy Zeroed

• Anti-Affinity Rules to add resilience for Service Engines and Controllers if deployed in
VMware

VMware Confidential - Internal Only 373

10-7 Lesson 2: VMware Write Cloud

10-8 VMware Write Cloud: vCenter Server

Integration
• Controllers run on virtual machines (VMs) managed by vCenter Server

• When deployed into VMware Cloud managed by vCenter Server, NSX Advanced Load
Balancer operates as a fully distributed, virtualized system consisting of the Controller and
Service Engines (SEs), each running as a VM

374
VMware Confidential - Internal Only
10-9 vCenter Server Integration: NSX
Advanced Load Balancer Controller (1)
• Controller provides a single point of control and management for the cloud

• Controller runs on a VM

• Can be managed using its web interface, CLI, or REST API

10-10 vCenter Server Integration: NSX

Advanced Load Balancer Controller (2)
• The Controller stores and manages all policies related to services and management

• Through vCenter Server, the Controller discovers VMs, data centers, networks, and hosts

• Based on this autodiscovered information, virtual services can quickly be added using the
web interface

• To deploy a virtual service, the Controller automatically selects an ESXi server, spins up an
SE and connects it to the correct networks (port groups)

• Requires access to each ESXi host over 443

10-11 vCenter Server Integration: NSX

Advanced Load Balancer Controller (3)
• The Controller also provides a management center for multiple cloud infrastructures

• For example, the Controller can be configured to communicate with a vCenter Server
system, an Azure VNET, and a No-Orchestrator cloud for Cisco CSP

• The Controller can be deployed as a single VM or as a high availability cluster of three

Controller instances, each running on a separate VM

VMware Confidential - Internal Only 375

10-12 vCenter Server Integration: Service
Engine
• SEs provide the application delivery services to the end-user traffic

• Collect real-time end-to-end metrics for traffic between end users and applications

• Each SE runs on its own VM

• The Controller manages the life cycles of SEs

• To deploy an SE, the Controller creates an SE VM, plumbs it into a network, and provisions
it with service policies as required to deploy virtual services

376
VMware Confidential - Internal Only
10-13 Lesson 3: Deployment Prerequisites

10-14 VMware Write Cloud

VMware Write Cloud Requirements

Deployment Prerequisites for NSX Advanced Load Balancer Controller are listed:

• Access to each ESXi host on port 443

• vCenter Server account with appropriate privileges (see KB): Data center level or above

• Appropriate resources available on ESX hosts (SE creations will fail otherwise)

• Reservation for CPU and memory is preferred (not required)

VMware Confidential - Internal Only 377

10-15 Deployment Prerequisites: VM
Requirements
• NSX Advanced Load Balancer runs on standard x86-based servers

• The more hardware capacity you have, the more you can expand the system capacity

• Controller: 8 vCPU cores, 24 GB RAM, and 128 GB of storage

• Service Engine: 1 vCPU cores, 1 GB RAM, and 10 GB of storage

• Reservation for CPU and memory is recommended, but not required

• Modifying resource settings on VMs, such as CPU cores or RAM, requires powering down
the VM, making the changes, and then powering the VM back on

• Appropriate physical resources must be present in the ESXi host. If appropriate resources
are not present in the ESXi host, SE creation will fail and manual intervention will be required

10-16 Deployment Prerequisites: IP Address

Requirements
• Each Controller requires one management IP address

• To use the cluster IP, each Controller management address must be in the same subnet

• Each SE requires one management IP address, an IP address for each Virtual Service, and
an IP address facing the network

378
VMware Confidential - Internal Only
10-17 Deployment Prerequisites: vCenter
Server Account Requirements
• During the Cloud Connector setup, a vCenter Server account must be entered to allow the
Controller to communicate with vCenter Server

• The vCenter Server account must have privileges to create folders in vCenter Server

• This is required for SE creation, which in turn permits virtual service placement

• The vCenter Server account must be assigned at the data center level or above

VMware Confidential - Internal Only 379

10-18 Lesson 4: Deployment

10-19 Deployment (1)

Step 1 – Controller OVAs deployed through vCenter Server

• Management Network Info, Host/Datastore selection, optional SSH key

Step 2 – VMware Cloud Connector configured

• Basic options (Name, IPAM profile, DNS profile, static routing preferences)

• VMware-specific options (vCenter Server FQDN/IP, data center, credentials)

• Additional VMware options (Management Network from Port Group list)

• Discovery of VMware objects (Clusters/Host, Port Groups, VMs, and so on)

Step 3 – Service Engines deployed by Controller

• Occurs as a result of creating or modifying an Virtual Service

• Initial creation, scale-out of a Virtual Service, Service Engine failure, and so on

• Controller manages Service Engine life cycle

• Power on, automatic registration to Controller, ready for consumption

• Controller awareness of attached networks through Port Groups and so on

380
VMware Confidential - Internal Only
10-20 Deployment (2)

VMware Confidential - Internal Only 381

10-21 Deploying the Controller
• Log into vCenter Server

• Click File on the top menu and choose Deploy OVF Template

• Follow the instructions of the Deploy OVA Template wizard

— Choose Thick Provision Lazy Zeroed for disk format

— Choose a port group for Destination Networks in Network Mapping. This port group will
be used by the Controller to communicate with vCenter Server

— Specify the management IP address and default gateway. Or, leave them empty if
using DHCP

• Power on the VM

Use a static IP address for Controller management unless your DHCP server can keep the
assigned IP address permanentlyLab

382
VMware Confidential - Internal Only
10-22 VMware Write Cloud (1)
VMware Cloud Type Configuration

• Enter a name in for the cloud

• Select VMware as for the Cloud Infrastructure Type

• Now let us jump into VMware write cloud example

• Name can be anything, but if more clouds will exist on the same controller, recommend
making it something that will be easy to reference

• From Infrastructure/Clouds the Create will bring up the following options, in this case
vCenter Server/vSphere will be used

VMware Confidential - Internal Only 383

10-23 VMware Write Cloud (2)
VMware Cloud Infrastructure Configuration

• Default IPAM and DNS profiles used when creating Virtual Services if not explicitly defined
during creation

• vCenter Server credentials for the user that will log into the vCenter Server environment

— This is what the cloud connector will use to log in and pull information and complete
Service Engine deployments

• vCenter Server Address - this can be FQDN or IP address, but is recommended to be

FQDN

— FQDN will allow for IP address changes without impacting the Cloud configuration (DNS
needs to resolve FQDN)

— If using IP address, the cloud and all objects contained in it will have to be deleted and
rebuilt with new IP, making IP address less ideal

• Deployment mode as discussed is the Access Permissions to the cloud

• SDN integrations will be discussed in a future module, but quick overview:

— Cisco APIC: Integration with Cisco ACI fabric

— VMware NSX: Integration with NSX Data Center for vSphere

384
VMware Confidential - Internal Only
10-24 VMware Write Cloud (3)
VMware Cloud Data Center Configuration

• This is where data center vCenter Server manages can be selected, in this case there is only
1 but if there are multiple the list will be as many as the user in the previous step has access
to

• The management IPs refer to the control plane, which will be in reference how service
engines will get their IPs, if these options are not checked configuration in next step will walk
through how and what IPs will be assigned to service engines for the control plane traffic

• Service Engines by default will attach to the network Pool members exist on and create the
virtual service on the same network, Virtual Service Placement settings methods to get
around this type of deployment.

— https:// [Link]/docs/18.2/virtual-service-placement-settings/

VMware Confidential - Internal Only 385

10-25 VMware Write Cloud (4)
VMware Cloud Management Network Configuration

• Search Bar at the top allows to easily search for a specific port-group to be used by
Vantage Service Engines for control plane use

• In addition to the cloud connector collection, a list of networks configured inside the Data
Center selected in previous steps is also collected. The default management (control plane)
port-group is selected and configured on a service engine as they get autodeployed.

• If DHCP is not used static IPs are required to be configured for management

— Static IPs can be assigned as a range, or individual IPs but will be required for Service
Engine deployment(can be modified at any time)

386
VMware Confidential - Internal Only
10-26 VMware Write Cloud (5)
VMware Cloud

• Summary of Cloud configuration

• Status icon confirms that cloud is ready

• VMware Write Cloud is now configured and status is green indicating that Virtual Services
can now be deployed which will engage Service Engines to autodeploy

• This also validates that the cloud connector is working as expected with the vCenter Server

VMware Confidential - Internal Only 387

10-27 Lesson 5: Configuring the Service
Engine Group

10-28 Configuring the Service Engine Group (1)

• Click Service Engine Group

• Click the drop-down menu next to Select Cloud to pick the newly created VMware Cloud

• Click Create

388
VMware Confidential - Internal Only
10-29 Configuring the Service Engine Group (2)
• Configure the SEG basic settings

• Click Advanced

VMware Confidential - Internal Only 389

10-30 Configuring the Service Engine Group (3)
• Configure the SEG advanced settings

390
VMware Confidential - Internal Only
10-31 Lesson 6: Networks

10-32 VMware Write Cloud (1)

VMware Cloud Exposed Options - Networks

• Port Groups auto discovered as Networks

• If no DHCP, static subnets are set

VMware Confidential - Internal Only 391

10-33 VMware Write Cloud (2)
VMware Cloud Exposed Options - Networks

Discovered

• Through vCenter Server and mapped by Controller

• Port Group names discovered

• Discovered Networks derived from VM connectivity

• Helps in identification of Port Group to subnet/network mapping

392
VMware Confidential - Internal Only
10-34 VMware Write Cloud (3)
Configured

• Allow specification of subnets and ranges of IPs for to use

— DHCP can optionally be used

— Can be used for IPAM

• Required for both Management and Data connectivity

— Service Engine requires a minimum of 2 vNICs

— VRFs/Namespaces used for management and data plane isolation

• Used by the Controller for Virtual Service placement

— Placement settings on Virtual Service when controller placement logic is not configured

VMware Confidential - Internal Only 393

10-35 VMware Write Cloud (4)
VMware Cloud Exposed Options

• Pool Creation can be done on information collected by cloud connector

394
VMware Confidential - Internal Only
10-36 Lesson 7: Placement

10-37 Virtual Service Placement

Placement

• A virtual service is placed on one or more Service Engines based on:

— Capacity: The SE has capacity to add an additional VS

— Reachability: The SE has network access to the VIP network, and to the server
network

— Anti-affinity: SEs for a VS must be on different hosts (servers)

• A pool that is not attached to a VS is not placed

— Being unplaced means servers are not health checked

• Disabling a virtual service removes the VS from the SE. It remains in the configuration of the
Controller

— While the VS is disabled, no health checks will be performed on the pool servers

• VS

VMware Confidential - Internal Only 395

10-38 One Arm Versus Two Arms
Default Behavior

• No functional difference between 1 or 2 NIC

• A different NIC for client and server network enables twice the link throughput

Remote Servers

• Servers may be a routed hop away from the SE

• By default, in clouds with Write Access mode, the Controller will attempt to place an SE
interface in the server network

• If SE does not have direct access to server networks:

— Static or default route is required

— VS > Ignore network reachability for server networks

— Pool placement settings

Further Reading

• Network Reachability

396
VMware Confidential - Internal Only
10-39 Virtual Service Placement in Write
Access Cloud
Default Behavior

• Service Engines NICs will be attached to the appropriate port groups

• Placement options determine where the NICs will be attached to

• By default, NICs will be attached to directly connected subnets

Prefer Static Routes vs Directly Connected Networks (Server)

• The option along with configured static routes identifies which port groups to plumb to for
server placement

Use Static Routes for Resolution of Network VIPs (Virtual Service)

• This option along with configured static routes identified which port groups will be used to
place virtual service

• Used when, virtual service address is not directly attached to service engine, for example:
when routing to virtual services across a transit network

VMware Confidential - Internal Only 397

10-40 Prefer Static Routes Versus Directly
Connected Networks (Server)
Prefer Static Routes vs Directly Connected Networks (Server)

• The option along with configured static routes identifies which port groups to plumb to for
server placement

398
VMware Confidential - Internal Only
10-41 Use Static Routes for Resolution of
Network VIPs (Virtual Service)
Use Static Routes for Resolution of Network VIPs (Virtual Service)

• This option along with configured static routes identified which port groups will be used to
place virtual service

• Used when, virtual service address is not directly attached to service engine, for example:
when routing to virtual services across a transit network

VMware Confidential - Internal Only 399

10-42 VMware Write Cloud

• Cloud Connector will take care of all the heavy lifting and admin/user has ability to add
controls around deployment and configuration of service engines

• This also makes life cycle and scale-out fully automated

• From the Cloud configuration to the SEG configuration the orchestration is now set up to
autodeploy SE based off user requirements to whatever folder/storage/compute specified
in SEG configuration

• On top of this when building a VS portgroups will get auto allocated through the
orchestration which is all controlled through the integration cloud connector provides

• The more information the cloud-connector has about the environment the more can simplify
provisioning and building of applications

400
VMware Confidential - Internal Only
10-43 Best Practices: VMware Write Cloud
• Ensure you have enough compute capacity, licenses, IP addresses to be able to scale as
needed if an increase in traffic, or hardware failure occurs.

• Make sure you have redundant DNS servers configured. If the Controller cannot resolve the
ESXi hostnames, it cannot create SEs

• Use the FQDN of the vCenter Server instead of the IP. If the vCenter Server changes and
you used the IP, you have to rebuild the cloud connector

• If you can use dedicated hypervisors for the SEs, do it. This will contain where SEs can be
created, ensure capacity, ease fears from the compute team

• Make sure you analyze the traffic profiles of the services that are going to run within each
SEG, so you can properly size

• Have a list of all PGs that will use, and confirm that the networks are discovered correctly

• Consider limited vSphere vMotion support for Controllers and Service Engines, DRS mode
has to be switch to Manual for Controllers and Service Engines

• VM Anti-Affinity Rules should be considered for Controllers and Service Engines

VMware Confidential - Internal Only 401

10-44 Lesson 8: NSX Data Center for vSphere
SDN Integration: Cloud Configuration

10-45 NSX Data Center for vSphere (1)

VMware Cloud

Steps 1-3 – Similar to Write Access.

Additional – VMware Cloud Connector NSX integration.

Controller manages necessary distributed NSX components.

NSX Manager REST API.

Distributed Firewall Rules.

Individual Security Groups can be used in pools.

402
VMware Confidential - Internal Only
• Previously covered vCenter Server integration, now adding NSX Data Center for vSphere
on top

• Need credentials to both NSX Manager and vCenter Server

• DFW or Distributed Firewall is required to be enabled on the cluster with Vantage will
operate. This will enable to dynamically update the load-balancing pool using NSX security
groups and publish the DFW rules to allow traffic in which is load balanced

— NOTE: if DFW is not enabled, integration of NSX Data Center for vSphere is not
required as no other features would be used.

10-46 NSX Data Center for vSphere (2)

VMware Cloud

• Integrate cloud connector with vCenter Server and NSX Data Center for vSphere

• Requirements:

— vCenter Server Write Requirements

— NSX User Credentials

— NSX Security Groups for pool members

• Use Cases:

— Microsegmentation

— Dynamic Pool Membership based on NSX Security Tags and Groups

VMware Confidential - Internal Only 403

10-47 VMware Write Access with NSX Data
Center for vSphere (1)
VMware Cloud

• Navigate to SDN Integration > VMware NSX

• Enter credentials and Prefix

• Example Prefix: Demo

• Cloud Connector configuration just requires NSX host name or IP Address and Credentials

• There is also a line called prefix, this refers to any object that Vantage Cloud Connector
creates on NSX Manager

• As long as Controller can now reach the NSX Manager, no other changes need to be made
and the cloud connector can now publish DFW rules

404
VMware Confidential - Internal Only
10-48 VMware Write Access with NSX Data
Center for vSphere (2)
VMware Cloud

VMware Confidential - Internal Only 405

10-49 VMware Write Access with NSX Data
Center for vSphere (3)

• Integration with NSX Advanced Load Balancer cloud connector allows all the features of
VMware cloud with additional DFW policy auto creation

406
VMware Confidential - Internal Only
10-50 VMware Write Access with NSX Data
Center for vSphere (4)
Security Automation

VMware Confidential - Internal Only 407

10-51 NSX Data Center for vSphere
Integration: Virtual Service Distributed
Firewall Rule
• Discovers NSX Security Groups

• Allows Dynamic Pool Membership for entities added to or removed from a Security Group

• Automated DFW Rules Management

• NSX security group is created, which will be a collection of server objects (pool members) -
all pool members must be in Security group

• During Creation of VS Security Group is selected as shown which was created before hand
• Once this VS is saved and created on NSX Advanced Load Balancer, Cloud connector
creates a DFW policy for the port the VS is created for (in this example port 80) to allow
the SEs to communicate to the pool members in the NSX security group

• If multiple ports are defined on the VS, cloud connector will add more ports to the service
details of the DFW policy

• The DFW security group can be seen with the prefix defined in the cloud

408
VMware Confidential - Internal Only
10-52 Lesson 9: NSX-T Data Center
Integration: Cloud Configuration

10-53 NSX-T Data Center

VMware Cloud

• NSX Advanced Load Balancer only supports No Access deployment in the NSX-T Data
Center environment

VMware Confidential - Internal Only 409

10-54 Configuration Steps
Step 1 – Controller OVAs deployed through vCenter Server

• Management Network Info, Host/Datastore selection, optional SSH key

Step 2 – No-Orchestrator Cloud Connector configured

• Basic options (Name, IPAM profile, DNS profile, static routing preferences)

Step 3 – Cloud secure tokens generated

• Unique secure token for each Service Engine, one-hour expiration, tied to Cloud

Step 4 – Service Engine OVAs deployed through vCenter Server

• Automatic registration with Controller, ready for consumption

• Controller can exist inside VMware or outside VMware, location is not important, just
reachability to the networks the controller or cloud connector must communicate with is
important

• vCenter Server should exist and inside there is typically a data center that contains host
objects that are clustered

410
VMware Confidential - Internal Only
10-55 Typical NSX Advanced Load Balancer
VMware NSX-T Data Center Design
VMware Cloud

• Service Engines are deployed in single arm mode, that is, same interface is used for client
and back-end server traffic

• The SE routes to back-end servers through Tier-1 router

• Multiple Tier-1 routers supported

• Management or Data Interfaces can be exposed to external third-party management port-

groups at distributed or standard vSwitches

VMware Confidential - Internal Only 411

10-56 Lesson 10: VMware Cloud Connector
Options Summary: Cloud Configuration

10-57 VMware Cloud Connector Options

Summary
VMware Cloud

Cloud Type Variation Resource Discovery SE Provisioning

No Access* No awareness of underlying Manual Manual

infrastructure; token based

VMware Read Read-only access to vCenter Automated Manual

(DEPRECATED) Server; token based (Controller)

VMware Write* Read and Write access to Automated Automated

vCenter Server (Controller)

VMware Write with Read and Write access to Automated Automated

NSX Data Center for vCenter, discovery of NSX (Controller)
vSphere Manager

Linux Server Cloud** Runs on VMs with Linux OS Manual Manual

(managed by customer,
uncommon in VMware
environments)

* Most common deployments

** Uncommon on VMware environments, usually in lab

412
VMware Confidential - Internal Only
Module 11
AWS Cloud Configuration

11-2 AWS Cloud Connector (1)

AWS Scenario

• AWS Account and Region

• Multi-AZ for Resilience

• VPC and objects inside the VPC

VMware Confidential - Internal Only 413

• Quick AWS overview

• AWS Account is the administrative domain for environment and components inside

— Region is the geographic location of the AWS infrastructure

— Multi availability zones for resilience, every region has different number of AZs

— VPC is where network components and the EC2 instances are configured/deployed

11-3 AWS Cloud Connector (2)

AWS Requirements for Avi Integration.

Requirements:

• AUTH

— Access/Secret Key

— IAM Policies

• VPC Creation (Region)

— Subnets (Per Intended AZ)

— Gateway

— Routes

— Route53 (DNS Zone Integration)

Use Cases:

• Life cycle SEs/IPAM/DNS Profile/Server Autoscale

• Multi-AZ Load balancing

414
VMware Confidential - Internal Only
• AviVantage Cloud Connector communicates with AWS API endpoint to collect information
and to execute create and delete actions

VMware Confidential - Internal Only 415

11-4 AWS Cloud (1)
AWS Cloud Type Configuration

• Enter a name for the cloud

• Select AWS as the Cloud Infrastructure Type

416
VMware Confidential - Internal Only
11-5 AWS Cloud (2)
AWS Cloud Infrastructure Configuration

• AWS-Region will define the region in which this AWS cloud will be configured for, so when
the “Next” button is clicked, the cloud connector will log in to that region for the account
obtained from auth method

• Access AWS through Proxy allows the configuration of a proxy for the Cloud Connector to
connect to the AWS API

• Auth Methods

— Identity and Access Management (IAM) roles: IAM roles are the set of policies that
define access to resources within AWS. The roles and the policies that define their
access are defined in JSON files. This method does not require an AWS account key.
Instead, the role and policy files must be downloaded from Avi Networks and installed
using the AWS CLI

— AWS customer account key: A unique authentication key associated with the AWS
account. Access credentials are needed by the Avi Controller to communicate with
AWS APIs. Note: AWS cloud configuration with Avi SaaS Controller only supports the
Use Access/Secret Key credentials method. For recommendations regarding using
access key on AWS, refer to
[Link]

VMware Confidential - Internal Only 417

 — Use Cross-Account AssumeRole: Avi Vantage can be deployed for Amazon Web
Services (AWS) with multiple AWS accounts utilizing the IAM AssumeRole functionality
that provides access across AWS accounts to the AWS resources/API from the
respective accounts, instead of sharing user Access Key ID and Secret Access Key
from different accounts

11-6 AWS Cloud (3)

AWS Cloud Infrastructure Configuration

• VPC: Based on region selected in infrastructure a list of VPCs will be populated, this will be
the VPC environment where AviVantage will be deployed

• Elastic IPs will by default be selected as enabled, this will leave the acquisition and removal
of EIPs up to Avi Vantage, meaning when a VirtualService is created/deleted, Avi Vantage
Cloud Connector will obtain/release the elastic IP obtained from AWS. If there is a need to
keep specific public IP’s, in other words never lose them, it may be ideal to disable this and
acquire elastic IPs manually and assign public IPs to VS from what has been acquired already

• Availability Zones provide a sense of high availability, the Avi Vantage solution fully supports
multiple AZ deployments, and for this, management networks for each AZ is required, so
the option to deploy in multiple AZs is available
• Default Service Engine group is provided, this will help accidental SE creations under less
ideal AWS flavors

418
VMware Confidential - Internal Only
• DNS Integrations for virtual service names can use Amazon Route 53 based off the
permissions acquired from the auth profiles, or DNS profile configured on the Avi platform
(which is covered in another module)

• Encryption Methods can be done on the SE S3 Bucket, or the SE AMI/EBS volume and
snapshots, Avi only supports AWS SSE KMS encryption

• AutoScale policies for pool members are a specific use case where Avi Vantage has
permission to add new servers to a pool based off autoscale policy defined on AWS. The
poll interval is frequency at which Avi Vantage will poll SNS/SQS

11-7 AWS Cloud (4)

AWS IAM Policies

• Controllers deployed in AWS can leverage IAM Roles instead of Access Key ID/Secret
Access Key combo

• IAM Roles allow Controllers to make API calls to AWS based on permissions of the policies

• NSX Advanced Load Balancer SEs required in multiple AWS Accounts can leverage a Cross
Account Assume Role

• High availability and redundancy

VMware Confidential - Internal Only 419

11-8 AWS Cloud (5)
AWS Cloud SEG Exposed Options

• Service Engine Group exposes AWS instance flavors

• C Flavor instances are only supported

• Security Groups can be auto generated/managed

— Custom Security Groups can also be configured

• Autoscale subnet to determine where VS will be placed

• SEG added feature is ability to select which AWS flavor SE instances will be deployed on

• Also have the ability to have AviVantage manage Security groups SE’s get deployed with,
or user-defined security groups can be created and used (should disable Avi Managed
Security Groups if using custom ones)

• Custom Security Groups are obtained by cloud connector and can be selected by the drop-
down menu for Management or Data networks

• VIP autoscale determines networks VSs can be created on

420
VMware Confidential - Internal Only
11-9 AWS Cloud (6)
AWS Pool Exposed Options

NSX Advanced Load Balancer scale in and out functionality exists as follows:

• Scale Virtual Service to more or fewer Service Engines

— Criteria such as Bandwidth, CPU Utilization, Connections Per Second, Packets Per
Second

• Scale Application server pool to more or fewer applications instances in the backend based
on resource needs

— AWS AutoScale group used and NSX Advanced Load Balancer updates pool
membership automatically

• High availability and redundancy

VMware Confidential - Internal Only 421

11-10 AWS Cloud (7)
AWS Pool Exposed Options

• One Service Engine Group to manage SEs across multiple Availability Zones

• Multiple Virtual Service IPs per Availability Zones but 1 DNS name/fqdn

• Route53 auto updates if integration configured in Cloud

• High availability and redundancy

422
VMware Confidential - Internal Only
11-11 AWS Cloud (8)
AWS Virtual Service Exposed Options

• Elastic IP option for configuring a public IP, each VS will acquire a public IP

• Multiple VIP addresses can be created to support Multi AZ deployment

• An application FQDN required for this setup

• High availability and redundancy

VMware Confidential - Internal Only 423

11-12 AWS Cloud (9)
AWS Summary

• One Cloud per VPC

• Within VPC – Multiple Availability Zones

• NSX Advanced Load Balancer Controllers can leverage IAM Roles or Access Key
ID/Security Key Combo

• NSX Advanced Load Balancer Controllers can connect across AWS accounts with assume
role policies

• One Service Engine Group can be used for multiple Availability Zones

• Security Groups can be Custom or NSX Advanced Load Balancer can manage them

• NSX Advanced Load Balancer can autoscale service engines or work with AWS autoscale
policy for pool membership

• High availability and redundancy

424
VMware Confidential - Internal Only
Module 12
DNS Foundation

12-2 Module Lessons

1. DNS Summary

2. NSX Advanced Load Balancer DNS Virtual Service

3. NSX Advanced Load Balancer IPAM and DNS Profile

VMware Confidential - Internal Only 425

12-3 Lesson 1: DNS Summary

12-4 Issue
Need an alternative method to using IP Address to talk to services:

• Remembering IPs and what applications they belong to is added complexity and not
scalable

• If an application IP is changed, all users and other applications need to be informed

426
VMware Confidential - Internal Only
12-5 About DNS
Domain Name System(DNS) takes a readable name and translates it to a number.

• Name is referred to as Fully Qualified Domain Name (FQDN)

• Number is the IP Address

• DNS transactions are done on port 53 UDP/TCP

Applications such as Ping or Web Browsers will translate this readable name into an IP address
with DNS

• Transparent to users

• Name used as a description of application making easier to read and understand

VMware Confidential - Internal Only 427

12-6 DNS Transaction
• DNS transaction consists of a DNS Query and Answer based on type of record requested

• Common record types

— A: Host Address

— AAAA: IPv6 Host Address

— CNAME: Canonical name for alias

— NS: Name Server

— SOA: Start of Authority

428
VMware Confidential - Internal Only
12-7 DNS Resolution
• DNS Resolution works as a hierarchy of domains

• Hierarchy has the Root(.) and TLDs at the top and then subdomains

• Each domain has a group of servers with zone files

— Often referred to as authoritative DNS server for specific domain or zone

• Zone files are where DNS records for the specific domain are configured

• DNS queries start from the top of hierarchy and work its way down

VMware Confidential - Internal Only 429

12-8 DNS Query
• DNS query objective: Translate Application Name(FQDN) into an IP address

• Authoritative Name Server help guide client DNS queries to find answers

430
VMware Confidential - Internal Only
12-9 Lesson 2: NSX Advanced Load Balancer
DNS Virtual Service

12-10 NSX Advanced Load Balancer DNS

Virtual Service (1)
NSX Advanced Load Balancer DNS Virtual Service can be configured as an authoritative Name
Server for one or more subdomains

VMware Confidential - Internal Only 431

12-11 NSX Advanced Load Balancer DNS
Virtual Service Deployment

432
VMware Confidential - Internal Only
12-12 NSX Advanced Load Balancer DNS
Application Profile

• SOA can be configured in NSX Advanced Load Balancer, but done in CLI

VMware Confidential - Internal Only 433

12-13 NSX Advanced Load Balancer DNS
Virtual Service (2)
NSX Advanced Load Balancer DNS Virtual Service acts as authoritative Name Server for one or
more subdomains.

434
VMware Confidential - Internal Only
12-14 NSX Advanced Load Balancer DNS
Virtual Service (3)
DNS policy consists of rules which in turn consist of match targets and actions.

VMware Confidential - Internal Only 435

12-15 NSX Advanced Load Balancer DNS
Virtual Service (4)
Quick Glance at DNS Logs.

436
VMware Confidential - Internal Only
12-16 Lesson 3: NSX Advanced Load Balancer
IPAM and DNS Profile

12-17 NSX Advanced Load Balancer IPAM and

DNS Profile (1)
IPAM/DNS Profile can also enable integration with different external providers

IPAM:

• Available Networks pulled from Provider into NSX Advanced Load Balancer

• Automatic IP assignments for Virtual Services

• IP Assignments tracked in cloud provider

DNS:

• Virtual Service configured with DNS name

• NSX Advanced Load Balancer can updated authoritative DNS provider directly for each
new Virtual Service

VMware Confidential - Internal Only 437

12-18 NSX Advanced Load Balancer IPAM and
DNS Profile (2)
Provider ------> Infoblox Avi Vantage Cloud-native
Internal

Cloud Infrastructure IPAM DNS IPAM DNS IPAM DNS

VMware vCenter Yes Yes Yes Yes N/A N/A

OpenStack No No No Yes Yes N/A (not

(default) used)

Amazon Web Services No No No Yes Yes Yes

(default) (default)

Google Cloud Platform No No No Yes Yes No

Azure (as of 18.2.5) No No No Yes Yes Yes

(default) (default)

Containers Yes Yes Yes Yes Yes No

(Mesos/Kubernetes/Rancher/Docker
UCP)

Linux Server (bare metal) Yes Yes Yes Yes Yes No

No access cloud Yes Yes Yes Yes Yes No

438
VMware Confidential - Internal Only
12-19 NSX Advanced Load Balancer IPAM and
DNS Profile (3)
• IPAM and DNS Profile can be local to the NSX Advanced Load Balancer

• Local profiles are where configuration is completed on NSX Advanced Load Balancer

• All managed in NSX Advanced Load Balancer - no visibility externally

VMware Confidential - Internal Only 439

12-20 NSX Advanced Load Balancer IPAM and
DNS Profile (4)
IPAM Provider Integration Use Case:

• A deployment where ecosystem is built on top of a cloud provider (AWS)

• NSX Advanced Load Balancer Cloud is Kubernetes in this case, so no AWS cloud connector

• VS creation under Kubernetes Cloud will be assigned IPs from AWS usable networks

12-21 NSX Advanced Load Balancer IPAM and

DNS Profile (5)
DNS Provider Integration Use Case:

• DNS domains exist in a remote provider already

• Use domains and subdomains from authoritative DNS provider to build applications

• When a new FQDN is configured as per Virtual Service configuration, the record is pushed
to DNS provider

440
VMware Confidential - Internal Only
Module 13
Global Server Load Balancer

13-2 Module Lessons

1. NSX Global Server Load Balancer Introduction

2. NSX Advanced Load Balancer Architecture and Objects

3. Configuring GSLB Infrastructure

VMware Confidential - Internal Only 441

13-3 Lesson 1: NSX Global Server Load
Balancer Introduction

13-4 About GSLB

GSLB provides the following capabilities:

• Intelligently load balance applications load across multiple instances of the application
deployed at geographically dispersed locations (typically, multiple data centers and public
clouds).

• In contrast, load at any one location is managed by a “local” NSX Advanced Load Balancer.

442
VMware Confidential - Internal Only
13-5 GSLB Benefits
GSLB provides the following benefits:

• Provide an optimal application experience to users/clients

— Route users to nearest and available Data Center based on geo, lowest-latency, least
loaded, and so on.

• Offer resilience to loss of a data center or a network connection

— Real-time monitoring of application across Data Center

• Perform non-disruptive migration to or addition of another Data Center

— Assign priority to Data Center

VMware Confidential - Internal Only 443

13-6 GSLB High-Level Functionality
GSLB provides the following functionality:

• DNS Service can be Authoritative for one or more zones (subdomains)

• Chooses the location (data center/cloud) to which to steer the client's requests

• Monitors health of the virtual services so that it can choose the best location (that is, rule
out unhealthy ones)

• Synchronizes configuration and state across GSLB sites

• Simplified and Centralized Operational Model

• DNS Analytics and Client Logs

• Support active-standby and active-active applications

444
VMware Confidential - Internal Only
13-7 Internet-Based DNS Query

VMware Confidential - Internal Only 445

13-8 GSLB Use Cases (1)
Here are a few use cases for NSX Advanced Load Balancer GSLB:

• Optimal application experience for geographically distributed users

— Applications are deployed in multiple data centers

— NSX Advanced Load Balancer GSLB can steer the user traffic to the most optimal
location.

• Application high availability across data center failures

— Applications are deployed in multiple data centers

— If a data center fails, application instances running in the remaining data centers can take
over the user traffic

• Disaster recovery

— Applications are deployed in two data centers

— While both are healthy, all traffic is directed to the primary DC

— If the primary DC fails, the global DNS directs all user traffic to the other

13-9 GSLB Use Cases (2)

Here are a few use cases for NSX Advanced Load Balancer GSLB:

• Hybrid cloud with “cloud bursting”

— Applications are deployed across private and public clouds

— When/if an application experience an unusually high request load, NSX Advanced Load
Balancer GSLB “bursts” to the public cloud site to absorb the load

446
VMware Confidential - Internal Only
13-10 Lesson 2: NSX Advanced Load Balancer
Architecture and Objects

13-11 Generic GSLB Deployment Architecture

The following are GSLB architectural elements:

• Application Advanced Load Balancer in five sites, and some sites have multiple Advanced
Load Balancer

• Some application would be hosted at all data center and some would not be deployed at all
data center

• GSLB service is running in three sites - Santa-Clara, Chicago, NY sites

— GSLB synchronizes stats across its members

— Active site GSLB polls Passive sites stats

• Clients DNS query is sent to one of the GSLB by Client LDNS

VMware Confidential - Internal Only 447

13-12 GSLB Object
There are two layers in GSLB Objects and multiple components:

• GSLB Infrastructure Layer Object

— GSLB Site

— GSLB SE Group

— GSLB Virtual Service

• GSLB Application Layer Object

• GSLB Service

• GSLB Pool

• GSLB Pool Member

• Weight

• Load-Balancing Algorithm

• GSLB Health Monitor

448
VMware Confidential - Internal Only
13-13 GSLB Object: GSLB Site
• GSLB Sites

— NSX Advanced Load Balancer

— Third party

• GSLB Sites Functions

— Definition and ongoing synchronization/maintenance of the GSLB configuration

— Monitoring the health of configuration components

— Optimizing application service for clients by providing GSLB DNS responses to their
FQDN requests

— Processing of global application requests

GSLB Sites Functions

• Definition and ongoing synchronization/maintenance of the GSLB configuration — an Avi

Controller responsibility.

• Monitoring the health of configuration components — a responsibility shared by Avi

Controllers and Service Engines.

VMware Confidential - Internal Only 449

• Optimizing application service for clients by providing GSLB DNS responses to their FQDN
requests — the responsibility of an Avi GSLB DNS running in one or more Service Engines.

• Processing of global application requests — the responsibility of services placed on Avi SEs
or running on third-party servers/load balancers.

450
VMware Confidential - Internal Only
13-14 GSLB Object: GSLB Site Classification (1)
• Active site

— Active sites have GSLB instance deployed

— Active site further classified as:

• Active Leader Sites

• Active Follower Sites

• Passive site

— Passive site does not participate in DNS resolution

— Passive site is classified a Passive Follower site

— Passive sites cannot assume the leader role

VMware Confidential - Internal Only 451

13-15 GSLB Object: GSLB Site Classification (2)
The Active Leader Site is responsible for:

• Active site participates in DNS resolution

• The leader active site is responsible for all GSLB site functions

• Exactly one active site is statically designated as the GSLB leader

Leader Active Site

• The leader active site is responsible for the functions 1, 2, 3, and 4 as mentioned in the
previous section.

• Followers are further classified as active or passive, based on their behavior vis-a-vis the
first three functions.

452
VMware Confidential - Internal Only
13-16 GSLB Object: GSLB Site Classification (3)
The Active Follower Site is responsible for:

• Receives the GSLB configuration from the leader

• Participates in DNS resolution

• If Active leader site fails or maintenance leadership is switched through a manual takeover
process from an active follower

• May actively (data plane) monitor the health of other GSLB sites

VMware Confidential - Internal Only 453

13-17 GSLB Object: GSLB Site Classification (4)
The Passive Follower Site is responsible for:

• Does not participate in DNS resolution

• Does not receive the GSLB configuration, and thus cannot take over leader

• Passive sites perform only one GSLB site function, that is, the hosting of virtual services that
respond to requests from the clients of global apps

• Does not monitor other sites

• Does provide health state information of local resources

A GSLB site may participate in exactly one Avi GSLB configuration. If site_A is a participant in
GSLB_config_1, an attempt to

454
VMware Confidential - Internal Only
13-18 GSLB Object: GSLB SE Group
• As a best practice, GSLB DNS Virtual Service should be exclusively placed on its own
Service Engine group

• For each controller cluster, configure a Service Engine group to host the DNS virtual service

• SE group names need not be identical across all GSLB sites

As a best practice, a DNS for GSLB should be exclusively placed on its own Service Engine
group. That is, place no other virtual services (DNS or other application types) on it.

Service Engine cannot perform Health Monitoring of locally hosted Virtual Services.

VMware Confidential - Internal Only 455

13-19 GSLB Object: GSLB Virtual Service
• DNS VS will be authoritative for the subdomains delegated to NSX for GSLB

• Virtual service with the application profile type of System-DNS and a network profile using
per-packet load balancing

• As a best practice, a VS for GSLB should be exclusively placed on its own Service Engine
group

• Configure a DNS virtual service on all the clusters where the DNS service needs to be
hosted and bound to the SE group dedicated for DNS VS

• As a best practice, DNS VS listens on both TCP and UDP port 53

Service Engine cannot perform Health Monitor of locally hosted Virtual Services.

456
VMware Confidential - Internal Only
13-20 GSLB Object: GSLB Service
A GSLB service is the representation of a global application. It front-ends instantiations of the
application that are deployed at multiple sites. The GSLB Service object identifies the following
data, which are key to understanding operation of the NSX Advanced Load Balancer GSLB:

• GSLB service’s name, by which administrators reference the GSLB configuration

• FQDN by which end-user clients reference the GSLB application. A list of domain names can
be provided for aliasing (that is, [Link] and [Link])

• A time-to-live (TTL) ranging from 1-86400 seconds that determines the frequency with
which clients need to obtain fresh steering information for client requests. If none is specified
for a particular GSLB service, the TTL value defaults to the one specified in the DNS
application profile (default there is 30 seconds)
• Backing virtual services in various GSLB sites, organized into virtual service pools

• Priorities for the various pools and ratios for their members

• Load-balancing scheme

• Health monitoring methods by which unhealthy components can be identified so that

alternatives may be selected

VMware Confidential - Internal Only 457

13-21 GSLB Object: GSLB Pools (1)
• A global application, AppA, spanning four sites, DC1 through DC4

• A GslbPool object “GslbPool_1” combines virtual services VS-A1 through VS-A4 into a
single entity and balances load across them

458
VMware Confidential - Internal Only
13-22 GSLB Object: GSLB Pools (2)
• AppB, which corresponds to the GslbService_B object, is composed of two pools, with
priorities 5 and 10

• As long as the pool of highest priority is up and not at its connection limit, all traffic will be
directed to it

• However, when/if a pool is inaccessible, down, or at maximum capacity, a lower priority pool
will be chosen instead.

VMware Confidential - Internal Only 459

13-23 GSLB Object: GSLB Pool Members
The virtual services that comprise a GSLB pool are called GSLB pool members

• Members can be specified by:

— Their NSX Advanced Load Balancer virtual service name

— An IP address, to specify standalone servers or VIPs defined by third-party load

balancers

— A DNS name, for example, to specify a DNS-based load balancer such as AWS ELB

460
VMware Confidential - Internal Only
13-24 GSLB Object: Weights
• All members of a GSLB pool share priority, but each member of the pool may potentially
have different weights

• When a pool is selected and its load is distributed across members by using the round-robin
algorithm, these weights dictate the fraction directed to each member

VMware Confidential - Internal Only 461

13-25 GSLB Object: Load-Balancing Algorithms
When a pool is selected, a [Link] balances load across the pool’s virtual services.

• The weighted-round robin algorithm balances evenly across all members

• Consistent hash is based on the client IP address (typically the local DNS IP address

• The geolocation algorithm directs client requests to the optimal site based on the longitude
and latitude of the client and the GSLB sites

Future load-balancing algorithm

• A load-aware configuration steers clients to the most optimal site based on the observed
load of the members. This option will be added in a future release

462
VMware Confidential - Internal Only
13-26 GSLB Object: GSLB Health Monitor
• To make its steering decision, the GSLB DNS service must know the health of GS members

• A GslbHealthMonitor object is configured

• Two kinds of health checking (control-plane and data-plane health) and their combination
are supported

VMware Confidential - Internal Only 463

13-27 Lesson 3: Configuring GSLB
Infrastructure

13-28 Configuration of GSLB Sites (1)

• A given NSX Advanced Load Balancer Controller participates in GSLB deployment or not

• If not, the first time it is approached to turn on GSLB, it assumes a leadership role

• Multiple GSLB configurations on a given Controller cluster are not supported

All sites participating in GSLB must run at the same software version and maintenance release.

• The Controller that assumes the GSLB leader role must not run a later software release or
maintenance version than any of its GSLB follower sites.

• This restriction applies both during the initial configuration of GSLB and during subsequent
upgrades of NSX Advanced Load Balancer (that is, the leader site must be upgraded after
all its follower sites have been upgraded).

13-29 Configuration of GSLB Sites (2)

You must complete the following steps to configure a GSLB Site:

1 Set up the individual Controller clusters

2 Configuration of GSLB Sites

3 Configuration of a GSLB Service

4 Configure local application virtual services

5 Designate the GSLB leader controller, and create the site configuration

For better audit trails, the recommended practice is to set up an admin user account for the
GSLB configuration. Create a user named ‘gslb’ and assign it admin roles in all the Controller
clusters

464
VMware Confidential - Internal Only
13-30 Setup of Individual Controller Clusters
Setup of Individual Controller clusters:

The recommended practice is to set up an admin user account for GSLB configuration. Create a
user named ‘gslb’ and assign it admin roles in all the Controller clusters

VMware Confidential - Internal Only 465

13-31 Configuration of GSLB Sites (3)
Configure a local DNS virtual service on all active sites that host DNS

• Configure a local DNS virtual service on all the clusters where the DNS service must be
hosted

• You must use the advanced mode

• Use a DNS app profile

• Use a UDP per packet L4 profile

• There is no need to provide a pool

• As a best practice, a DNS VS for GSLB should be exclusively allocated its own Service
Engine group

The virtual service and SE group names need not be identical across all GSLB sites.

There is no attribute associated with a DNS virtual service object to define it as a “GSLB” DNS.

If planning on using geolocation as the load-balancing algorithm, the GSLB SE must have a
minimum of 8 Gb memory, and the Host Geo Profile must be checked within the SEG.

13-32 Configuration of GSLB Sites (4)

466
VMware Confidential - Internal Only
13-33 Configuration of GSLB Sites (5)
Assign the DNS VS to the GSLB site

• Designates which DNS virtual service or virtual services will be used for GSLB purposes
(host FQDN records, answer queries, actively probe local VS)

• Must be done on all sites that will be ‘active’

13-34 Configuration of GSLB Sites (6)

Configure local application virtual services

• Configure a local VS like normal on each Avi Controller that it will participate in the GSLB
service

VMware Confidential - Internal Only 467

13-35 Configuration of GSLB Sites (7)
Designate the GSLB leader controller, and create the site configuration

• Choose one of the Controller clusters as the leader

• Perform the GSLB site configuration

13-36 Configuration of GSLB Sites (8)

468
VMware Confidential - Internal Only
13-37 Configuration of GSLB Sites (9)

VMware Confidential - Internal Only 469

13-38 Configuration of GSLB Sites (10)

13-39 Configuration of GSLB Sites (11)

470
VMware Confidential - Internal Only
13-40 Configuring a GSLB Service (1)
• A GSLB service is the representation of a global application deployed at multiple sites

• Defines the FQDN of the application, the backing virtual services in various sites, and the
priority/ratios governing selection of a particular VS

• Defines the health-monitoring methods by which unhealthy components can be identified so

that the best alternatives may be selected

13-41 Configuring a GSLB Service (2)

Scenario:

• A web application is hosted in Santa Clara, Virginia, and London

• LB algorithm is round-robin

• Health monitor is New-GSLB-Monitor

• If failure occurs, return all IPs

VMware Confidential - Internal Only 471

13-42 Configuring a GSLB Service (3)

472
VMware Confidential - Internal Only
13-43 Configuring a GSLB Service (4)

Click edit to add

application

VMware Confidential - Internal Only 473

13-44 Configuring a GSLB Service (5)

Type a unique name

for the GSLB Pool

Choose load
balancing Algorithm

Choose Virtual
Service to identify a
native Avi Vantage

Select a pre-configured
VS from the pulldown
list.

Select a pre-configured
VS from the pulldown
list.

474
VMware Confidential - Internal Only
13-45 Configuring a GSLB Service (6)
Add another VS from other Controller Cluster site to the GSLB pool

VMware Confidential - Internal Only 475

13-46 Configuring a GSLB Service (7)

476
VMware Confidential - Internal Only
13-47 Configuring a GSLB Service (8)

VMware Confidential - Internal Only 477

13-48 Configuring a GSLB Service (9)

478
VMware Confidential - Internal Only
Module 14
VMware NSX Advanced Load Balancer:
Troubleshooting

14-2 Module Lessons

1. Data Plane Versus Control Plane Troubleshooting

2. Data Plane Troubleshooting Overview

3. Control Plane Troubleshooting Overview

VMware Confidential - Internal Only 479

14-3 Lesson 1: Data Plane Versus Control
Plane Troubleshooting

14-4 Data Plane Versus Control Plane

Troubleshooting
Data Plane Troubleshooting:

• Service Engines are affected

• Application traffic affected / Application degraded

• Controller unable to spin up new SEs

• Controller unable to place Virtual Services

Control Plane Troubleshooting:

• Command and Control is affected, that is, Unable to access the UI, Cloud Connector issues,
and so on

• Cluster nodes impacted

SEs still continue to function in Headless mode even when the entire controller cluster is down.
New configuration is not possible, but the VSs already placed on the SEs continue to function
without issues.

480
VMware Confidential - Internal Only
14-5 Lesson 2: Data Plane Troubleshooting
Overview

14-6 Significant and Non-Significant Logs (1)

Significant Logs

• Comprise of common network and application errors

• HTTP Errors such as server or Vantage-generated 4xx and 5xx errors

• Networks errors such as canceled connections, abnormal latency or Out of order packets

• Vantage records all Significant logs by default

• Although not recommended, Significant logs may be disabled from the Analytics Profile
configuration

VMware Confidential - Internal Only 481

14-7 Significant and Non-Significant Logs (2)
Non-Significant Logs

• Logs that do not fall under the umbrella of Significant logs are classified as Non-Significant
Logs. Includes all successful transactions

• May be turned on for debugging purpose on a per-VS basis from the Virtual Service
Analytics tab

482
VMware Confidential - Internal Only
14-8 Analytics Profile (1)
• Provides a way to define what the typical end-user experience should be for an application

• Scope of Significant/Non-Significant logs generated by an Application is defined by the

Analytics Profile bound to it

• Used to determine the health score of a Virtual Service

14-9 Analytics Profile (2)

Client Response Apdex and Server Response Apdex under the HTTP Analytics section may be
used to influence the Performance Score of an HTTP/HTTPs Virtual Service.

Network Analytics section may be used to influence the Performance score of any Virtual
Service running in TCP Proxy mode.

VMware Confidential - Internal Only 483

14-10 Data Plane Troubleshooting Tools
• Client Logs

• Virtual Service Analytics

• Service Engine Logs

• Packet Capture

• CLI

484
VMware Confidential - Internal Only
14-11 Client Logs
• NSX Advanced Load Balancer logs TCP connections and HTTP requests/responses
information of Client-to-Application interactions

• Useful for troubleshooting and obtaining insights about the end-user experience

• Logs are indexed and viewed locally on the Controller

• Navigate to a Virtual Service from the Dashboard and click ’Logs’ tab to view all the Client
logs for that VS

• Client Logs

— Significant logs

— Full Clients (Non-Significant) logs

— All Request/Response Headers Logs

— Navigating logs using Search

14-12 Client Logs: Significant Logs (1)

• Network and application errors logs during client-to-application interactions

• Useful for troubleshooting issues experienced by clients while accessing the application VS

• To view these logs, On GUI Navigate to Application > Virtual Services, select the specific
virtual service name and click Logs tab, ensure that Significant logs is checked

VMware Confidential - Internal Only 485

14-13 Client Logs: Significant Logs (2)
Use Case: User is unable to access an https website (Virtual Service) but the same website is
accessible for other users.

• Administrator can apply client IP filters to Virtual Service logs search option.

• From the logs search result, find the logs when the problem occurred.

• In the sample log shown in the image, Client connection was closed because of disallowed
SSL protocol version used by client to establish the SSL session.

486
VMware Confidential - Internal Only
14-14 Client Logs: Significant Logs (3)
Use Case: User complains of an issue accessing an application at 10:43am

• The administrator can use the time window selection feature on the Client Logs to narrow
down the duration during which the issue was observed. In this instance, 73 significant logs
were generated in the 3min span when the issue was reported

VMware Confidential - Internal Only 487

14-15 Client Logs: Full Clients (Non-Significant)
Logs (1)
• Capture the “good” network connection and HTTP traffic

• Typical Client Log for a successful transaction will contain the following important
information:

— Client IP

— Virtual Service IP

— Service Engine (core)

— SE connection IP to the back end

— Back-end Server IP

— Request and Response information

— Response code

488
VMware Confidential - Internal Only
14-16 Client Logs: Full Clients (Non-Significant)
Logs (2)
Use Case

• Use the Client Log of a successfully completed transaction to identify the back-end server
which handled the request

• Navigate the Client Log of the successful transaction and click the ’+’ icon to expand the log.
In this instance, server [Link] handled the request

VMware Confidential - Internal Only 489

14-17 Client Logs: Navigating Logs Using
Search
• Filter Client Logs according to the specified search term

• Search term can be arbitrary [case insensitive] or based on a syntax

• To use syntax-based search, the search term can be typed manually or by clicking any blue
text in contextual help of the search field

• Syntax-based search combined with an operator will create the appropriate filter

• Multiple filters can be applied to further refine a search

• The filter can be a combination of arbitrary strings and syntax-based search terms

490
VMware Confidential - Internal Only
14-18 Client Logs: Log All Request/Response
Headers (1)
• A Virtual Service can be configured to log all Client connections/HTTP requests

• Full Client Logs option includes Significant logs, custom full log filters, and any logs
generated by customer policies or data scripts

• By default, a newly configured Virtual Service captures full client logs only for 30 mins after
which only the significant logs are captured

• Full Client Logs may also be enabled with the ’Client Log Filter’ option

• Full Client Logs are turned on from the Analytics tab of the Virtual Service

14-19 Client Logs: Log All Request/Response

Headers (2)
Use Case

• The Application profile assigned to a Virtual Service is configured to include both X-

Forwarded-For and X-Forwarded-Proto headers in the request to the back end. Use the
Log All headers option to verify that the request to the back end contains the configured
headers
• Navigate to the logs tab of the Virtual Service. Click the + icon to expand the client log

• Select the ’View All Headers’ option to display all the request and responses headers

VMware Confidential - Internal Only 491

14-20 Client Logs: Log All Request/Response
Headers (3)

492
VMware Confidential - Internal Only
14-21 Virtual Service Log Analytics (1)
• Log Analytics section in the Virtual Service logs displays a series of pre-built filters

• Summarize the clients logs on a pop-up according to the selected summary filter

• Log summaries reflect the currently applied filters, including the displayed log period and the
Significant/Non-significant log setting

14-22 Virtual Service Log Analytics (2)

Use Case

• Use Virtual Service Log Analytics to identify which Virtual Service has the greatest number
of HTTP 404 Errors

• From the VS Analytics, navigate to Response Code under the Request Analytics section
and select response code 404. With this prebuilt filter applied, navigate to Load Balancer
Analytics section and select VS IP Address to identify the subject VIP

VMware Confidential - Internal Only 493

14-23 Packet Capture (1)
How packet capture aids troubleshooting:

• In most cases, Virtual Service logs, Virtual Service Analytics and Log All Headers options are
good starting points to troubleshoot Application degradation issues

• Packet Capture serves as an additional tool if complete visibility into packet transmission is
needed for debugging

• Only captures Application traffic between the client and server

• When a Virtual Service is scaled out, traffic capture will run automatically on all SEs that host
the subject VS. When the capture is complete, the participating SEs forward their pcap files
to the controller which then aggregates and sorts the captures into a single file containing
the client and server exchanges

• Capture for a particular Virtual Service includes both Client-to-SE and SE-to-Server
communication

494
VMware Confidential - Internal Only
14-24 Packet Capture (2)
• Choose Client IP, IP Range or subnet mask: Captures traffic on the VS only specific to the
applied filter

• Duration: In minutes. Use ’0’ for infinite duration and manually stop the capture

• Size: In bytes. Use ’0’ to capture entire packet

• Advanced Settings: Using either of the health monitor options captures health monitor
traffic configured for all pools on those SEs

VMware Confidential - Internal Only 495

14-25 Packet Capture (3)

496
VMware Confidential - Internal Only
14-26 Advanced Topics: Traffic Cloning
• Stateless L2 cloning

• Conceptually, this feature mimics the behavior of a SPAN port

• The request is tapped on the interface from which the SE sends it to the application pool
server, hence, all the HTTP policies and DataScripts are first applied to the request before it
is replicated

• The response from application servers is also tapped on same interface, hence, it gets
replicated before the response policies or DataScripts are applied to it

VMware Confidential - Internal Only 497

14-27 Data Plane Troubleshooting with CLI (1)
Using the CLI for troubleshooting:

• It is possible to use CLI for identifying some common issues in the Data Plane

• Few of the basic commands for troubleshooting data plane issues are mentioned here

• Not all fields from the command outputs might be relevant to the troubleshooting

14-28 Data Plane Troubleshooting with CLI (2)

Common CLI commands for debugging Service Engine/ Datapath issues:

• Show serviceengine <se_name> cpu

• Show serviceengine <se_name> flowtablestat
• Show serviceengine <se_name> interface
• Show serviceengine <se_name> mallocstats
• Show serviceengine <se_name> mbufstats
• Show serviceengine <se_name> memdist
• Show serviceengine <se_name> meminfo
• Show servicengine <se_name> placement
• Show serviceengine <se_name> shmallocstats
• Show serviceengine <se_name> summary
• Show serviceengine <se_name>

498
VMware Confidential - Internal Only
14-29 Data Plane Troubleshooting with CLI (3)
Show serviceengine <se_name> cpu
• Total_linux_cpu_utilizaton: CPU utilization at host level

• Total_cpu_utilization: CPU utilization on the SE itself

• From the KB

VMware Confidential - Internal Only 499

14-30 Data Plane Troubleshooting with CLI (4)
Show serviceengine <se_name> interface
• Lists the stats for all the VNICs on the Service Engine

• Useful for identifying any input/output errors on any of the VNICs

500
VMware Confidential - Internal Only
14-31 Data Plane Troubleshooting with CLI (5)
Show serviceengine <se_name> flowtablestat
• Displays the flow table information per vNIC on the Service Engine

• Helpful in identifying connection/flow throttling issues

VMware Confidential - Internal Only 501

14-32 Data Plane Troubleshooting with CLI (6)
Show serviceengine <se_name> mbufstat

• Displays information on Buffer Utilization

• Buffers are Application Layer packet buffers

• Used to queue packets for improved network performance

• High latency connection on Client side and Low latency connection on Server side is the
most common use case for buffering server responses

• Issues such as Intermittent packet drops could be potentially because of buffer allocation
failures

502
VMware Confidential - Internal Only
14-33 Data Plane Troubleshooting with CLI (7)
Show serviceengine <se_name> memdist / Show serviceengine
<se_name> shmallocstats

• Displays information on Connection memory utilization

• Shared memory pool is divided into two components, Connections and Buffers

• Memory allocated to connections directly impacts the total concurrent connections a

Service Engine can maintain

• Configured on the Service Engine group ’Connection Memory Percentage’ slider. Takes
effect only on newly created SEs

VMware Confidential - Internal Only 503

14-34 Service Engine Logs
For offline analysis of the Service Engine logs, following outputs/logs can be captured:

• From the CLI shell of the controller, run the command show tech-support
serviceengine <se_name> to gather the SE tech-support logs. Successful
completion of this command will generate a tarball in the /tmp directory on the Controller

• Alternatively, files in /opt/avi/log on the Service Engine can be tarred manually. Use
attach serviceengine <se_name> command from the CLI shell of the controller
to log in to the required SE

• /opt/avi/archive directory consists of any cores generated during an SE crash

504
VMware Confidential - Internal Only
14-35 Health Monitor Troubleshooting (1)
• Servers within a pool may have down status because of health monitor failure

• The status is determined by the associated health monitors applied to the server pool

• NSX Advanced Load Balancer may mark down a server for several reasons

• The reason a server is marked down may be accessed three different ways

• Down Health Score Icon: Point to the server’s red status icon in the UI. In this example,
the “Custom-HM” monitor is marking down the server, because of Response code
mismatch.

• Server Page: Navigate to Applications > Pools > pool-name > Servers > server-name.
This displays the analytics page for the server. In this example, the Custom-HM monitor
is marking down the server

For further details, see [Link]

VMware Confidential - Internal Only 505

14-36 Health Monitor Troubleshooting (2)
• Down Event: Navigate to the events and filter for the SERVER_DOWN event. Expand the
event to see the full details.

• Manually Validate Server Health:

— It is often helpful to manually validate the response of a server while troubleshooting

reasons a server may be marked down

— Ensure that the test is from a specific Service Engine, using the same VS tenant.

• Ping

— Execute ping request to the back-end server from specific Service Engine and SE
namespace. In the example, you are getting ping response from server which means
server is physically UP. Next, you can check content checked in HM is available

For further details, see [Link]

506
VMware Confidential - Internal Only
14-37 Health Monitor Troubleshooting (3)
• Curl

— Execute CURL request to the back-end server from specific Service Engine and SE
namespace.

— Example: Assigned health monitor to pool is configured to check that

/[Link] content is available on the server. From manual curl test, you can
see the server replying with 404 Not Found response for /[Link] request,
while the server is replying with 200 OK for ‘/’ request. This means that monitored
content is not available on the server, because this server status will be marked down.

For further details, see [Link]

VMware Confidential - Internal Only 507

14-38 BGP Session Troubleshooting
How to check if a BGP session does not come up:

• Double-check the configuration on the router and Service Engine. Make sure the peer IPs,
subnets, and AS numbers are correct

• Verify that the MD5 passwords are the same on the router and Service Engine, if in use

• Invoke show serviceengine bgp to determine the state of the BGP session
initiated by the Service Engine

• Verify there are no ACLs/route-maps on the router preventing the

sessions/advertisements

• Additionally, if needed, check the packet capture using tcpdump (tcpdump -M ) on the
router and check BGP negotiations

• Access and Use Quagga Shell to check BGP configuration and status of BGP peer

508
VMware Confidential - Internal Only
14-39 BGP Session Troubleshooting: Quagga
Shell (1)
• Avi Vantage uses Quagga for BGP based scaling of virtual services

• To Access Quagga shell access SE, then switch to sudo mode

Verify BGP neighbor

session state

VMware Confidential - Internal Only 509

14-40 BGP Session Troubleshooting: Quagga
Shell (2)

Verify running BGP

configuration

510
VMware Confidential - Internal Only
14-41 BGP Session Troubleshooting: Quagga
Shell (3)

Datapath namespace

Login to quagga shell from SE

VMware Confidential - Internal Only 511

14-42 Lesson 3: Control Plane
Troubleshooting Overview

14-43 Control Plane Troubleshooting Overview

Control Plane issues can be broadly categorized into

• Cluster issues that is, One of the controller nodes has a problem

• UI unavailable / UI throws an error when making config changes

• Cloud Connector issues that is, configured cloud is down or unable to be discovered

• Log indexing, Metrics, Postgresql issues

Tools for Control Plane Troubleshooting

• Events/Audit Trail

• System Logs

• CLI

512
VMware Confidential - Internal Only
14-44 Events/Audit Trail (1)
Events are used throughout the system to provide a history of relevant changes

• Permanent record, can be used to generate alerts

• Events are also viewable in the context of specific objects, such as Virtual Service, a Pool, or
a Server

• Config Audit Trail provides an audit trail of user activity events and changes to the system
configuration over a period of time

14-45 Events/Audit Trail (2)

Use Case

• Use the Audit Trail feature to identify a configuration change and the user responsible for
the change

• Navigate to Operations > Events > Config Audit Trail. Select a CONFIG_* event and click
the + icon to view details of the relevant changes and the user who enforced it

VMware Confidential - Internal Only 513

14-46 System Logs
• Typical System operation logs

• Includes logs for configuration, monitoring, and management of the entire Vantage
infrastructure

• Not meant to be easily readable. Useful for Support and Engineering to troubleshoot any
issues reported

• Stored on each individual controller node

• Running the show tech-support debuglogs command on the CLI shell of the
leader node captures and collates system log files from all 3 controller nodes. This in
addition to the Service engine tech-support should encompass most of the information
needed for offline analysis

514
VMware Confidential - Internal Only
14-47 Cluster Issues
• Verify CPU, memory, and disk space on each of the 3 controller nodes manually using
top/htop and df –kh / du -sh commands
• Verify the space of /var/log folder. It is possible that [Link] and [Link]
files grow to huge size

• /opt/avi/log/cluster_manager.INFO file contains more information related to

controller clustering. Verify for any obvious errors here.

• show cluster status and show cluster nodes commands can be used from
the CLI shell of the leader node to verify that the cluster is up

VMware Confidential - Internal Only 515

14-48 Cloud Connector Issues (1)
vCenter Server Ecosystem Integration

• With vCenter Server integration, issues such as unavailable cloud [cloud in red status on UI],
Cloud not ready for VS placement or failure in the discovery of vCenter Server objects may
be observed

• vCenter Server ecosystem integration is handled by two processes: vcenter-mgr and

vi-mgr
• [Link], [Link], and vi_mgr.INFO contain system
logs related to vCenter Server cloud configuration and operation. Verify these logs for any
errors

• Use the following CLI to verify the vCenter Server login credentials: verifylogin
vcenter vcenter_url <vcenter_ip> username <username>
password <password>

14-49 Cloud Connector Issues (2)

vCenter Server Ecosystem Integration

• On occasion, the controllers and vCenter Server can lose sync

• To fix, enter rediscover vcenter cloud <cloud name> from the shell

• All other cloud integrations are handled by the Cloud_Connector process

• Configuration and operation logs are located under /opt/avi/log/cc_agent_<name>.log

516
VMware Confidential - Internal Only
14-50 Other Control Plane Debug Logs
• Portal Logs: /opt/avi/log/portal*.log

• Postgresql Logs: /var/log/postgresql/*.log

• NGINX Logs: /var/log/nginx/*.log

• Upstart Logs: /var/log/upstart

• For issues related to Vantage upgrades, debug logs can be generated using the command
show tech-support upgrade

VMware Confidential - Internal Only 517

VMware Confidential - Internal Only
Module 15
Events and Alerts

15-2 Module Lessons

1. Overview of Events

2. Overview of Alerts

3. Overview of SNMP

VMware Confidential - Internal Only 519

15-3 Lesson 1: Overview of Events

15-4 Introduction to Events (1)

Events - used throughout to provide a history of relevant changes that have occurred
systemwide.

• Events are a permanent record and can be used to generate alerts

• Can be viewed within the context of specific objects, such as a virtual service, a pool, or a
server

• In the debugging situation, this should be the first place to go for information

• Internal events are not shown and are less relevant for general purpose

15-5 Introduction to Events (2)

Over 490+ predefined events

• SE_DOWN, SE_CPU_HIGH, and SE_MEM_HIGH

• VS_UP and VS_DOWN

• VS_SCALOUT_COMPLETE

• GS_MEMBER_UP and GS_MEMBER_DOWN

• GSLB_SITE_SYNC_STATUS

Alerts can be configured with any event using Alert Config

520
VMware Confidential - Internal Only
15-6 Events Flow
• NSX Events are generated in the ecosystem

• Events are sent to the Controller database

• Events can be viewed through Operation > Events > All Events

VMware Confidential - Internal Only 521

15-7 Events Analytics: Object/EventID
• Events scope view can be narrowed down by:

— Object Type

— Object Name

— EventID

— Module

• Each one will generate a list:

— Number of occurrences

— % overall

522
VMware Confidential - Internal Only
15-8 Events Analytics: Timestamp
• Events scope view can be narrowed down by:

— Preset system duration

— Custom time range

— Click and Drag Zooming

VMware Confidential - Internal Only 523

15-9 Events Analytics: Searching
• Not case-sensitive

• Will match strings of object type/name and also any content within the event itself

• NOT or != are not supported

524
VMware Confidential - Internal Only
15-10 Events Analytics: Use Case for Virtual
Service DOWN
• TestVS indicate a Down State Issue

• Operation > Events > All Events

• Search on ObjectName of TestVS Reason: Pool issue

• Search on ObjectName of TestVS-pool Reason: Ping health monitor failure

VMware Confidential - Internal Only 525

15-11 Events Analytics: Use Case for
Configuration Change
• TestVS search will show changes

• Health monitor was changed from System-HTTP to System-Ping

526
VMware Confidential - Internal Only
15-12 Lesson 2: Overview of Alerts

15-13 About Alerts (1)

Alerts - intended to inform administrators of significant events within NSX Adv. LB (Avi
Networks). An alert defines the condition under which the action should be taken.

• Alerts can be triggered based any system Events

• Alerts can be triggered base on system metrics that are being tracked

• Alerts can take the following actions:

— Email Notification

— Syslog

— SNMP Trap

• Alerts can take automation actions:

— Perform AutoScaling automation

— Start ControlScript

VMware Confidential - Internal Only 527

15-14 About Alerts (2)
• Configuring custom alerts requires building an alert trigger

• When the alert is triggered, NSX Adv. LB processes the alert by referring to the associated
alert action

• Alerts are sent from the leader controller’s physical IP Address

— If a cluster IP is configured, the source of the alert will be the cluster IP

— If no cluster IP is configured, the source will be the leader controller so all three
controllers need to be allowed to send notifications through any necessary firewalls

528
VMware Confidential - Internal Only
15-15 Alerts View Scope
• Primary location - Operation > Alerts

• Alerts for specific objects under Virtual Services, Pools, Pool Groups and Service Engines

• Alerts are transient, and will only exist or be true for a defined period of time

• The underlying event or metric that triggered the alert still exists as a permanent log of what
has transpired

VMware Confidential - Internal Only 529

15-16 Alert Notifications
• Alerts can be sent out by using Syslog message, Email, or SNMP

• Multiple notifications can be configured within a single type

• Multiple destinations can be configured within a single notification

530
VMware Confidential - Internal Only
15-17 Alerts: Syslog Notifications Configuration
• Pushes messages are sent unencrypted through UDP datagram

• Multiple destinations can be defined

• Required information:

— Name: User defined

— Syslog Server: IP address of syslog server

— Port: Port used for syslog server

VMware Confidential - Internal Only 531

15-18 Alerts: Email Notifications Configuration
• Sends alerts to email addresses

• Requires valid DNS and gateway configuration

• Multiple email addresses can be defined separated by comma

• Required information

— Name – User defined

— To Address– Primary email address destinations

— CC Address – Secondary email address

532
VMware Confidential - Internal Only
15-19 Alerts: SNMP Notifications Configuration
• Sends SNMP Trap to SNMP Server

• Multiple SNMP Servers can be defined

• Support SNMPv2 or SNMPv3

• SNMPv3 supports:

— Auth Types: MD5 or SHA

— Privacy Settings : AES or DES

• Required information

— Name: User defined

— Trap Server IP Address: address to send SNMP Trap

— SNMP Version: Version supported

— SNMP Community: String configured on SNMP Server

VMware Confidential - Internal Only 533

15-20 Alerts Actions
• Alert Actions are configured by selecting Operations > Alerts > Alert Actions

• Alert Actions are called by an Alert Config when an alert has been triggered

• Alert Actions may be used to define the alert level, and notify administrators through Avi
Vantage UI (default), email, syslog, or SNMP traps.

• 6 Predefined system Alert Actions

534
VMware Confidential - Internal Only
15-21 Alerts Actions: Configuration
• Alert Actions are configured by selecting Operations > Alerts > Alert Actions

• Any single or multiple combinations of notifications can be configured

• Required Information:

1 Name: User defined

2 Alert Level: Low, Medium, High Severity

3 Email/Syslog/SNMPTrap/ControlScript: If left blank the Avi UI will be the

destination

VMware Confidential - Internal Only 535

15-22 Alert Config
• Alert Config is used to define the triggers that will generate an alert

• 11 predefined Default Alert Config.

• Defaults Alert Config can be modified or disabled but not deleted.

536
VMware Confidential - Internal Only
15-23 Alert Config: Configuration (1)
• Configuration is by selecting Operations > Alerts > Alert Config

• Required Information:

— Name: User defined

— Enable Alert Trigger: Enabled or Disabled after creation

— Source: Alert based on Events or Metrics

— Object: Object to monitor for triggers

VMware Confidential - Internal Only 537

15-24 Alert Config: Configuration (2)
• Configuration is by selecting Operations > Alerts > Alert Config

• Required Information:

— Instance: Select from drop-down. Populated based on Object selected

— Category: Alert raised immediately or after some time.

— Metrics/Events Occurs: Populated based on Source selected

— Alert Action: Action to take when Alert Config is triggered

538
VMware Confidential - Internal Only
15-25 Alert Config: Use Case
• Administrator needs to be notified NOC immediately when a specific VS is responding with
HTTP 5XX error code.

• Metric = L7_Client.AvgResp 5XX

• Object: TestVS Virtual Service

• Action: Email Action by GeneralNOCAction

• Enable: True

VMware Confidential - Internal Only 539

15-26 Lesson 3: Overview of SNMP

15-27 SNMP: Flow

• SNMP Polling uses UDP port 161

• SNMP GET query arrives at Controller SNMP daemon

• Verifies the community string for authorization

• Controller response with corresponding string

• Supports following PDU types:

— WALK, GET, GETNEXT, GETBULK, and GETSUBTREE

540
VMware Confidential - Internal Only
15-28 SNMP: Configuration v2
• Configured via Administrator > Settings > Access Settings

• SNMP v2 - simple but nonencrypted datagrams

• SNMP v2 settings for SNMP require only an SNMP community string

VMware Confidential - Internal Only 541

15-29 SNMP: Configuration v3
• Configured by using Administrator > Settings > Access Settings

• Provide authentication

• SNMP v3 encrypts the UDP datagram

• Requires EngineID: unique identifier of system

• Requires Auth and Privacy digest passphrase

542
VMware Confidential - Internal Only
Module 16
Introduction to Avi REST API

16-2 Module Lessons

1. Avi API Overview

2. Accessing Avi REST API

3. Swagger-Based API Documentation

4. SDK at a Glance

VMware Confidential - Internal Only 543

16-3 Lesson 1: Avi API Overview

16-4 Avi API Overview

• Supported Methods:

— GET, used to retrieve information

— POST, used to create objects

— PUT, used to update full objects

— DELETE, used to delete an object

— PATCH, used to edit a portion of an object

• Content-Type: application/json

• Session-based authentication:

— Login API to generate a session, indicated by sessionid cookie

— Logout API terminates the session

• Basic authentication on Avi Controller is disabled by default, it can be enabled through:

— Administration > Settings > Access Settings > Access Mechanisms > Basic
Authentication

[Link]

544
VMware Confidential - Internal Only
16-5 Avi API Objects (1)
• All objects have a UUID (universally unique identifier) that can be referenced

• In the UI, CLI, Ansible objects are always identified by a name:

— Names are unique within a tenant

— You cannot have two objects with the same name in a single tenant

• Lists of objects can be retrieved by:

— /api/<object-type>
— /api/<object-type>-inventory
• Specific objects can be retrieved by:

— /api/<object-type>/<object-uuid>
— /api/<object-type>?name=<object-name>+
This call will return a count, and a results array not the object by itself like directly with uuid. For
further details, see [Link]

16-6 Avi API Objects (2)

VMware Confidential - Internal Only 545

16-7 Avi API Object References (1)
• Objects may and often reference other objects

• Some objects may share these references, ex. ApplicationProfile, HealthMonitor

• References within an object are modeled as URIs and contain a _ref(s) suffix

• By default references do not include reference name:

— Query objects with ?include_name to have names appended to the references

— Names are appended behind # sign

— cloud_ref: [Link]
aba1-480b-ab62-22fc70f18396#Default-Cloud
For further details, see [Link]

16-8 Avi API Object References (2)

546
VMware Confidential - Internal Only
16-9 Avi API Tenancy and API Version
• Objects are associated with Tenants

• A single user may have access to multiple tenants

• Specific tenants can be managed by setting the X-Avi-Tenant header in the request

• In the UI, and CLI, switching tenant context automatically sets X-Avi-Tenant

• Users can only access objects or an object reference in another tenant if they have the
proper role for that tenant

• Avi API version supported for backward compatibility

— Request Header X-Avi-Version must be provided, if not provided default is 16.4.3

— POST and PUT requests will be rejected if data is not compatible with the specified
version

For further details, see [Link]

VMware Confidential - Internal Only 547

16-10 Lesson 2: Accessing Avi REST API

16-11 Accessing Avi REST API (1)

Reading JSON

• Browsers can be used to view Avi REST API objects, however JSON output is not parsed
by default, browsers require JSON view plugin (JSON Formatter, JSONView)

16-12 Accessing Avi REST API (2)

Reading JSON

• Pretty-print view of JSON in browsers can be done through plugins:

— Safari: JSON Formatter [Link]

formatter
— Chrome: JSONView
[Link]
nhfefbnpoihckbnefhakgolnmc?hl=en
— Firefox: JSONView [Link]
us/firefox/addon/jsonview/

548
VMware Confidential - Internal Only
16-13 Accessing Avi REST API (3)
Reading JSON with JSONView

VMware Confidential - Internal Only 549

16-14 Understanding API Usage (1)
Both the GUI and CLI are built on top of the API, which means every CLI command maps to a
corresponding API call(or calls) to be executed. Log into the CLI shell; set terminal
display_api-details:

For further details, see [Link]

550
VMware Confidential - Internal Only
16-15 Understanding API Usage (2)
Creating an object:

For further details, see: [Link]

guide/[Link]
POST: [Link]

VMware Confidential - Internal Only 551

16-16 Understanding API Usage (3)
Showing specific fields from an object:

GET [Link]

For further details, see: [Link]

guide/[Link]

552
VMware Confidential - Internal Only
16-17 Understanding API Usage (4)
Showing specific fields from all objects:

GET [Link]

For further details, see [Link]

guide/[Link]

VMware Confidential - Internal Only 553

16-18 Understanding API Usage (5)
Searching for objects by attribute, cannot search reference fields.

GET [Link]

For further details, see: [Link]

554
VMware Confidential - Internal Only
16-19 Understanding API Usage (6)
Object Modification:

PUT [Link]
c2b37c897ca7

For further details, see [Link]

guide/[Link]

VMware Confidential - Internal Only 555

16-20 Using PATCH API (1)
• Used to selectively edit scalar, list fields

• Ex. add/delete a server to/from a pool without modifying the entire object

— Add/edit a scalar field

— Unset a scalar field

— Add an entry to a list

— Edit an entry in a list

— Delete an entry from a list

• Limitations:

— Support edit of nested fields only at the top level

— Deep nested fields are not supported

For further details, see [Link]

patch-support/

16-21 Using PATCH API (2)

For further details, see [Link]

patch-support/

556
VMware Confidential - Internal Only
16-22 Using Macro API (1)
• Grouping Objects in a single transaction for an application with Macro API

• An Application can be created with multiple required objects

— virtualservice, pool, health monitor, http policies, ssl certificates, and so on

• Can be used for creations or deletions of top-level objects and objects bound to it

For further details, see [Link]

guide/[Link]

16-23 Using Macro API (2)

Creation of Virtual Service, pool, health monitor and application profile already exist.

VMware Confidential - Internal Only 557

16-24 Using Macro API (3)
• Macro API for deletions completes a calculation on which objects should be deleted

• Example: Top-level object Virtual Service created in previous example is now deleted:

— Virtual Service – Deleted

— Pool – Deleted

— Health Monitor – Deleted

— Application Profile – Used by other applications, so not deleted

Delete Virtual Service Object

558
VMware Confidential - Internal Only
16-25 Lesson 3: Swagger-Based API
Documentation

16-26 Swagger-Based API Documentation (1)

• Describes APIs

• Open API Spec [Link]

• Support in 9 languages, and more than 15 different frameworks

• Support for code generation

• Lists possible operations on object

• Shows CLI options

• Shows Object examples

• Generates CURL from Try it out feature

• Providers details on all parameters of an object

• Swagger specs are integrated into the Avi Controller

— [Link]
For further details, see [Link]
swagger-2-0-specification-integration/

VMware Confidential - Internal Only 559

16-27 Swagger-Based API Documentation (2)
Describes APIs by model reference

For further details, see [Link]

swagger-2-0-specification-integration

560
VMware Confidential - Internal Only
16-28 Swagger-Based API Documentation (3)
Some objects will include example objects

For further details, see [Link]

swagger-2-0-specification-integration

VMware Confidential - Internal Only 561

16-29 Swagger-Based API Documentation (4)
Generates CURL from the Try it out feature so that you can reuse what you created

For further details, see [Link]

swagger-2-0-specification-integration

562
VMware Confidential - Internal Only
16-30 Lesson 4: SDK at a Glance

16-31 Software Development Kit

• SDK in Python and Go

• Simplifies CSRF tokens and Authentication

• Further used with Terraform Provider and Ansible Modules

For further details, see [Link]

VMware Confidential - Internal Only 563

16-32 Ansible Modules/Roles
• Modules for individual object management

• Roles for added integrations and ease of configuration

• Easy to read as objects identical to API

564
VMware Confidential - Internal Only
16-33 Ansible Role avi_config Example
• Playbooks as code

• Roles to support integrations or simplify tasks

• Avi_config uses benefits of modules with added benefits"

— Idempotent

— Easy iteration, picks up at last failure

— Check mode, no changes made

— Audit trail

— Repeatability

For further information on Modules:

[Link]
k_modules.html#avi
and for Roles:
[Link]

VMware Confidential - Internal Only 565

16-34 Terraform
• Officially supported Terraform Provider

• Based on GO SDK

• Deploy and destroy with ease

For further information on Modules:

[Link]
k_modules.html#avi
and for Roles:
[Link]

566
VMware Confidential - Internal Only
VMware Confidential - Internal Only