Network Analysis While Preserving

Privacy*
Karl F. Lutzen
Information Security Officer
kfl@[Link]

*meaning contents of transmission

1
 Introduction
• Privacy concerns today
• Analyzing traffic usually is done by examining
packets – Deep packet inspection
• Looking at “calling information” can reveal
much
• Can be used as an IDS
• Can be use as policy enforcement

2
 “Calling Information”
• Source IP address and port
• Destination IP address and port
• Protocol
• Other data
– Timestamps
– Number of packets
– Bytes
• Technology used: Netflow
3
 NetFlow Background

• Developed by Cisco to:

– Characterize traffic
– Account for how and where it flows
– Help optimize network investment
– Traffic engineering/network planning
– Provide usage-based billing

4
 NetFlow Background

• Three key characteristics:

– Be scalable
– Be manageable
– Be reliable

5
 NetFlow Example
Computer A Web browses to Computer B will
generate 2 flows:
•Request flow:
– A: (TCP) [Link]:3365 -> [Link]: 80
•Reply Flow:
– B: (TCP) [Link]:80 -> [Link]:3365

6
 NetFlow Typical Record
• Source and destination IP • TCP flags
address • Routing information (next-
• Source and destination ports hop address, source
• Transport protocol: TCP,UDP, autonomous system (AS)
ICMP, etc. number, destination AS
• Type of service (ToS) number, source prefix mask,
destination prefix mask)
• Packet and byte counts
• Start and end timestamps
• Input and output interface
numbers

7
 NetFlow Data Cache
• Available on Cisco routers/switches
• Available on Juniper routers
• Cached on devices

• WARNING! Not all devices are NetFlow-

enabled!

8
 NetFlow Cache Example
#show ip cache flow
IP packet size distribution (78630M total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.002 .448 .062 .027 .013 .011 .008 .011 .003 .003 .002 .006 .005 .003 .002
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.002 .003 .015 .033 .331 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 6553988 bytes
32929 active, 32607 inactive, 524367786 added
4111490554 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 794824 bytes
32895 active, 16257 Inactive, 519171584 added, 519168554 added to flow
0 alloc failures, 12911870 force free
3 chunks, 1155 chunks added
last clearing of statistics never
--More—

9
 NetFlow Cache Example (cont.)
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-Telnet 3833510 0.8 10 179 9.2 9.0 26.8
TCP-FTP 12511306 2.9 6 132 19.7 6.3 16.5
TCP-FTPD 1194796 0.2 544 866 151.5 86.7 21.2
TCP-WWW 944754736 219.9 13 627 2871.0 3.2 23.7
TCP-SMTP 53320030 12.4 14 399 185.8 6.6 19.2
TCP-X 913841 0.2 41 631 8.9 19.2 24.5
TCP-BGP 1867 0.0 1 49 0.0 0.5 20.5
TCP-NNTP 1086658 0.2 252 874 63.8 15.2 26.8
TCP-Frag 228697 0.0 9 131 0.5 6.5 25.3
TCP-other 2264274585 527.1 23 568 12466.6 12.9 24.4
UDP-DNS 231113128 53.8 2 79 114.7 3.6 26.0
UDP-NTP 6394017 1.4 3 76 5.2 9.7 27.2
UDP-TFTP 13567 0.0 1 95 0.0 3.1 29.3
UDP-Frag 211973 0.0 3165 1266 156.2 116.2 27.5
UDP-other 1177902953 274.2 5 293 1385.8 6.1 25.8
ICMP 103453714 24.0 2 62 57.9 3.8 26.0
IGMP 726 0.0 2 300 0.0 3.5 29.1
IPINIP 1 0.0 1 40 0.0 0.0 27.2
GRE 10272668 2.3 309 774 740.5 35.3 23.1
IP-other 544060 0.1 533 484 67.5 123.7 22.6
Total: 4812026833 1120.3 16 567 18305.6 8.7 24.7
--More—

10
 NetFlow Limitations of Cache
• Difficult to read
• Only shows recent activity
• No automation on devices for analysis
• No accounting of flows (besides overall totals)

11
 NetFlow Export of Data
• Greatly enhances NetFlow and turns the
technology into a analysis tool!
• Data sent to external collector(s)
• Analyzed by one or more systems
• Archived for other concerns
• Efficient: Uses multiple records per UDP
packet

12
 NetFlow Export: Establish Policies!

• Ensure policies are in place before deploying

covering:
– Retention of network usage statistics
– Establish a retention policy.
– Privacy protection of the data, who is authorized,
no offloading without sanitizing personal data (the
host portion)

13
 Privacy
• While the contents of the packet are not
recorded, the calling information can still be a
concern.
• However, with virtual servers, it is impossible
to know the true destination
• Mostly it can only be used as verification that
something occurred.

14
Deployment Diagram
NetFlow Device 1 NetFlow Device 2

Network

Netflow Collector

Analysis Station 1 Analysis Station 2 Analysis Station n

Typical NetFlow Collector Setup

15
 Securing The Data Stream
• Hmm, hold on there!
• Using UDP, any Security in the transmission?
– Nope!
• Most deployments use a private, maintenance
network that only admins can touch.

16
 Introduction to flow-tools

• Full featured NetFlow tool set

• Open-source software
• Available from:
[Link]
• Entire package compiles on most Linux,
FreeBSD, etc. systems

17
 Flow-tools Contains
• flow-capture • flow-merge
• flow-cat • flow-nfilter
• flow-dscan • flow-print
• flow-expire • flow-receive
• flow-export • flow-report
• flow-fanout • flow-send
• flow-filter • flow-split
• flow-gen • flow-stat
• flow-header • flow-tag
• flow-import • flow-xlate
• flow-mask

18
 Single Source to Many Collectors

flow-fanout
• Inline replacement for flow-capture
• Replicates a NetFlow stream to multiple
locations
• Ideal for simultaneous:
– Replicated storage
– Multiple systems for near real time analysis

19
 No NetFlow? Use fprobe
• [Link]
• Open source NetFlow probe
• Linux, FreeBSD, etc.
• Uses a SPAN port or network tap
• Consider libpcap-ring kernel or MMAPed pcap for
high-speed collections (over 100 Mbits)
• Can match CPU to traffic load
• Downside: loss of interface on reports

20
 Portable Probe
Build a couple of portable probes to have on
hand for remote probes for network analysis.
Install flow-tools, fprobe, tcpdump on a system
with a large hard drive.

When a problem occurs, you can deploy it on a

SPAN port or with a hub or network TAP.

21
 Traffic Analysis: Know Thy Network!
• NetFlow records the communication between
systems
• Quickly tells you what is happening on your
network at a high level
• Can be used to spot anomalies
• Simple IDS capabilities
• Locate all stations doing the same thing on the
network
• Policy enforcement

22
 Knowing Where to Look
• Deploy NetFlow on devices at keys location
– Border of network(s)
– Core points
• Deploy probes where needed
– Specific problem areas
– Network aggregation points without NetFlow
capabilities.

23
 Planning/Policies Make for Success

• Establish policies as to what traffic is allowed

• Establish specific pathways or gateways for
traffic like SMTP, IRC, HTTP, etc.
• Any traffic not flowing through these
gateways are your indicator for problems
• Segregate servers and workstations with
subnets.

24
 Analysis: Finding the Needles
• Which State?
• Which County?
• Which Field?
• Which haystack?
• What part of the haystack?
• How many needles?

25
 Finding the Needle(s): flow-stat
• The flow-stat command provides quick
statistics
• Can provide reports on SRC/DST IP’s or
ports, as well as others
• Can sort by flows, octets or packets in
ascending or descending order
• Coupled with flow-nfilter removes known
good traffic or look known problem traffic

26
 Using flow-stat
flow-stat –f format –S sort field < filename
Report format. Sort format:
0 Overall Summary -s (lower case) sort field ascending
1 Average packet size distribution -S (upper case) sort field descending
2 Packets per flow distribution
3 Octets per flow distribution Remember that the first field or column will
4 Bandwidth per flow distribution be “0”, not “1”!
5 UDP/TCP destination port
6 UDP/TCP source port
7 UDP/TCP port
8 Destination IP
9 Source IP
10 Source/Destination IP
11 Source or Destination IP

27
 Stats sorted by flows
# --- ---- ---- Report Information --- --- ---
#
# Fields: Total
# Symbols: Disabled
# Sorting: Descending Field 1
# Name: Source IP
#
# Args: flow-stat -f9 -S1
#
#
# IPaddr flows octets packets
#
[Link] 8676 3579254 26397
[Link] 5442 107280977 131352
[Link] 4943 1570499 8382
[Link] 4572 514276 10864
[Link] 4374 9645573 57475
[Link] 4176 8924188 11501
[Link] 4110 28260371 42063
[Link] 4057 29485053 29288
28
 Filtering traffic: flow-nfilter
filter-primitive test filter-primitive servers
type ip-port type ip-address-prefix
permit 135 deny 131.151.0/23
permit 139 deny 131.151.2/24
permit 445 default permit
# or permit 135,139,445
default deny filter-definition ms-scan
match dst-ip-port test
filter-primitive ipok match src-ip-addr ipok
type ip-address match src-ip-addr ip
deny [Link] match dst-ip-addr
default permit servers
match src-ip-addr
servers
filter-primitive ip
type ip-address-prefix
permit 131.151/16
default deny

29
 Sasser Worm Example
flow-nfilter -f ms-scan -F ms-scan < ft-v07.2004-05-
02.155141-0500 flow-stat -f9 -S1 | more
# IPaddr flows octets packets
[Link] 2380 860065 8302
[Link] 2368 807897 8150
[Link] 2365 797034 8165
[Link] 2357 782567 8032
[Link] 2353 787680 8045
[Link] 2337 662575 7775
[Link] 2329 843737 7848
[Link] 2329 745276 7840
[Link] 2321 653594 7697
[Link] 2317 723814 7755
[Link] 2317 640136 7640
[Link] 2311 641091 7629
[Link] 2301 623940 7568
[Link] 2298 596356 7519
[Link] 2297 789097 7801

30
 Netflow Stats for TCP 445 during Sasser Worm at UMR May 2004

600000

500000
Number of Netflows (excluding data center)

400000

300000

200000

100000

0
5/2/04 0:00 5/3/04 0:00 5/4/04 0:00 5/5/04 0:00 5/6/04 0:00

31
 Netflow Stats for TCP 445 during Sasser Worm at UMR May 2004

350

300

250
Number of Infected Hosts

200

150

100

50

0
5/2/04 0:00 5/3/04 0:00 5/4/04 0:00 5/5/04 0:00 5/6/04 0:00

32
 Profile of a Worm in NetFlow
• Can use different protocols
• High flow count
• Low packet count – 3 packets or less per flow
• Use flow-stat or flow-report to spot them
Downside: If the stations generate other
traffic, it can obscure the worm activity

33
 Flow File Size Can Tell a Story
• Always keep an eye on the NetFlow file sizes
• Works best after a baseline of a few days or
weeks of observation.
• General fluctuations are normal traffic
patterns, but a sudden surge indicates
something new is going on.
• Sudden drops could indicate network
problems.

34
 Pig in the Python
-rw-r--r-- 1 netflow afsuser 2019685 May 2 15:21 ft-v07.2004-10-02.151649-0500
-rw-r--r-- 1 netflow afsuser 2031767 May 2 15:26 ft-v07.2004-10-02.152148-0500
-rw-r--r-- 1 netflow afsuser 2032419 May 2 15:31 ft-v07.2004-10-02.152647-0500
-rw-r--r-- 1 netflow afsuser 2072933 May 2 15:36 ft-v07.2004-10-02.153145-0500
-rw-r--r-- 1 netflow afsuser 2062822 May 2 15:41 ft-v07.2004-10-02.153645-0500
-rw-r--r-- 1 netflow afsuser 2120842 May 2 15:46 ft-v07.2004-10-02.154144-0500
-rw-r--r-- 1 netflow afsuser 7013906 May 2 15:51 ft-v07.2004-10-02.154643-0500
-rw-r--r-- 1 netflow afsuser 11331622 May 2 15:56 ft-v07.2004-05-02.155141-0500
-rw-r--r-- 1 netflow afsuser 15607255 May 2 16:01 ft-v07.2004-05-02.155640-0500
-rw-r--r-- 1 netflow afsuser 15748046 May 2 16:06 ft-v07.2004-05-02.160139-0500
-rw-r--r-- 1 netflow afsuser 14216272 May 2 16:11 ft-v07.2004-05-02.160638-0500
-rw-r--r-- 1 netflow afsuser 12008287 May 2 16:16 ft-v07.2004-05-02.161137-0500
-rw-r--r-- 1 netflow afsuser 9972353 May 2 16:21 ft-v07.2004-05-02.161636-0500
-rw-r--r-- 1 netflow afsuser 8973700 May 2 16:26 ft-v07.2004-05-02.162135-0500
-rw-r--r-- 1 netflow afsuser 9042363 May 2 16:31 ft-v07.2004-05-02.162635-0500
-rw-r--r-- 1 netflow afsuser 8017690 May 2 16:36 ft-v07.2004-05-02.163133-0500
-rw-r--r-- 1 netflow afsuser 8109288 May 2 16:41 ft-v07.2004-05-02.163632-0500
-rw-r--r-- 1 netflow afsuser 7301829 May 2 16:46 ft-v07.2004-05-02.164131-0500
-rw-r--r-- 1 netflow afsuser 7175834 May 2 16:51 ft-v07.2004-05-02.164630-0500
-rw-r--r-- 1 netflow afsuser 7029650 May 2 16:56 ft-v07.2004-05-02.165129-0500
-rw-r--r-- 1 netflow afsuser 7063063 May 2 17:01 ft-v07.2004-05-02.165628-0500

35
 NetFlow: Email Virus Detection
• Systems infected with Email viruses can be
detected via NetFlow due to:
– Multiple mail messages per host in the same flow
file (over 15 messages in 5 min)
– Mail going directly to the border instead of
authorized servers (requires policies).
• Policy enforcement example!

36
 Typical Email Virus
flow-cat ft-v07.2004-11-18.190* | flow-nfilter -f ~/kfl/emailblocked
-Femail | flow-print -f3|more
srcIP dstIP prot srcPort dstPort octets
packets
[Link] [Link] 6 1148 25 144 3
[Link] [Link] 6 1150 25 144 3
[Link] [Link] 6 1152 25 144 3
[Link] [Link] 6 1154 25 144 3
[Link] [Link] 6 1156 25 144 3
[Link] [Link] 6 1158 25 144 3
[Link] [Link] 6 1160 25 144 3
[Link] [Link] 6 1162 25 144 3
[Link] [Link] 6 1165 25 144 3
[Link] [Link] 6 1167 25 144 3
[Link] [Link] 6 1169 25 144 3
[Link] [Link] 6 1171 25 144 3
[Link] [Link] 6 1174 25 144 3
[Link] [Link] 6 1176 25 144 3
[Link] [Link] 6 1179 25 144 3
[Link] [Link] 6 1181 25 144 3
[Link] [Link] 6 1184 25 144 3
[Link] [Link] 6 1186 25 144 3
[Link] [Link] 6 1189 25 144 3
[Link] [Link] 6 1191 25 144 37 3
 Email checks: Not all are bad
flow-print –f3 < last | grep [Link] | grep " 25 6 “
srcIP dstIP prot srcPort dstPort octets packets
[Link] [Link] 6 3828 25 96 2
[Link] [Link] 6 3842 25 96 2
[Link] [Link] 6 3852 25 96 2
[Link] [Link] 6 3867 25 96 2
[Link] [Link] 6 3879 25 96 2
[Link] [Link] 6 3897 25 96 2
[Link] [Link] 6 3912 25 96 2
[Link] [Link] 6 3922 25 96 2
[Link] [Link] 6 3934 25 96 2
[Link] [Link] 6 3943 25 96 2
[Link] [Link] 6 3957 25 96 2
[Link] [Link] 6 3972 25 96 2
[Link] [Link] 6 4003 25 96 2
[Link] [Link] 6 4018 25 96 2
[Link] [Link] 6 4035 25 96 2
[Link] [Link] 6 4053 25 96 2
[Link] [Link] 6 4072 25 96 2
[Link] [Link] 6 4091 25 96 2
[Link] [Link] 6 4105 25 96 2

All destinations are Hotmail addresses: configuration issue

38
 Controlled Mass Mailer
flow-nfilter -f ~/kfl/emailblocked -F email < ft-v07.2004-11-17.220025-0600 | \
flow-print -f3
srcIP dstIP prot srcPort dstPort octets packets
[Link] [Link] 6 3281 25 144 3
[Link] [Link] 6 3282 25 144 3
[Link] [Link] 6 3283 25 144 3
[Link] [Link] 6 3284 25 144 3
[Link] [Link] 6 3285 25 144 3
[Link] [Link] 6 3286 25 144 3
[Link] [Link] 6 3287 25 144 3
[Link] [Link] 6 3288 25 144 3
[Link] [Link] 6 3289 25 144 3
[Link] [Link] 6 3290 25 144 3
[Link] [Link] 6 3291 25 144 3
[Link] [Link] 6 3292 25 144 3
[Link] [Link] 6 3293 25 144 3
[Link] [Link] 6 3294 25 144 3
[Link] [Link] 6 3295 25 144 3
[Link] [Link] 6 3296 25 144 3
[Link] [Link] 6 3297 25 144 3
[Link] [Link] 6 3298 25 144 3
[Link] [Link] 6 3299 25 144 3
[Link] [Link] 6 3300 25 144 3
[Link] [Link] 6 3301 25 144 3
[Link] [Link] 6 3302 25 144 3

39
 Multiple Control Hosts
flow-print -f3 < ft-v07.2004-11-17.220025-0600 | grep [Link] |
more
[Link] [Link] 6 3281 25 144 3
[Link] [Link] 6 1438 12942 217 5
[Link] [Link] 6 12942 1438 168 4
[Link] [Link] 6 3282 25 144 3
[Link] [Link] 6 3900 12942 217 5
[Link] [Link] 6 12942 3900 168 4
[Link] [Link] 6 3283 25 144 3
[Link] [Link] 6 4559 12942 217 5
[Link] [Link] 6 12942 4559 168 4
[Link] [Link] 6 3284 25 144 3
[Link] [Link] 6 3570 12942 261 6
[Link] [Link] 6 12942 3570 210 5
[Link] [Link] 6 3285 25 144 3
[Link] [Link] 6 1863 3010 356 1
[Link] [Link] 6 3010 1863 40 1
[Link] [Link] 6 2857 12942 261 6
[Link] [Link] 6 12942 2857 210 5
[Link] [Link] 6 3286 25 144 3

40
 Traffic Analysis: Worm?
flow-nfilter -f ms-scan -F ms-scan < ft-v07.2004-10-25.141702-0500 | \
flow-stat -f9 -S1 | more
# --- ---- ---- Report Information --- --- ---
#
# Fields: Total
# Symbols: Disabled
# Sorting: Descending Field 1
# Name: Source IP
#
# Args: flow-stat -f9 -S1
#
#
# IPaddr flows Octets packets
#
[Link] 3365 186336 3882
[Link] 137 89054 654
[Link] 66 1913668 21807
[Link] 41 60142 734
[Link] 32 12131 93
[Link] 31 27468 213
[Link] 20 1247965 26429
[Link] 19 315891 2558
[Link] 18 197144 1630
[Link] 17 36404 200
[Link] 16 100253 1414
<snip>

41
 Traffic Analysis: Zeroing In
flow-print -f3 < ft-v07.2004-10-25.141702-0500 | grep [Link] | more
[Link] [Link] 6 1332 445 96 2
[Link] [Link] 6 1333 445 96 2
[Link] [Link] 6 1334 445 96 2
[Link] [Link] 6 1336 445 144 3
[Link] [Link] 6 1338 445 144 3
[Link] [Link] 6 1339 445 144 3
[Link] [Link] 6 1340 445 144 3
[Link] [Link] 6 1341 445 144 3
[Link] [Link] 6 1343 445 96 2
[Link] [Link] 6 1246 445 96 2
[Link] [Link] 6 1344 445 144 3
[Link] [Link] 6 1345 445 96 2
[Link] [Link] 6 1347 445 144 3
[Link] [Link] 6 1348 445 96 2
[Link] [Link] 6 1349 445 144 3
[Link] [Link] 6 1286 445 96 2
[Link] [Link] 6 1351 445 144 3
[Link] [Link] 6 1352 445 96 2
[Link] [Link] 6 1297 445 815 7
[Link] [Link] 6 1353 445 8765 15
[Link] [Link] 6 1318 445 48 1
[Link] [Link] 6 1322 445 48 1
--More--

42
 Traffic Analysis: Check the Border
flow-cat ft-v07.2004-10-25.141* | flow-print –f3 | grep [Link]
srcIP dstIP prot srcPort dstPort octets packets
[Link] [Link] 6 3828 80 9173 15
[Link] [Link] 6 80 3828 38860 72
[Link] [Link] 6 2127 80 1516 7
[Link] [Link] 6 80 2127 17834 32
[Link] [Link] 6 2523 80 7872 123
[Link] [Link] 6 80 2523 287793 374
[Link] [Link] 6 1034 6667 11291 269
[Link] [Link] 6 6667 1034 42958 290
[Link] [Link] 6 6667 1034 105 1

Turns out that this was an IRC controlled Bot: [Link].j

43
 Situation: IFRAME Exploit
• System suddenly generated a virus warning after
visiting a well known, trusted website.
• System scan removed the known virus and
downloader, but an undetectable trojan was
downloaded during the event.
• Trojan NOT detectable after virus definition update
and full system scan.
• System now displays ads and runs very slow
• Analysis of system required. Noted traffic involving
Netherlands IP address.

44
 IFRAME Exploit: Examining traffic
flow-cat ft-v07.2004-11-18.08* | flow-print –f3 | grep " 62\.4\.84“

srcIP dstIP prot srcPort dstPort octets packets

[Link] [Link] 6 3585 80 1106 23
[Link] [Link] 6 3585 80 42562 34
[Link] [Link] 6 3586 80 12637 313
[Link] [Link] 6 80 3586 879907 590
[Link] [Link] 6 3587 80 1633 7
[Link] [Link] 6 80 3587 2257 6

• We knew approximate time of the event.

• Search on the network portion of the IP address in question.
• Three systems on foreign network are involved in the exploit.
• Banned IP range to contain problem.
• Now we can search an entire day’s logs to find the number of infected
systems.

45
 Create “iframe” Filter
filter-primitive test-address
type ip-address
permit [Link]
permit [Link]
permit [Link]
default deny

filter-primitive test-protocol
type ip-protocol
permit tcp

filter-definition iframe
match dst-ip-addr test-address
match ip-protocol test-protocol

46
 Run iframe Filter
flow-cat f* | flow-nfilter -f iframe -F iframe | flow-print -f3 > \
~/out

Then run this output through awk, sort and grep:

awk '{print $1}' ~/out | sort –u | grep 131.151
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
<snip>

47
 Spot Who is Using Services
• Netflow is very useful for determining
– Who is using various services
– Impact on closing down ports
– Location of servers

48
 Other Types of Detection
• Spyware
• Verify claims on traffic from your network
– DMCA reports
– Attacks reports
– Scanning reports
– Email – spoofed or real
• Can aid with determining access controls and
Firewall rules – null interface for dropped
traffic
49
 Other Links
• Cisco: [Link]
• flow-tools home: [Link]
• Great selection of links for various NetFlow tools:
[Link]
• Fprobe source: [Link]
• Libcap-ring, nprobe, ntop: [Link]
• Well known IP ports (very good reference for analysis) :
[Link]

If you ever have any questions of any sort, please email me at kfl@[Link].

50