|
Question : Internet Usage Overages
|
|
I support a company that keeps nailed with internet usage overage bills. There doesn't appear to be any malware or P2P sharing going on. I can't find where the overage is coming from and it's driving me crazy! They have an ADSL connection with a Fortinet 60 firewall and a site to site VPN connection to their DC and exchange server. Any ideas?
|
Answer : Internet Usage Overages
|
|
Introduction
All FortiGate units have a powerful packet sniffer on board. If you know tcpdump you should feel comfortable using the FortiGate Sniffer.
Additional sniffer tips can be found in the Fortinet Knowledge Center article "Packet capture (sniffer) tips".
This document is based on FortiOS v2.80-MR5 and later firmware.
DMZ
|
|
+-----------+
----internal----| FortiGate |---external-----
+-----------+
Sniffer Basics
The packet sniffer "sits" in the FortiGate and can sniff traffic on a specific Interface or on all Interfaces. There are 3 different Level of Information, a.k.a. Verbose Levels 1 to 3, where verbose 1 shows less information and verbose 3 shows the most information.
Verbose levels in detail:
1: print header of packets
2: print header and data from IP of packets
3: print header and data from Ethernet of packets
We'll go through some examples and different levels of verbosity to show the different possibilities you have for debugging.
Basic sniffing command
All Packetsniffing commands start like:
# diag sniffer packet <'filter'>
Where...
can be an Interface name or "any" for all Interfaces.
means the level of verbosity as described already.
the number of packets the sniffer reads before stopping.
<'filter'> is a very powerful filter functionality which will be described in more detail.
Example 1: Simple Trace
Sniff 3 packets of all traffic with verbose Level 1 on internal Interface
# diag sniffer packet internal none 1 3
192.168.0.1.22 -> 192.168.0.30.1144: psh 2859918764 ack 1949135261
192.168.0.1.22 -> 192.168.0.30.1144: psh 2859918816 ack 1949135261
192.168.0.30.1144 -> 192.168.0.1.22: ack 2859918884
As you can see we caught some Packets in the middle of a communication. Because the 192.168.0.1 IP Address uses Port 22 (192.168.0.1.22) we can assume that we've caught some Packets from a running SSH Session. The "none" variable means 'no filter applies', "1" means 'verbose 1' and "3" means 'catch 3 packets and stop'.
Example 2: Simple Trace
Sniff 3 packets of all traffic with verbose Level 1 on Internal interface
# diag sniffer packet internal none 1 3
192.168.0.30.1156 -> 192.168.0.1.80: syn 2164883624
192.168.0.1.80 -> 192.168.0.30.1156: syn 3792179542 ack 2164883625
192.168.0.30.1156 -> 192.168.0.1.80: ack 3792179543
Apparently we caught some more interesting information, just when a TCP session was being set up. 192.168.0.30 tries to connect to 192.168.0.1 on Port 80 with a syn and gets a syn ack back. Finally the session is acknowledged and established after the 3-way TCP handshake.
With information level set to Verbose 1, we see Source and Destination IP Address, as well as Source and Destination Port. We can also see the corresponding Sequence numbers.
If you don't enter a value, the Sniffer runs forever until you Stop it with
Hint: For further investigation it's always a good idea to log to a file. If you're using Putty (a free SSH client for Windows) you can easily log all Output to a file which you can search/sort/process.
Verbose 2 and Verbose 3 levels:
Verbose 2 contains much more information
1. The IP Header as we've already seen in Verbose 1
2. The Payload of the IP packet itself
An Output of Verbose 2 looks like this:
# diag sniffer packet internal none 2 1
192.168.0.1.22 -> 192.168.0.30.1144: psh 2867817048 ack 1951061933
0x0000 4510 005c 8eb1 4000 4006 2a6b c0a8 0001 E..\..@.@.*k....
0x0010 c0a8 001e 0016 0478 aaef 6a58 744a d7ad .......x..jXtJ..
0x0020 5018 0b5c 8ab9 0000 9819 880b f465 62a8 P..\.........eb.
0x0030 3eaf 3804 3fee 2555 8deb 24da dd0d c684 >.8.?.%U..$.....
0x0040 08a9 7907 202d 5898 a85c facb 8c0a f9e5 ..y..-X..\......
0x0050 bd9c b649 5318 7fc5 c415 5a59 ...IS.....ZY
Verbose 3, finally, even includes Ethernet (Ether Frame) Information. This is the format that technical support will usually request when attempting to analyze a problem.
A script is available (fgt2eth.pl), which will convert a captured verbose 3 output, into a file that can be read and decoded by Ethereal. See the end of this article for details.
Filter Functionality
As already mentioned: diag sniffer includes a powerful filter functionality that will be described here.
FortiOS tells us:
filter for sniffer
Syntax: '[[src|dst] host] [[src|dst] host] [[arp|ip|gre|esp|udp|tcp] [port_no]] [[arp|ip|gre|esp|udp|tcp] [port_no]]'
If a second host is specified, only the traffic between the 2 hosts will be displayed.
flexible logical filters for sniffer (or "none").
For example: To print udp 1812 traffic between forti1 and either forti2 or forti3
'udp and port 1812 and host forti1 and \( forti2 or forti3 \)'
Imagine you only want to sniff the traffic from one PC to another PC. Without Filter the sniffer will display all packets which is far too much and painful to debug.
Example 3: Trace with Filters
To see what's going on between two PCs (or a PC and a FortiGate),(Don't forget to put your filter expressions in single quotes ' ' ):
# diag sniffer packet internal 'src host 192.168.0.130 and dst host 192.168.0.1' 1
192.168.0.130.3426 -> 192.168.0.1.80: syn 1325244087
192.168.0.1.80 -> 192.168.0.130.3426: syn 3483111189 ack 1325244088
192.168.0.130.3426 -> 192.168.0.1.80: ack 3483111190
192.168.0.130.3426 -> 192.168.0.1.80: psh 1325244088 ack 3483111190
192.168.0.1.80 -> 192.168.0.130.3426: ack 1325244686
192.168.0.130.1035 -> 192.168.0.1.53: udp 26
192.168.0.130.1035 -> 192.168.0.1.53: udp 42
192.168.0.130.1035 -> 192.168.0.1.53: udp 42
192.168.0.130 -> 192.168.0.1: icmp: echo request
192.168.0.130.3426 -> 192.168.0.1.80: psh 1325244686 ack 3483111190
192.168.0.1.80 -> 192.168.0.130.3426: ack 1325244735
192.168.0.130 -> 192.168.0.1: icmp: echo request
Assuming there is a lot of traffic on the wire, this filter command will only display traffic (but all traffic) from Source 192.168.0.130 to Destination 192.168.0.1. It will NOT show Traffic TO 192.168.0.130 (for example the ICMP reply) because we said:
'src host 192.168.0.130 and dst host 192.168.0.1'
As you can see we also captured some other things like ICMP or DNS queries from a PC. If we're just interested in a specific type of traffic (let's say TCP Traffic only) we need to change our filter command slightly like this:
# diag sniffer packet internal 'src host 192.168.0.130 and dst host 192.168.0.1 and tcp' 1
192.168.0.130.3569 -> 192.168.0.1.23: syn 1802541497
192.168.0.1.23 -> 192.168.0.130.3569: syn 4238146022 ack 1802541498
192.168.0.130.3569 -> 192.168.0.1.23: ack 4238146023
Though ICMP (ping) was also running, the trace only shows the TCP part. As we can see the Destination is: 192.168.0.1.23 which is IP 192.168.0.1 on Port 23. Apparently we found a Telnet Session to 192.168.0.1 right during initial setup.
The same the other way around:
# diag sniffer packet internal 'host 192.168.0.130 and icmp' 1
192.168.0.130 -> 192.168.0.1: icmp: echo request
192.168.0.1 -> 192.168.0.130: icmp: echo reply
In this example we're sniffing for ICMP only, to and from 192.168.0.130
Another cool thing is logical combinations. Let's assume you want to sniff for ICMP and TCP only (but not for UDP, ARP, etc). You can combine protocols in the following manner:
# diag sniffer packet internal 'host 192.168.0.130 and (icmp or tcp)' 1
This sniff will display all tcp or icmp traffic to and from host 192.168.0.30, in verbose 1 level.
Now we're going to limit the sniffer even more:
We want to sniff traffic between 2 hosts, but only TCP and only port 80.
# diag sniffer packet internal 'host 192.168.0.130 or host 192.168.0.1 and tcp port 80' 1
192.168.0.130.3625 -> 192.168.0.1.80: syn 2057246590
192.168.0.1.80 -> 192.168.0.130.3625: syn 3291168205 ack 2057246591
192.168.0.130.3625 -> 192.168.0.1.80: ack 3291168206
192.168.0.130.3625 -> 192.168.0.1.80: psh 2057246591 ack 3291168206
192.168.0.1.80 -> 192.168.0.130.3625: ack 2057247265
Even if telnet and ssh is running between the two hosts, we only see port 80 TCP traffic.
Finally, attached is a detailed explanation on how to use the sniffer to capture traffic for only certain TCP flags.
Also attached is the fgt2eth.pl script that will convert a verbose 3 sniffer output, into a file readable and decodable by Ethereal.
Note: This attached scripts are provided "as is". These files are not supported by Technical Support.
Note: Stephen Dennis recently sent us an updated version of the file explanation_on_how_to_packet_capture_for_only_certain_TCP_flags_v2.txt . The updated file is included below:
fgt2eth.pl | View | Download
explanation_on_how_to_packet_capture_for_only_certain_TCP_flags_v2.txt | View | Download
|
|
|
|
|
