Proactive Security through
Firewall Log Analysis
By David M. Piscitello, President, Core Competence
Routinely examining firewall logs is important, even for small business firewall administrators. Analyzed regularly, logs help you confirm that the policy you've implemented in your firewall matches the security policy your organization expects to enforce. Logs also help identify new and suspicious application activity. They reveal the latest, most popular attacks and methods. Most importantly, timely log analysis might help you predict and deter an attack.
What should you be logging?
The simple answer: log everything. Some firewall administrators log and analyze only incoming traffic that is denied. This method provides a partial picture of how hosts outside your protected network use, probe, or misuse your network. However, if you log denies only, you won't know whether your incoming traffic policy properly blocks traffic to destinations you intend to protect. If a configuration change allows undesirable access to protected hosts, you won't know who made the change creating this "attack opportunity," nor when. Coldly put, you'll have little information to offer the expensive forensic expert you'll call in to investigate a serious security incident.
If you don't care to log everything, consider logging allowed incoming traffic for ports you expect your firewall to block (for example, ports used to support services among hosts in your protected networks that you wouldn't want accessed from "the outside"). Correlate logging activities you implement at your firewalls, servers and other application security systems. Where possible, try to log security-related events close to the asset. For example, if you allow incoming services like HTTP/Web, and you use Microsoft's URLscan and you log Web request handling at your IIS servers, you may find the individual Web server log "granularity" more helpful than Web traffic logged at your firewall.
If you also log allowed outgoing traffic, you'll know quickly whether your network becomes a party to an attack. By logging and examining allowed outgoing traffic, you might detect a keystroke-logging program or RAT (Remote Administration Tool) installed by a virus or spyware pest because you've blocked the unusual port it uses to talk to its master collection program. Or, you might determine that unauthorized applications are (incorrectly) permitted through your firewall. In all these situations, you can use log information to identify resource abusers, a compromised host, or a mis-configured policy at your firewall. You'll also have sufficient information to begin removing malicious code from infected systems.
A method for analyzing
incoming log events
Now that you're logging inbound and outbound activity, it's time to use those logs. Examine incoming traffic log fields in the following order.
- What is the disposition of the packet (allowed or denied)?
- What is the policy I expected to apply to this service port?
- What is the target host's address?
- What is the attacker's address?
Use the disposition and port fields to verify that sensitive services on all your trusted hosts are protected from unauthorized access. For example, the ports running protected services should always have a disposition (and direction) of "denied in." Verify that publicly offered services are accessible to external hosts; e.g., all the ports running publicly accessible services should have a disposition (and direction) of "allowed in." Now use the destination address field of allowed incoming packets to verify that external hosts can reach only those trusted systems that host publicly offered services.
The log record sequence illustrated here shows someone's attempt to find Secure Shell (TCP/22) running on addresses in my public IP space. (Note: if you don't know how to interpret Firebox log messages, read "What does a Deny or Allow log message mean?")
02/17/05 09:14 firewalld[130]: deny in eth0 48 tcp 20 111
63.197.102.21 64.53.71.145 20500 22 syn (blocked site)
02/17/05 09:14 firewalld[130]: deny in eth0 48 tcp 20 111
63.197.102.21 64.53.71.146 20500 22 syn (blocked site)
02/17/05 09:14 firewalld[130]: deny in eth0 48 tcp 20 111
63.197.102.21 64.53.71.147 20500 22 syn (blocked site)
02/17/05 09:14 firewalld[130]: deny in eth0 48 tcp 20 111
63.197.102.21 64.53.71.148 20500 22 syn (blocked site)
02/17/05 09:14 firewalld[130]: deny in eth0 48 tcp 20 111
63.197.102.21 64.53.71.149 20500 22 syn (blocked site)
02/17/05 09:14 firewalld[130]: deny in eth0 48 tcp 20 111
63.197.102.21 64.53.71.150 20500 22 syn (blocked site)
This is probably an automated scan. If the port were listening (open), the attacker would attempt a series of secure shell exploits. Because it's blocked, he moves on.
Another common log record sequence is a selected port scan, illustrated here:
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3110 6129 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3119 3410 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3134 5000 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3134 3306 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3134 5554 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3127 1433 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3127 65506 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3127 2745 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 113
64.53.139.2 64.53.71.157 3143 80 syn (blocked site)
Selected port scans often focus on the current, most popular exploits. In this case, the scan is looking to exploit known vulnerabilities of commercial software...
- Windows Universal Plug and Play (UpnP, TCP/5000)
- SQL weak authentication exploiter (mySQLbot, TCP/3306)
- SQL server (TCP/1433)
- Microsoft IIS exploits: CodeRed, WebDAV, Nimda (TCP/80)
...or malicious code installed by a virus or copied via a file sharing (P2P) application:
- The Dameware remote administration tool (TCP/6129)
- The OptixPro Trojan (TCP/3410)
- Dabber worm scanning for Sasser-infected hosts on FTP server port (TCP/5554)
- MyDoom spam relay scans (TCP/65506)
- Bagel/Beagle worm backdoor (TCP/2745)
Several resources can help you identify port use and abuse. I referred to Linklogger.com, Lurhq.com, Dshields.org, and SANS.org here, but a simple Google search of "TCP <port number>" will provide you with even more resources that describe attacks and exploits. Many resources identify patches available to mitigate the vulnerability. Portpeeker, at Linklogger.com, even provides a packet capture of exploits so you can appreciate attacker methodologies. Consider running selected port scans of the current, most popular malicious code ports at hosts on your trusted server, to verify your hosts are not infected.
If you are only logging denied incoming traffic, you won't know if your firewall configuration is failing to block these and other potentially exploitable ports. If you were logging all traffic, however, you might see:
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3110 6129 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3119 3410 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3134 5000 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3134 3306 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3134 5554 syn (blocked site)
02/17/05 07:49 firewalld[130]: allow in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3127 1433 syn
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3127 65506 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 114
64.53.139.2 64.53.71.157 3127 2745 syn (blocked site)
02/17/05 07:49 firewalld[130]: deny in eth0 48 tcp 20 113
64.53.139.2 64.53.71.157 3143 80 syn (blocked site)
Here, someone's succeeded in port-forwarding traffic to a SQL server from an external network, which is not the intended security policy. Would you rather be notified by someone outside your organization that a host in your network is a spam relay, or discover it yourself?
These examples show automated and heavy-handed scan techniques. More sophisticated scan techniques use fewer probe packets, directed at your network, at different ports, with random time intervals between probes. When I see a single denied ICMP (0) message in my logs, I might search my logs for other occurrences of its source IP address in the last few hours or days. A pattern often emerges: this type of attacker sends single packet probes to well-known or exploitable ports -- TCP 22/ssh, 23/telnet, 25/smtp, 53/dns, 500/ike, dabber/5554 -- stealthily spaced over time.
Do not obsess over the origin of potential attack packets. Source addresses used in attack packets are often spoofed IP addresses, or they may be hosts an attacker has already compromised and "0wnz." Chances are very slim that you will trace back to an attacker's actual location using only your log records. However, if you see a very high frequency of heavy-handed scans, denial of service attacks, or a persistent pattern of stealthy probes, you may want to dig the DNS, query WHOIS and search name registrars to obtain the contact information of the administrator who runs the network hosting the probe or attack source. This often results in a dead lead or frustrating exchange (as exemplified in "When Bad Guys Happen to Good Admins"). Many attacks emanate from ISP dynamic address pools, and service providers typically won't take action until the volume of complaints compels them, or legal action is threatened. Occasionally, you might actually assist a network administrator in identifying a compromised system under his purview. My experience is that few people actually admit the problem exists, but you won't see probes from that address again, at least for awhile.
There is a second category of denied incoming traffic: traffic you blocked in error. This is commonly the result of a misconfiguration. Mistaken blocking interferes with revenue and productivity, but (hopefully) it happens infrequently. When it happens, analyze log records to discover a pattern that exposes a flaw in your administrative procedures or workflows, or to identify an individual who is not complying with procedures.
Conclusion
Security event logging is of little benefit unless you take the time to analyze the events. Make time to look at your logs regularly. Even cursory analyses can be educational. If you need a motivation, think of log analysis as the network security corollary to a 15-minute daily walk (and a daily aspirin) to reduce your risk of heart attacks and strokes. A little routine log analysis can be a potent preventative security measure to help keep your network healthy. ##
Further Resources:
Copyright© 2005, WatchGuard Technologies, Inc. All rights reserved. WatchGuard, LiveSecurity, Firebox and ServerLock are trademarks or registered trademarks of WatchGuard Technologies, Inc. in the United States and other countries.