Splunk System Logger

Posted by William
I am back to posting on here again! I've been working on several projects and I think I have enough information to bring a steady stream of posts. Lately, I have been using Splunk Server (http://www.splunk.com) as a centralized logging server. For any of you network administrators who have to deal with the immense amount of system logs know how much of a hassle it can be. The most common ways of dealing with them is directing them to a central log server and then creating scripts or putting them into a sql database to search through them. Splunk happens to be an extremely brand new approach to searching through logs! It's pretty much "Google" for system logs. It will index -everything- and search results are clickable. By clickable I mean for example I do the following search query on Splunk:

host::210.10.112.1 ipmon

In this query I am searching for all firewall logs (netmon) from my router 210.10.110.112.1. When you receive the results and start sifting through them and you see a result like. 

May 8 02:09:21 210.10.112.1 ipmon[82]: 02:09:21.118776 sis1 @0:21 b 57.61.61.63,13364 -> 60.23.110.23,1026 PR udp len 20 393 IN
eventtype::?154 EventType Menu | Check splunk.com | 05/08/2006, 02:09:21 | Similar | Related
sourcetype::syslog rename | host::210.10.112.1 | source::/var/syslog-ng/syslog_fifo | Show Source

I now want to further investigate this entry by searching for more entries with the IP 57.61.61.63 so I can either type that into the search box OR click on the IP and doing so gives me the following results. 

May 8 02:09:21 210.10.112.1 ipmon[82]: 02:09:21.118909 sis1 @0:21 b 57.61.61.63,13364 -> 60.23.110.23,1029 PR udp len 20 393 IN
eventtype::?154 EventType Menu | Check splunk.com | 05/08/2006, 02:09:21 | Similar | Related
sourcetype::syslog rename | host::210.10.112.1 | source::/var/syslog-ng/syslog_fifo | Show Source

May 8 02:09:21 210.10.112.1 ipmon[82]: 02:09:21.118776 sis1 @0:21 b 57.61.61.63,13364 -> 60.23.110.23,1026 PR udp len 20 393 IN

eventtype::?154 EventType Menu | Check splunk.com | 05/08/2006, 02:09:21 | Similar | Related
sourcetype::syslog rename | host::210.10.112.1 | source::/var/syslog-ng/syslog_fifo | Show Source

May 7 21:59:07 210.10.112.1 ipmon[82]: 21:59:06.739855 sis1 @0:21 b 57.61.61.63,13364 -> 60.23.110.23,1032 PR udp len 20 393 IN
eventtype::?154 EventType Menu | Check splunk.com | 05/07/2006, 21:59:07 | Similar | Related
sourcetype::syslog rename | host::210.10.112.1 | source::/var/syslog-ng/syslog_fifo | Show Source
May 7 21:59:07 210.10.112.1 ipmon[82]: 21:59:06.739730 sis1 @0:21 b 57.61.61.63,13364 -> 60.23.110.23,1026 PR udp len 20 393 IN

eventtype::?154 EventType Menu | Check splunk.com | 05/07/2006, 21:59:07 | Similar | Related
sourcetype::syslog rename | host::210.10.112.1 | source::/var/syslog-ng/syslog_fifo | Show Source

From the results I can see all activity related to the 57.61.61.63 address. This is an extremely easy and effective way to look through firewall logs! I took the logging a step further by adding a Cisco router between my main router and switch so all traffic would have to enter and exit through the Cisco router. The setup is simple, interface eth0 which connects to the main router has an access list applied to it to log all inbound and outbound traffic. 

access-list 101 permit tcp any any gt 0 log-input
access-list 101 permit udp any any gt 0 log-input
access-list 101 permit ip any any

This access list permits maximum verbosity by including the source/destination IP address, MAC address, and port number. The access list must specify protocol type and port ranges to include the port number in the system log and in this case 'gt 0' means any port greater than 0. I configure the Cisco router to send logs to log server and Splunk will start to receive all logs. The Cisco router is 210.10.112.5 so an appropriate search query is.
host::210.10.112.5
[+] The results: 

May 8 02:25:50 210.10.112.5 67: 00:43:24: %SEC-6-IPACCESSLOGP: list 102 permitted tcp 70.117.13.4(34082) (Ethernet0 01e0.28a3.87f6)
 -> 210.10.112.10(23), 80 packets eventtype::?160 tcp EventType Menu | Check splunk.com | 05/08/2006, 02:25:50 | Similar |
Related sourcetype::syslog rename | host::210.10.112.5 | source::/var/syslog-ng/syslog_fifo | Show Source

May 8 02:25:29 210.10.112.5 66: 00:43:03: %SEC-6-IPACCESSLOGP: list 102 permitted tcp 63.10.210.1(60880) (Ethernet0 01e0.223c.f4fa) 
-> 210.10.112.10(80), 1 packet eventtype::?160 tcp EventType Menu | Check splunk.com | 05/08/2006, 02:25:29 | Similar | 
Related sourcetype::syslog rename | host::210.10.112.5 | source::/var/syslog-ng/syslog_fifo | Show Source

May 8 02:25:27 210.10.112.5 65: 00:43:01: %SEC-6-IPACCESSLOGP: list 102 permitted tcp 63.10.210.1(60879) (Ethernet0 01e0.223c.f4fa) 
-> 210.10.112.10(80), 1 packet eventtype::?160 tcp EventType Menu | Check splunk.com | 05/08/2006, 02:25:27 | Similar | 
Related sourcetype::syslog rename | host::210.10.112.5 | source::/var/syslog-ng/syslog_fifo | Show Source

May 8 02:22:26 210.10.112.5 63: 00:39:59: %SEC-6-IPACCESSLOGP: list 102 permitted udp 63.10.210.1(33921) (Ethernet0 01e0.223c.f4fa) 
-> 210.10.112.10(161), 100 packets eventtype::?162 EventType Menu | Check splunk.com | 05/08/2006, 02:22:26 | Similar | 
Related sourcetype::syslog rename | host::210.10.112.5 | source::/var/syslog-ng/syslog_fifo | Show Source

With the access list on the Cisco router and Splunk I can keep track of all traffic exiting and entering the network. As the log grows with time I can use Splunk's search directives to entries from 4 hours ago or 1 day go. Splunk does well to keep everything nice and neat.

Splunk offers Splunk Server which is completely free, but permits only 500MB of indexing a day. The Professional versions use licenses based on how much data you can index a day and are quite expensive. Of all software that is sold this is well worth it if you must manage a huge amount of system logs and keep track of firewall logs. A Splunk Demo (http://demo.splunk.com) is available to see and use Splunk before you decide to install it yourself. The professional version has other neat features like RSS feeds and have alert notifications based on a predetermined search query. A completely open source alternative exists called php-syslog-ng and two versions exist, http://www.phpwizardry.com/php-syslog-ng.php and http://www.vermeer.org/projects/php-syslog-ng . I use the latter one with custom modifications and I will offer it on the site later on, with my modifications, when I type a nice howto on using and customizing its interface.

Free Projects