Monash Uni splunks server logs

Monash University has licensed the next version of a tool called Splunk that it uses to index system log data from 600 physical and 100 virtual servers.

The tool, which is available in free and paid versions, enables companies to search across system, security incident and audit logs to troubleshoot IT issues and investigate issues.

Senior systems and storage administrator at Monash, Joshua Edmonds, told iTnews the University trialled the free version of the tool for a year before upgrading.

Versions are priced on the amount of log data that is indexed and searched each day. The free license included a 500MB limit, which Splunk said would cover "a handful of servers generating a modest amount of [log] data".

Edmonds said the University's license enabled it to index and search up to 1GB of data per day.

"At the moment we're only indexing our syslog data from the servers," he said. "We're not capturing log data from network equipment or storage devices.

"On a typical day we're looking at about 300 to 400MB but when there are problems that can peak at up to 1.2GB," he said.

The tool provided some leeway if data limits were breached. Edmonds said they had not been prevented from examining occasional data spikes.

Edmonds said Splunk had been deployed on a central logging server that captured log data from across its entire server environment.

He said it was possible the University would examine a more distributed architectural approach to Splunk's deployment.

Splunk said one of the advantages of version four of the tool was that the processing required for indexing and searching the log data could be conducted in a distributed fashion.

Splunk co-founder Michael Baum said rather than collect logging data and aggregating it in a single point, Yahoo! had installed Splunk in each of its 32 data centres to enable processing to occur locally.

"We then federate search across all those different locations in real time," he said.

Baum said the company had a "couple of hundred thousand" free users and 1100 paid customers. The paid version starts at US$9000 (AU$10,860) in Australia.

Existing customers in Australia included government agencies and Telstra, which had deployed the tool to support multimedia delivery on mobile phones.

Splunk said it was "talking to Telstra for more projects."

Baum believed there was a need for tools like Splunk - in part because vendors were "notoriously bad at giving [customers] good tools to analyse log data.

"Companies like Telstra can no longer afford to leave these logs out on end network devices where they typically get overwritten every couple of hours because there's not enough on-board memory," Baum said.

"[IT departments] really need a few days of data to be able to get a baseline and determine trends."