Splunk Tips

Deleting data from indexes

Splunk only lets you mark data as deleted (using the | delete command in a search query and the can_delete role) but this does not free up disk space. You can delete all of the data in an index using splunk clean , but that's no good if you only want to get rid of a subset of the data.

The solution is to export the data in each of your index's database buckets to CSV, which excludes deleted items, and re-import them. As database buckets are split into manageable chunks, this will even work if you're perilously low on disk space! :)

 #!/bin/bash

 exporttool="splunk cmd exporttool"
 importtool="splunk cmd importtool"

 # Set this to the correct directory for the index you're cleaning up
 for bucket in /opt/splunk/var/lib/splunk/defaultdb/db/*; do
        echo $bucket
        $exporttool $bucket - -csv meta::all | $importtool $bucket.new -
        splunk-optimize -d $bucket.new
        splunk-optimize-lex -d $bucket.new
        find $bucket.new/rawdata -name '[0-9]*[0-9]' -size +1k -print0 | xargs -0 -r gzip -v9
        rm -r $bucket && mv $bucket.new $bucket
 done

This will probably generate a bunch of alarming errors (no events / error reading csv file: -) but this means that the bucket contained no events for whatever reason.
There's no error checking so this is probably not something you want to do on an important production system and certainly not without backups!
It may take quite some time to run! You will also probably want to stop Splunk while this is going, or else only run it on non-current buckets and then restart Splunk after you do the rm / mv combo.

Forms based authentication for Splunk Free

I've written an article on using Apache 2.4's forms-based authentication for Splunk Free. This is much fancier and nicer than HTTP BASIC authentication!

My blog posts about Splunk

(:RSS http://tumblr.tristesse.org/tagged/splunk/rss short 50 :)