Splunk Tips

Deleting data from indexes

Splunk only lets you mark data as deleted (using the | delete command in a search query and the can_delete role) but this does not free up disk space. You can delete all of the data in an index using splunk clean , but that's no good if you only want to get rid of a subset of the data.

The solution is to export the data in each of your index's database buckets to CSV, which excludes deleted items, and re-import them. As database buckets are split into manageable chunks, this will even work if you're perilously low on disk space! :)

 #!/bin/bash

 exporttool="splunk cmd exporttool"
 importtool="splunk cmd importtool"

 # Set this to the correct directory for the index you're cleaning up
 for bucket in /opt/splunk/var/lib/splunk/defaultdb/db/*; do
        echo $bucket
        $exporttool $bucket - -csv meta::all | $importtool $bucket.new -
        splunk-optimize -d $bucket.new
        splunk-optimize-lex -d $bucket.new
        find $bucket.new/rawdata -name '[0-9]*[0-9]' -size +1k -print0 | xargs -0 -r gzip -v9
        rm -r $bucket && mv $bucket.new $bucket
 done
  • This will probably generate a bunch of alarming errors (no events / error reading csv file: -) but this means that the bucket contained no events for whatever reason.
  • There's no error checking so this is probably not something you want to do on an important production system and certainly not without backups!
  • It may take quite some time to run! You will also probably want to stop Splunk while this is going, or else only run it on non-current buckets and then restart Splunk after you do the rm / mv combo.

Forms based authentication for Splunk Free

I've written an article on using Apache 2.4's forms-based authentication for Splunk Free. This is much fancier and nicer than HTTP BASIC authentication!

My blog posts about Splunk