Splunk Commands
Oct 11th, 2019 - written by Kimserey with .
Last week we saw the different ways to plot charts and tables in Splunk. We saw how eval and bin could be used to manipulate logs in order to plot useful charts. Continuing in the same line, today we will be looking at more useful commands used to join different event results, group events and identify events.
append and appendcols
When we want to create queries spanning over multiple indexes, we can use append to append events from multiple queries to the same stream of events.
1
2
3
4
index=prod ...
| append [ search index=qa ... ]
| append [ search index=dev ... ]
| timechart count by host
If we need to add another column to plot on the same graph, we can use appendcols [search ...].
1
2
3
...
| timechart span=10m count
| appendcols [ search ... | timechart span=10m count ]
Appendcols will add a column in the stats table instead of adding to the stream of events which we can then simply visualize in the Visualization tab on Splunk.
spath
Spath is used to extract information from structured logs.
1
... | spath output=corId path=properties.corId
This will add a property corId specified by output in the event by extracting the value from the path properties.corId. To get a value on an array, we can use the notation {}. For example if properties was an array:
1
... | spath output=corId path=properties{1}
transaction
Transaction is used to group events based on one or more fields into a single event.
1
2
3
...
| spath output=corId path=properties.corId
| transaction corId startswith="Received Request" endswith="Completed Request"
This will group all events with the same correlation identifier and start the transaction with the Received Request event and end it with the Completed Request. On top of grouping it also adds a duration property which is the duration between the start and end event.
Using the duration and the eventcount we can then plot a chart or compute stats which will tell us the response time of our system:
We also have access to all the properties from all events mashed together in a single event, which we can use to make query against the transaction itself.
1
2
3
4
5
6
...
| spath output=corId path=properties.corId
| transaction corId startswith="Received Request" endswith="Completed Request"
| spath output=price path=properties.price
| spath output=region path=properties.region
| timechart limit=0 span=5m max(price) by region
This query will group all events between Received Request and CompletedRequest with the same corId and extract price and region out of the group of events and then timechart the maximum price per region in a span of five minutes, limit=0 disable the limit of split so that we can see all regions.
rex
Lastly rex can be used to extract groups of values out of events to be used in queries. This is useful when the message log doesn’t have a clear way of extracting values.
As logs are predictable, a nice trick to extract data can be built done using dots (.) to match single characters easily in an event.
For example if our transaction contains multiple events but not all the properties are understood by Splunk, we can use rex to extract pieces of the events using _raw which contains the raw grouping of events.
1
2
3
4
5
...
| spath output=corId eventProperties.corId
| transaction corId startswith="Received Request" endswith="Completed Request"
| rex field=_raw "price..(?<price>[0-9]*)"
| table corId, price
Here we want to match price"=123 and extract 123, so we look for price in _raw and match the next two character "= and extract a group named price which we can then use.
To be used with moderation, as on top of coupling the message itself, we couple the exact amount of characters.
Conclusion
Today we looked at Splunk commands which are commonly used to extract information from logs. We started by looking at append and appendcols which allow us to construct a query made from multiple queries, we then looked into transaction
which allows us to group events into a single transaction and allows us to work with that transaction, and lastly we looked into rex which allows us to apply regular expressions on events and extract fields. Hope you liked this post and I see you on the next one!