elasticsearch


Logstash - complicated filtering


I have ELK setup. I am parsing several completely different log formats. They were all one liners. Now I need to add a different one which is a multi line.
Example of two log entries:
error: callback invoked exception. sent payload: [{"key": "values"}]
custom status response: [{"key": "values"}]
callback headers: [{"key": "values"}]
error stack: [ something really bad happened
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)]
error: callback invoked exception. sent payload: [{"key": "values"}]
custom status response: [{"key": "values"}]
callback headers: [{"key": "values"}]
error stack: [ something really bad happened
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)
at here loremisptul (/xx/xx/x)]
Those two log entries have many JSON key/value entries (removed for clarity).
Those two log entries are separated by empty line.
This is my logstash configuration file:
input {
beats {
port => "5043"
codec => json
}
}
filter {
if [#metadata][type] == "qa-error"
{
grok {
match => { "message" => "{GREEDYDATA:stackone}" }
}
}
}
output
{
...write it to ES
}
With this setup, every line is written as a separate document in ES. I made a bit research, and it seems I might have to end up using multiline codec inside my filebeat.
But if I place that codes, then all of the other filters are not working properly any more.
How can I approach this ?

Related Links

How to filter using any of the full text queries?
Elasticsearch: private user documents
I am looking for ElasticSearch equivalent of below query -
Retrieve docs that contains only allowed tags (exactly equals)
Get max-score of Elasticsearch in CakePHP 3
ElasticSearch multiple terms aggregation order
Copy one column in elasticsearch index into another column applying analyzers
Why is my query ignoring my filter aggregation?
ElasticSearch analyzer that prioritizes order of term appearance in a search
Default elasticsearch configuration for docker container
Add alias to index from template?
Save ElasticSearch snapshot physical files in specific location
Run elasticsearch with docker
elasticsearch: How to reinitialize a node?
Elasticsearch filter on mulitple nested paths
Need to Sort the _term in elastic search aggregation result

Categories

HOME
jboss
generics
blast
verification
bots
postgresql-9.4
systemd
adobe
android-emulator
google-webmaster-tools
azure-data-lake
capistrano
jpa-criteria
elastic-load-balancer
hystrix
scrape
alljoyn
watch-os-3
pm2
createjs
asp.net-core-1.0
global-variables
vertica
fosuserbundle
psql
yii2-basic-app
activeadmin
tableview
amazon-sns
quantitative-finance
device
eclipse-emf
raima
high-availability
network-analysis
onchange
react-bootstrap-table
bonita
fusion
publishing
quick-nimble
mongodb-3.4
sca
advanced-filter
android-alarms
microsoft-ui-automation
right-click
raytracing
julius-speech
docker-ucp
wysihtml5
section508
amazon-elastic-beanstalk
redisson
oracle-fusion-apps
xcode6.4
mathjs
nsdata
wolfram-language
urbit
teamviewer
wif4.5
ioctl
helm
openshift-cartridge
replicaset
compositetype
deque
sablecc
nhibernate-criteria
cakephp-2.2
dc
storing-data
webautomation
jini
commenting
jms-serializer
lync-2010
yii-booster
delphi-xe4
web-frameworks
ofstream
jzmq
will-paginate
botnet
space-partitioning
jquery-1.4
asp.net-mvc-views
printqueue
turbine
suphp
ccl
aggregator
reliability
cleartype
processors
regioninfo
service-factory

Resources

Encrypt Message