elasticsearch


Combining multiple message fields using multiline codec in logstash?


I am using logstash 2.4.0
My output is like this:
{
"#timestamp" => "2017-05-10T18:14:47.269Z",
"message" => "[2017-01-14 10:59:58,591][WARN ][index.search.slowlog.query] [yaswanth] [bank][3] took[50ms], took_millis[50], types[details], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}], extra_source[], \r",
"#version" => "1",
"path" => "F:\\logstash-2.4.0\\logstash-2.4.0\\bin\\picaso.txt",
"host" => "yaswanth",
"TIMESTAMP" => "2017-01-14 10:59:58,591",
"LEVEL" => "WARN",
"QUERY" => "index.search.slowlog.query",
"QUERY1" => "yaswanth",
"INDEX-NAME" => "bank",
"SHARD" => "3",
"TOOK" => "50ms",
"TOOKM" => 50,
"types" => "details",
"search_type" => "QUERY_THEN_FETCH",
"total_shards" => "5",
"source_query" => "{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}"
}
{
"#timestamp" => "2017-05-10T18:14:47.270Z",
"message" => "[2017-01-14 10:59:58,591][WARN ][index.search.slowlog.query] [yaswanth] [bank][2] took[50.2ms], took_millis[50], types[details], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}], extra_source[], \r",
"#version" => "1",
"path" => "F:\\logstash-2.4.0\\logstash-2.4.0\\bin\\picaso.txt",
"host" => "yaswanth",
"TIMESTAMP" => "2017-01-14 10:59:58,591",
"LEVEL" => "WARN",
"QUERY" => "index.search.slowlog.query",
"QUERY1" => "yaswanth",
"INDEX-NAME" => "bank",
"SHARD" => "2",
"TOOK" => "50.2ms",
"TOOKM" => 50,
"types" => "details",
"search_type" => "QUERY_THEN_FETCH",
"total_shards" => "5",
"source_query" => "{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}"
}
But what i want is like this
{
"#timestamp" => "2017-05-10T18:14:47.269Z",
"message" => "[2017-01-14 10:59:58,591][WARN ][index.search.slowlog.query] [yaswanth] [bank][3] took[50ms], took_millis[50], types[details], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}], extra_source[], \r",[2017-01-14 10:59:58,591][WARN ][index.search.slowlog.query] [yaswanth] [bank][2] took[50.2ms], took_millis[50], types[details], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}], extra_source[], \r"
"#version" => "1",
"path" => "F:\\logstash-2.4.0\\logstash-2.4.0\\bin\\picaso.txt",
"host" => "yaswanth",
"TIMESTAMP" => "2017-01-14 10:59:58,591",
"LEVEL" => "WARN",
"QUERY" => "index.search.slowlog.query",
"QUERY1" => "yaswanth",
"INDEX-NAME" => "bank",
"SHARD" => "3",
"TOOK" => "50ms",
"TOOKM" => 50,
"types" => "details",
"search_type" => "QUERY_THEN_FETCH",
"total_shards" => "5",
"source_query" => "{\"sort\":[{\"balance\":{\"order\":\"asc\"}}]}"
}
I want to send all the message fields from multiple events to a single event for sending email .
Is there anything wrong in the above config ? Do i have to use aggregate filter for this type of requirement?
Thanks
What you could do is to aggregate a number of events of at the level of the file input plugin before sending them to the output plugin. A good example is given here.
You might have to modify your grok filter a little bite.

Related Links

ElasticSearch assign own IDs while indexing with LogStash
How to retrieve the latest event with a given field value?
Converting nginx access log bytes to number in Kibana4
Elasticsearch minBy
Snowball analyzer vs language analyzer
AND between tokens in elasticsearch
Elasticsearch Stemmer Override Token Filter not working when usind rules path
Logstash extracting values from sp_executesql
Logstash Grok Modifying and overwriting #timestamp
how to restore elasticsearch.yml config file to original?
Index a graph with ElasticSearch
ElasticSearch query not returning expected results
Fluentd High Availability Custom Index
How do I add an attribute to an Elasticsearch node for the purpose of Shard Allocation Filtering?
Configure ElasticSearch attachment mapper to use OCR plugin
ElasticSearch: How to search for a value in any field, across all types, in one or more indices?

Categories

HOME
knockout.js
macos-sierra
mc
blast
long-integer
android-emulator
websocket
dependencies
google-sheets-query
elk-stack
impala
gprs
roku
sap-fiori
smooks
jquery-select2-4
cuba-platform
sonicwall
phonegap-build
closures
parceler
router
cellular-network
high-availability
offline-caching
onchange
boolean-logic
premake
xcrun
counting
restful-architecture
android-canvas
autoencoder
xvfb
selenide
uipickerview
cloudera-sentry
latexml
react-intl
rm
django-static-precompiler
nothing
amazon-elastic-beanstalk
nivo-slider
deedle
sesame
gmaps.js
business-rules
gawk
annotatorjs
cefpython
mathjs
p6spy
coovachilli
xcode8-beta4
meld
noclassdeffounderror
sharepoint-apps
jemdoc
utf-16
as3-api
wikimedia-commons
ajax4jsf
nette
ngcordova
deque
mathematical-expressions
baseadapter
google-hangouts
flurry-analytics
iodocs
inequality
jquery-mobile-flipswitch
commenting
wso2bam
acitree
acceptance-testing
ntlmv2
dojo-1.9
spark-view-engine
clrprofiler
pageheap
opends
kohana-auth
aspbutton

Resources

Encrypt Message