elasticsearch


Logstash conditional output to elasticsearch (index per filebeat hostname)


I have several web servers with filebeat installed and I want to have multiple indices per host.
My current configuration looks as
input {
beats {
ports => 1337
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
if [beat][hostname] == "luna"
{
hosts => "10.0.1.1:9200"
manage_template => true
index => "lunaindex-%{+YYYY.MM.dd}"
document_type => "apache"
}
}
}
However the above conf results to
The given configuration is invalid. Reason: Expected one of #, => at
line 22, column 6 (byte 346)
which is where the if statement takes place. Any help?
I would like to have the above in a nested format as
if [beat][hostname] == "lina"
{
index = lina
}
else if [beat][hostname] == "lona"
{
index = lona
}
etc. Any help please?
A solution can be :
Define in each of your filebeat configuration file, in the prosperctor section define the document type :
document_type: luna
And in your pipeline conf file, check the type field
if[type]=="luna"
Hope this help.
To access any inner field you have to enclosed it with %{}.
Try this
%{[beat][hostname]}
See this for more explanations.
UPDATE:
Using %{[beat][hostname]} with == will not work, try
if "lina" in [beat][hostname]{
index = lina
}

Related Links

ELK Type Conversion - Not a number but a string
How to get the definitiion of a search analyzer of an index in elasticsearch
ElasticSearch query using match or term?
How to combine aggregations in ElasticSearch/Kibana?
ElasticSerach cluster performance
Nxlog unable to send eventlog after certain time
Sort elasticsearch search hits by document count
Elastic search date range max, min date
Elastic search river mongodb _meta returning action not found error
Seeing many open Elasticsearch connections even after using singleton pattern
What would be a good approach for sending logs from multiple servers a centralized logging server?
does elasticsearch support queries of queries?
Data modelling with elastic search
match or term query on a long property for exact match?
Updating filtered documents in elasticsearch
Testing ElasticSearch custom analyzers

Categories

HOME
rust
oracle
entity-framework
powerbi
yahoo-weather-api
enums
activemq
histogram
yql
ipv6
share
port
reportviewer
orange-api
reselect
nest-api
hidden
robolectric
home
kamailio
multi-upload
managed-c++
tortoisemerge
hdmi
onchange
matlab-app-designer
source-insight
tightvnc
objectmapper
windowsiot
john-the-ripper
listjs
custom-keyboard
jade4j
django-crispy-forms
ng-repeat
task-parallel-library
resampling
type-inference
doctrine-extensions
kitura
common-table-expression
viewmodel
windows-media-player
return-type
sony-future-lab-n
amazon-elastic-beanstalk
deedle
connect-direct
d3v4
allegro
scheduledexecutorservice
okio
gcloud-node
spring-mongo
bbc-micro
distributed-cache
flush
ui4j
postal-code
supersocket.net
sequence-sql
camus
famo.us
lifetime
datagridcomboboxcolumn
ildasm
squeezebox
csslint
accountmanager
mod-perl
ax
xs
dynamic-rdlc-generation
uipangesturerecognizer
liveconnect
exponent
haiku
version-control-migration
twitter-feed
activex-exe
levels

Resources

Encrypt Message