elasticsearch


Fluentd High Availability Custom Index


I've setup a fluentd/elasticsearch/kibana stack very similar to what is described here. When I look at the logs in kibana I notice that they are automatically indexed by day using the format "logstash-[YYYY].[MM].[DD]. Based on the documentation for the fluentd elasticsearch plugin it seems that you can create a custom index by setting the "index_name" property.
I've tried this on both the log forwarder and the log aggregator but I still seem to get the default index name in elasticsearch. Is there something else required to customize this index name in a HA setup?
Here is the log forwarder config:
<source>
type tail
path /var/log/debug.json
pos_file /var/log/debug.pos
tag asdf
format json
index_name fluentd
time_key time_field
</source>
<match *>
type copy
<store>
type stdout
</store>
<store>
type forward
flush_interval 10s
<server>
host [fluentd aggregator]
</server>
</store>
</match>
And here is the log aggregator config:
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<match *>
type copy
<store>
type stdout
</store>
<store>
type elasticsearch
host localhost
port 9200
index_name fluentd
type_name fluentd
logstash_format true
include_tag_key true
flush_interval 10s # for testing
</store>
</match>
I found an issue on the fluent-plugin-elasticsearch repo that explains this behavior. When setting the "logstash_format" option to true the "index_name" field is ignored.
remove logstash_format true from .You will get your custom index.But you will not get timestamp in your data.For getting timestamp you have to update version of ruby and then pass time format to config file of fluentd.

Related Links

Elasticsearch: querying on both nested object properties and parent properties
elasticsearch first setup - create index
Using ElasticSearch as alternative data store with applications updating both the DB and ES(with the help of Kafka). Is this a good idea?
Where Elasticsearch store the data on Mac
Elastica - multiple bool queries - subqueries
How do i add boosting to the elasticsearch mapping in ONGR?
Is it possible to use doc_count as a term in elasticsearch?
Elastic search aggregation with sort
Elasticsearhc- how to do multi search request on Sense or Head plugin
How do I configure/initialize a custom Elasticsearch Tokenizer
Combining log entries with logstash
How to import CSV data using Logstash for field type completion in Elasticsearch suggester
How can I interpret the result of percentiles aggregation type
elastic search select fields to return in aggregation
How to Focus Search On One Part of a Document
How to calculate cooccurrences on a set of texts with Elasticsearch

Categories

HOME
visual-studio
automapper
angular-ui-grid
scroll
websocket
yql
mathematical-optimization
apache-kafka-connect
haxe
mithril.js
gprs
onvif
tweepy
pushdown-automaton
max-msp-jitter
synthesis
user-defined-types
l20n
varnish-vcl
node-webkit
device
firemonkey-style
unordered-multimap
ipa
python-imageio
math.js
sfml
boolean-logic
sales
guice
credit-card
serversocket
bootstrap-sass
buildforge
cups
spinner
installanywhere
llvm-ir
reset
game-center
grails-spring-security
cloudera-sentry
raytracing
password-hash
renderman
visual-format-language
rustdoc
fitbit
yargs
thread-exceptions
dcount
visual-studio-monaco
ftp-client
callouts
i2b2
currency-exchange-rates
openmrs
diawi
bayesglm
nastran
calcite
knife
compositetype
ampersand
incron
jcheckbox
hp-idol-ondemand
comctl32
cakephp-2.2
vs-unit-testing-framework
inequality
xhprof
visual-c++-2012
full-text-indexing
android-2.3-gingerbread
pre
yii-booster
notin
oscilloscope
jquery-cycle
struts2-json-plugin
playorm
nsmutablestring
image-formats
taskkill
struts2-s2hibernate
pcspim
platform-independent
will-paginate
qt-mobility
cewolf
pyjamas
turbine
suphp
eaccelerator
ccl
.net-services
zend-test
objectinstantiation
word-automation

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile