Fluentd High Availability Custom Index
I've setup a fluentd/elasticsearch/kibana stack very similar to what is described here. When I look at the logs in kibana I notice that they are automatically indexed by day using the format "logstash-[YYYY].[MM].[DD]. Based on the documentation for the fluentd elasticsearch plugin it seems that you can create a custom index by setting the "index_name" property. I've tried this on both the log forwarder and the log aggregator but I still seem to get the default index name in elasticsearch. Is there something else required to customize this index name in a HA setup? Here is the log forwarder config: <source> type tail path /var/log/debug.json pos_file /var/log/debug.pos tag asdf format json index_name fluentd time_key time_field </source> <match *> type copy <store> type stdout </store> <store> type forward flush_interval 10s <server> host [fluentd aggregator] </server> </store> </match> And here is the log aggregator config: <source> type forward port 24224 bind 0.0.0.0 </source> <match *> type copy <store> type stdout </store> <store> type elasticsearch host localhost port 9200 index_name fluentd type_name fluentd logstash_format true include_tag_key true flush_interval 10s # for testing </store> </match>
I found an issue on the fluent-plugin-elasticsearch repo that explains this behavior. When setting the "logstash_format" option to true the "index_name" field is ignored.
remove logstash_format true from .You will get your custom index.But you will not get timestamp in your data.For getting timestamp you have to update version of ruby and then pass time format to config file of fluentd.
Elasticsearch: querying on both nested object properties and parent properties
elasticsearch first setup - create index
Using ElasticSearch as alternative data store with applications updating both the DB and ES(with the help of Kafka). Is this a good idea?
Where Elasticsearch store the data on Mac
Elastica - multiple bool queries - subqueries
How do i add boosting to the elasticsearch mapping in ONGR?
Is it possible to use doc_count as a term in elasticsearch?
Elastic search aggregation with sort
Elasticsearhc- how to do multi search request on Sense or Head plugin
How do I configure/initialize a custom Elasticsearch Tokenizer
Combining log entries with logstash
How to import CSV data using Logstash for field type completion in Elasticsearch suggester
How can I interpret the result of percentiles aggregation type
elastic search select fields to return in aggregation
How to Focus Search On One Part of a Document
How to calculate cooccurrences on a set of texts with Elasticsearch