elasticsearch


How can I use Kafka to retain logs in logstash for longer period?


Currently I use redis -> s3 -> elastic search -> kibana stack to pipe and visualise my logs. But due to large volume of data in elastic search I can retain logs upto 7 days.
I want to bring kafka cluster in this stack and retain logs for more number of days. I am thinking of following stack.
app nodes piping logs to kafka -> kafka cluster -> elastics search cluster -> kibana
How can I use kafka to retain logs for more number of days?
Looking through the Apache Kafka broker configs, there are two properties that determine when a log will get deleted. One by time and the other by space.
log.retention.{ms,minutes,hours}
log.retention.bytes
Also note that if both log.retention.hours and log.retention.bytes are both set we delete a segment when either limit is exceeded.
Those two dictate when logs are deleted in Kafka. The log.retention.bytes defaults to -1, and I'm pretty sure leaving it to -1 allows only the time config to solely determine when a log gets deleted.
So to directly answer your question, set log.retention.hours to however many hours you wish to retain your data and don't change the log.retention.bytes configuration.

Related Links

Get latest records in elastic search
Put filtered mustache search template on AWS elasticsearch cluster using javascript API
Get Percentage of Values in Elasticsearch
elasticsearch - aggregating counts on array matches
ElasticSearch has_child query does not support query_string
elastic search autocomplete, searching with space creating an issue
(ELK) logstash grok for SBR accounting log
aws cloudsearch/lucene query street names
Getting cardinality of multiple fields?
Aggregating a Key/Value list in ElasticSearch
“reverse cardinality” in elasticsearch?
ElasticSearch- Using Fields doesn't return any documents on Nest
Analyzer to find , e.g: “starbucks” when mistakenly querying “star bucks”
Elasticsearch - boost document based on field's specific value
How to get elasticsearch most used words?
Umlaut in Elastic Suggesters

Categories

HOME
events
macos-sierra
plesk
lambda
yii
android-emulator
biztalk-2010
repair
tfs2010
port
driver
documentum
amazon-elb
trace32
texas-instruments
synthesis
hidden
playframework-2.0
google-sites
varnish-vcl
raml
bluemix-mobile-services
spyder
libgit2
liferay-6.2
ipa
macromedia
source-insight
infopath2010
image-optimization
microsoft-ui-automation
contactless-smartcard
google-prediction
latexml
nomad
asihttprequest
audio-converter
flume-twitter
cyanogenmod
sonicmq
subforms
htmlspecialchars
nivo-slider
portability
dymola
adobe-reader
gawk
annotatorjs
gapi
ndk-build
system.web.optimization
coovachilli
odp.net-managed
flush
search-form
fantom
relational-model
erlog
shoes
innerhtml
maven-shade-plugin
datagridcomboboxcolumn
iodocs
squeezebox
jms-serializer
yii-booster
insertonsubmit
disjoint-union
void-pointers
sly-scroller
smartfox
cgrectmake
towers-of-hanoi
xcode4.5
websphere-6.1
server-name
kolite
page-curl
bass
ccl
.net-services
mysqli-multi-query
projectpier
version-control-migration

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile