elasticsearch


Elasticsearch : Indexing rate slows down gradually


I am writing to ES from Spark streaming at a rate of around 80,000 EPS.
which was running fine earlier, but recently the indexing rate slows down gradually and the results in spark-jobs piling up.
what are the cluster settings which I can tweak/verify to address this issue
cluster settings :-
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"cluster_concurrent_rebalance": "160",
"node_concurrent_recoveries": "2",
"disk": {
"threshold_enabled": "true",
"watermark": {
"low": "85%",
"high": "95%"
}
},
"node_initial_primaries_recoveries": "40",
"enable": "all"
}
}
},
"indices": {
"breaker": {
"fielddata": {
"limit": "50%"
}
},
"recovery": {
"concurrent_streams": "80",
"max_bytes_per_sec": "50mb"
},
"store": {
"throttle": {
"max_bytes_per_sec": "500gb"
}
}
},
"threadpool": {
"bulk": {
"queue_size": "5000",
"size": "40",
"type": "fixed"
}
},
"discovery": {
"zen": {
"minimum_master_nodes": "2"
}
}
},
"transient": {
"cluster": {
"routing": {
"allocation": {
"enable": "all"
}
}
}
}
}
cluster status:-
{
"cluster_name": "**********",
"status": "green",
"timed_out": false,
"number_of_nodes": 105,
"number_of_data_nodes": 100,
"active_primary_shards": 7315,
"active_shards": 7330,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}

Related Links

Optimal way to set up ELK stack on three servers
Cluster Level Logging with Elasticsearch and Kibana does not work in kubernetes
ElasticSearch Couchbase Replication Issue
How to query for inner_hits against grandparents in multi-generational setup
Request timedout during delete/create on elasticsearch while snapshot is being taken
how to use two parallel Aggregation for elasticsearch nest
How do I enable remote access/request in Elasticsearch 2.0?
elasticsearch faking index per user - how are routing values inferred when updating?
elasticsearch: Proper config in 3 node cluster for each node to have full copy of index?
Multiple Nested Aggregations in Elasticsearch
Reconnection in Elasticsearch Cluster
Elasticsearch - Has_Parent or Has_Child query returning empty results
Elasticsearch search body from URL search
Fields not getting sorted in alphabetical order in elasticsearch
Sorting strings in elastic search
Missing data when using unique count and creating an aggregation in Kibana

Categories

HOME
hpoo
tomcat
workflow
couchdb
semantic-web
tabs
yql
mirc
gentelella
tfs2015
joomla3.2
selection
port
fbloginview
freertos
django-rq
office-ui-fabric
http-authentication
scheduled-tasks
spring-shell
spring-ldap
gitkraken
connection-refused
backendless
password-generator
riotjs
djcelery
ng-repeat
grunt-contrib-watch
suds
react-intl
gcal
section508
ol3-google-maps
head.js
algebra
apache-modules
rustdoc
visual-studio-monaco
gnucash
timestamping
gravatar
okta-api
cortana
diawi
arbre
ruby-on-rails-4.1
sqlbase
roxygen2
swagger-maven-plugin
android-handler
cache-manifest
actionpack
prism-4
tidyr
findersync
nhibernate-criteria
thoughtworks-go
relationships
oracle-adf-mobile
ecos
angular-ui-select
box2dweb
xmi
mod-perl
lync-2010
iosched
saga
infinite
postgres-xc
blobstorage
web-frameworks
chefspec
red-system
idispatch
playorm
zend-server-ce
office-automation
simile
mygeneration
ixmldomdocument
code-camp
synthesizer

Resources

Encrypt Message