elasticsearch


Elasticsearch Unreachable: Connection refused


I have the ELK stack running inside docker containers inside a VM.
I can curl stuff to ES and it shows up in Kibana just fine.
I can read files with Logstash and output them to stdout.
But Logstash fails to send the data to ES
docker-compose.yml
version: '2'
services:
elasticsearch:
image: elasticsearch
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
xpack.security.enabled: "false"
xpack.monitoring.enabled: "false"
xpack.graph.enabled: "false"
xpack.watcher.enabled: "false"
networks:
- elk
logstash:
image: docker.elastic.co/logstash/logstash:5.3.2
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
- ./logstash/pipeline:/usr/share/logstash/pipeline
- ./data:/usr/share/data
ports:
- "5000:5000"
- "9600:9600"
networks:
- elk
depends_on:
- elasticsearch
kibana:
# kibana config
networks:
elk:
driver: bridge
logstash.yml (enabling or disabling xpack doesnt seem to make a difference)
#log.level: debug
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.url: http://127.0.0.1:9200
xpack.monitoring.enabled: false
pipeline.conf
input {
# some files
}
filter{
# csv filter
}
output {
if ([type] == "GPS") {
stdout { }
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
index => "GPS"
template_overwrite => true
user => logstash_system
password => changeme
}
}
}
The output of docker-compose up
elasticsearch_1 [2017-05-03T05:06:29,821][INFO ][o.e.n.Node ] initialized
elasticsearch_1 [2017-05-03T05:06:29,821][INFO ][o.e.n.Node ] [SzDlw_j] starting ...
elasticsearch_1 [2017-05-03T05:06:30,352][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: 74:70:60:88:e2:f8:41:36
elasticsearch_1 [2017-05-03T05:06:30,722][INFO ][o.e.t.TransportService ] [SzDlw_j] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
elasticsearch_1 [2017-05-03T05:06:30,744][WARN ][o.e.b.BootstrapChecks ] [SzDlw_j] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
elasticsearch_1 [2017-05-03T05:06:34,027][INFO ][o.e.c.s.ClusterService ] [SzDlw_j] new_master {SzDlw_j}{SzDlw_jiQsWcXN00BZfZHQ}{ELB1oDR4SlW6atQIM88hfg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch_1 [2017-05-03T05:06:34,176][INFO ][o.e.h.n.Netty4HttpServerTransport] [SzDlw_j] publish_address {172.18.0.2:9200}, bound_addresses {[::]:9200}
elasticsearch_1 [2017-05-03T05:06:34,179][INFO ][o.e.n.Node ] [SzDlw_j] started
elasticsearch_1 [2017-05-03T05:06:34,832][INFO ][o.e.g.GatewayService ] [SzDlw_j] recovered [1] indices into cluster_state
elasticsearch_1 [2017-05-03T05:06:36,519][INFO ][o.e.c.r.a.AllocationService] [SzDlw_j] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
logstash_1 Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1 [2017-05-03T05:06:55,865][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
logstash_1 [2017-05-03T05:06:55,887][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
logstash_1 [2017-05-03T05:06:56,131][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x7b0aac02 URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash_1 [2017-05-03T05:06:56,140][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
logstash_1 [2017-05-03T05:06:56,147][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>http://127.0.0.1:9200/, :error_message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash_1 [2017-05-03T05:06:56,148][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:287:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:363:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:272:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:280:in `get'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:83:in `get_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:in `get_es_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:20:in `get_es_major_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/common.rb:54:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/common.rb:21:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:41:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:257:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:277:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:207:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `start_pipeline'"]}
logstash_1 [2017-05-03T05:06:56,149][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x6d11738 URL://127.0.0.1:9200>]}
logstash_1 [2017-05-03T05:06:56,163][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
logstash_1 [2017-05-03T05:06:56,703][INFO ][logstash.pipeline ] Pipeline main started
logstash_1 2017-05-03T05:06:56.776Z b68ff5316e02 time,lat,lon,elevation,accuracy,bearing,speed
logstash_1 2017-05-03T05:06:56.778Z b68ff5316e02 2014-09-14T00:26:23Z,98.404222,99.999021,9.599976,44.000000,0.000000,0.000000
logstash_1 2017-05-03T05:06:56.779Z b68ff5316e02 2014-09-14T00:48:45Z,98.404297,99.999338,9.000000,35.000000,102.300003,0.500000
logstash_1 [2017-05-03T05:06:57,035][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
logstash_1 [2017-05-03T05:06:57,058][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
logstash_1 [2017-05-03T05:06:57,069][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}
logstash_1 [2017-05-03T05:06:59,085][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
logstash_1 [2017-05-03T05:06:59,086][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
logstash_1 [2017-05-03T05:07:01,146][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
logstash_1 [2017-05-03T05:07:01,158][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x7b0aac02 URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
after that it keeps looping between "sending bulk request", "performing health check", "attempting to resurrect" and "unexpected pool error".
With log.level: debug every once in a while i also see Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/system.slice/docker-<containerId>.scope/cpuacct.usage"}
One of docker's purposes is isolation, from the point of view of the containers, 127.0.0.1 only refers to themselves.
Here you have 3 containers:
elasticsearch
logstash
kibana
You should modify your logstash configuration to put elasticsearch instead of 127.0.0.1, as it's the name it knows for the elasticsearch container on the elk bridge network you have defined

Related Links

Elasticsearch - Has_Parent or Has_Child query returning empty results
Elasticsearch search body from URL search
Fields not getting sorted in alphabetical order in elasticsearch
Sorting strings in elastic search
Missing data when using unique count and creating an aggregation in Kibana
elasticsearch filter array data with aggs
Sorting based on different properties with elasticsearch
Filter through goods and showing available options using Elasticsearch
Elasticsearch get field mapping for all fields without using asterisk (*)
Calculate conversion rate on events data Elasticsearch aggregations
Convert existing field mapping to geoip
How to filter using any of the full text queries?
Elasticsearch: private user documents
I am looking for ElasticSearch equivalent of below query -
Retrieve docs that contains only allowed tags (exactly equals)
Get max-score of Elasticsearch in CakePHP 3

Categories

HOME
flask
long-integer
subdomain
nsview
telegraf
scheduler
polymer-2.x
mutation-testing
parsley.js
mahout-recommender
heap-dump
c#-7.0
playframework-2.0
cellular-network
spreadsheetgear
code-climate
sendmessage
javax.persistence
jsonresult
elastica
utf
google-prediction
bit-shift
cordova-ios
comm
amazon-elastic-beanstalk
openstack-glance
watir-webdriver
line-intersection
winmerge
emacs25
pecl
nrf51
annotatorjs
streamwriter
mathjs
svnserve
nvda
system.net.mail
fiber
vimperator
outlook.com
360-degrees
r.java-file
string.format
rubber
libgcc
.htpasswd
cidr
tabletools
jchartfx
sql-view
apportable
ui-design
ofstream
examine
handshake
user-forums
tridion-2011
sfinae
msinfo32
series-40
office-automation
active-record-query
activestate
index.dat
autobench
error-detection

Resources

Encrypt Message