elasticsearch


logstach kafka input decode __consumer_offsets message


I'm trying to use logstash to consume messages from the __consumer_offsets toipc. I can successfully get the messages via logstash but the messages are either encoded or compressed.
My logstash config looks like this:
input {
kafka {
topics => ["__consumer_offsets"]
bootstrap_servers => "kafka1.cloud.domain.com:9092"
jaas_path => "/home/ec2-user/config/kafka.client.jaas.conf"
kerberos_config => "/home/ec2-user/config/krb5.conf"
security_protocol => "SASL_SSL"
ssl_truststore_location => "/home/ec2-user/config/kafka.client.truststore.jks"
ssl_truststore_password => "pass1"
ssl_truststore_type => "jks"
sasl_kerberos_service_name => "kafka"
type => "kafka_mbeans"
consumer_threads => 4
}
}
output {
stdout { codec => rubydebug }
}
but the output comes out like this:
{
"#timestamp" => 2017-03-29T17:54:57.333Z,
"#version" => "1",
"message" => "\u0000\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0000\u0001[\e4�\\\u0000\u0000\u0001[ [\u0019\\",
"type" => "kafka_mbeans"
}
{
"#timestamp" => 2017-03-29T17:54:57.333Z,
"#version" => "1",
"message" => "\u0000\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0000\u0001[\e4�\\\u0000\u0000\u0001[ [\u0019\\",
"type" => "kafka_mbeans"
}
{
"#timestamp" => 2017-03-29T17:54:57.333Z,
"#version" => "1",
"message" => "\u0000\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0003\u0000\u0000\u0000\u0000\u0001[\e4�\\\u0000\u0000\u0001[ [\u0019\\",
"type" => "kafka_mbeans"
}
If I run console consumer, I console from that topic like this:
kafka-console-consumer --bootstrap-server kafka1.cloud.domain.com:9092 --topic __consumer_offsets --new-consumer --consumer.config /home/ec2-user/client.properties --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"
notice the kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter, I beieve I need to use something similar in Logstash to decode the messages...

Related Links

ElasticSearch add sorting attribute to mget call
Couch Db - indexing and searching
ElasticsearchIllegalStateException[[index.version.created] is not present in the index settings for index with uuid: [null]]
Join two separate nodes into one cluster?
Elasticsearch - combining query_string and bool query in filter
Issue while querying on a field that store a file path on ElasticSearch
elasticsearch/logstash and logstash-contrib: Couldn't find any plugin named 'x'
What is the difference between Lucene and Elasticsearch
Cannot create Phoenix JDBC river in Elasticsearch
Disabling field analyzing by default in elastic search
time difference in elasticsearch took and that calculated manually
elasticsearch comparison between fields
Find all ID where ID are not in my blacklist
How to order results by custom value and then by _score
Kibana homepage shows blank after deleting all indices
elasticsearch: update in a nested object (HTTP)

Categories

HOME
opencart
eclipse-plugin
livecode
couchdb
yql
tcp
fogbugz
openflow
swift2
implicit-conversion
video-streaming
scheduler
cisco
mailing-list
scrape
cname
reselect
google-cloud-logging
robolectric
gmp
spam
angular2-highcharts
node-webkit
gitkraken
pyopencl
raima
cellular-network
offline-caching
android-cardview
miniprofiler
cube
dnsmasq
region
jsonresult
intune
d3.js-v4
fotorama
appcode
spring-rabbitmq
jfxtras
subclassing
cordys-opentext
text-decorations
sbjson
netapp
suds
data-uri
secure-coding
optionaldataexception
jqgrid-asp.net
audio-converter
dredd
sonarqube5.2
rm
icecast
amazon-elastic-beanstalk
line-intersection
mura
gnome-terminal
popen
zend-server
wildfly-9
email-injection
petsc
s-function
livereload
raw-sockets
static-methods
processor
i2b2
voronoi
outlook.com
ibm-was
opencyc
http.client
auto-generate
knife
uitest
modern.ie
cache-manifest
karma-coverage
ampersand
scala-macros
tkx
hp-idol-ondemand
comctl32
xhprof
back
rpg
forio-contour
thruway
jomsocial
assetic
armv6
pbkdf2
sql-view
libxml-js
unicoins
aero
cmath
django-filebrowser
daap
clearinterval
image-formats
pageheap
hardcode
prototypal-inheritance
drawimage
uipasteboard
upshot
nsconnection
google-translator-toolkit
main-method
inversion
rtti
efs
bindable-linq

Resources

Encrypt Message