elasticsearch


Converting nginx access log bytes to number in Kibana4


I would like to create a visualization of the sum of bytes sent using the data from my nginx access logs. When trying to create a "Metric" visualization, I can't use the bytes field as a sum because it is a string type.
And I'm not able to change it under settings.
How do I go about changing this field type to a number/bytes type?
Here is my logstash config for nginx access logs
filter {
if [type] == "nginx-access" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}
}
Since each logstash index is being created as an index, I'm guess I need to change it here.
I tried adding
mutate {
convert => { "bytes" => "integer" }
}
But it doesn't seem to make a difference.
Field types are configured using mappings, which is configured at the index level and can hardly change. With Logstash, as a new index is created everyday, so if you wan't to change these mappings either wait for the next day or delete the current index if you can.
By default these mappings are generated automatically by Elasticsearch depending on the syntax of the indexed JSON document and the applied Index Templates:
# Type String
{"bytes":"123"}
# Type Integer
{"bytes":123}
In the end there are 2 solutions:
Tune Logstash, to make it generate an integer and let Elasticsearch guess the field type → Use the mutate/convert filter
Tune Elasticsearch, to force the field bytes for the document type nginx-access to be of type integer → Use Index Template:
Index Template API:
PUT _template/logstash-nginx-access
{
"order": 1,
"template": "logstash-*",
"mappings": {
"nginx-access": {
"properties": {
"bytes": {
"type": "integer"
}
}
}
}
}

Related Links

Any way to have kibana 4 send alerts or take action on specific conditions
Create a geo_point from two different fields in ElasticSearch
Does ElasticSearch store a duplicate copy of each record?
Dynamic Index with SpringData ElasticSearch
how can I get ElasticSearch cluster configuration
How to leverage logstash to index data but not generating extra fields from logstash
Understanding multi-fields to analyze the text once per language
Elasticsearch: get for a substring in the value of a document field?
Elasticsearch completion suggester context for nested fields
Elasticsearch: type of fields in mapping is different from type in query result
It appears that we have not received any data for this cluster?
Removing From ElasticSearch by type last 7 day
Elasticsearch: accuracy on a filter aggregation
Wildcard query over _all field on Elasticsearch
Showing unmatched word in elasticsearch results
Elasticsearch: querying on both nested object properties and parent properties

Categories

HOME
php
knockout.js
freeradius
windows-8.1
hid
biztalk
greasemonkey
stored-procedures
wagtail
log4j2
driver
scrape
alljoyn
game-maker
webstore
rest-assured
tomcat8
fosuserbundle
scheduled-tasks
stackexchange.redis
intel
quartz.net
animated-gif
django-import-export
unmarshalling
envoy
opnet
char-pointer
netstat
pdfnet
android-n
stat
utf
classnotfoundexception
vesta
docker-ucp
qvtkwidget
android-5.0-lollipop
supertest
photoswipe
brightscript
redisson
webvtt
ptvs
bpms
rational-performance-test
e
openquery
raw-sockets
bbc-micro
sharepoint-apps
teamviewer
as3-api
vim-plugin
webgrind
cortana
astropy
dnx50
asp.net-web-api-helppages
android-studio-import
gluon-desktop
unhandled
wlanapi
libz
swagger-maven-plugin
mxe
signed
ssmtp
inputbox
jmenubar
disjoint-union
eager-loading
jquery-cycle
smartfox
wndproc
jdom
gamesalad
trialware
hardcode
mkannotation
wcf-callbacks
groovy-console
qt-mobility
nosetests
nsconnection
printqueue
bass
downcasting
levels
rendering-engine

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile