elasticsearch


How to parse date in elasticsearch 5.x and Filebeat


I am using elasticsearch 5.x and Filebeat and want to know if there is a way of parsing date(timestamp) directly in filebeat (don't want to use logstash). I am using json.keys_under_root: true and it works great, but the problem is that timestamp (on us) is recognised as string. All of the other fields were automatically recognised as correct types only this one isn't.
How can I map it as date?
You can use Filebeat with the ES Ingest Node feature to parse your timestamp field and apply the value to the #timestamp field.
You would setup a simple pipeline in Elasticsearch that applies a date to incoming events.
PUT _ingest/pipeline/my-pipeline
{
"description" : "parse timestamp and update #timestamp",
"processors" : [
{
"date" : {
"field" : "timestamp",
"target_field" : "#timestamp"
}
},
{
"remove": {
"field": "timestamp"
}
}
],
"on_failure": [
{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
Then in Filebeat configure the elasticsearch output to push data to your new pipeline.
output.elasticsearch:
hosts: ["http://localhost:9200"]
pipeline: my-pipeline

Related Links

Elasticsearch: Updating a field that has been set as a document _id via mapping with a path
Return parent data with child document from Elasticsearch
Elasticsearch Giving Incorrect Result When Using “must_not” operator alongwith “must” Operator
ElasticSearch _suggest queries are case sensitive. Want them to be case insensitive
How to normalize periods in elastic search query (such that JJ Abrams == J.J Abrams)?
Elastic Search - Sort By Doc Type
ElasticSearch Filtering aggregations from array field
Full text search for exact match_phrase (with leading and trailing whitespace) in elasticsearch
Do two equal documents in elasticsearch double the needed disc space
In logstash/ElasticSearch/Kibana, how do I augment records following a particular stateful request?
How to know which shard is overloaded in Elasticsearch?
ElasticSearch - searching different doc_types with the same field name but different analyzers
ELK Type Conversion - Not a number but a string
How to get the definitiion of a search analyzer of an index in elasticsearch
ElasticSearch query using match or term?
How to combine aggregations in ElasticSearch/Kibana?

Categories

HOME
hpoo
elasticsearch
openshift
mc
list
lambda
windows-8.1
loopbackjs
visual-studio-2010
semantic-web
dronekit-python
gallery
biztalk-2010
bittorrent
ag-grid
firebase-dynamic-links
tweepy
port
log4j2
yeoman-generator-angular
slide
watch-os-3
erd
phpstorm-2017.1
createjs
distributed-computing
xbox360
orange-api
google-sites
varnish-vcl
pfobject
spyder
weex
coreldraw
maven-surefire-plugin
liferay-6.2
gollum-wiki
conda
sybase-ase
pyopencl
running-object-table
marquee
python-imageio
macromedia
sparkle
unmarshalling
openbr
instructions
windows2012
seafile-server
gsp
gnu-classpath
wdk
installanywhere
gitolite
optionaldataexception
doctrine-extensions
flume-twitter
hibernate-ogm
simple-schema
prefetch
mifos
cefpython
carmen
meld
deferred-rendering
gnucash
android-syncadapter
adjacency-list
swiffy
persistent-object-store
univocity
helm
miniconda
smart-tv
largenumber
place
asp.net-web-api-helppages
canopy
espresso
org-babel
whitespace-language
nanomsg
tabris
communicate
prism.js
code-complexity
sqlbindparameter
usb-flash-drive
vlab
mp4parser
lru
rikulo
contextswitchdeadlock
office-app
gprof
xcode4.5
voice-recording
fbml
nsrangeexception
winsxs
startupscript
kernel32
simile
icicles
main-method
icon-language
server-load
remote-working

Resources

Encrypt Message