Conveniently, Filebeat can parse JSON since 5.0. Ideally, you’d log in JSON and push directly to Elasticsearch. It turned out that network was the bottleneck, which is why pushing raw logs doesn’t saturate the CPU:Įven though we got a healthy throughput rate of 12-14K EPS:īut raw, unparsed logs are rarely useful. We also installed Sematext agent to monitor Elasticsearch performance. We used an AWS c3.large for Filebeat (2 vCPU) and a c3.xlarge for Elasticsearch (4 vCPU).
To get a baseline, we pushed logs with Filebeat 5.0alpha1 directly to Elasticsearch, without parsing them in any way. This way we could also check how both Ingest ’s Grok processors and Logstash ’s Grok filter scale when you start adding more rules.īaseline performance: Shipping raw and JSON logs with Filebeat Specifically, we tested the grok processor on Apache common logs (we love logs here), which can be parsed with a single rule, and on CISCO ASA firewall logs, for which we have 23 rules. Is it worth sending data directly to Elasticsearch or should we keep Logstash? We decided to take it for a spin and see how this new functionality (called Ingest) compares with Logstash filters in both performance and functionality. Unless you are using a very old version of Elasticsearch you’re able to define pipelines within Elasticsearch itself and have those pipelines process your data in the same way you’d normally do it with something like Logstash. Application Performance Monitoring Guide.
0 kommentar(er)
