Monitoring Elasticsearch Log Statistics Using the Moein Platform
Elasticsearch is one of the most widely used search and data analytics engines, enabling fast storage, search, and analysis of data in JSON format. One of its most common use cases is storing and analyzing service and application logs. To use Elasticsearch effectively, you first need to understand the structure of the stored data and learn how to extract it properly.
In this article, we show how to perform more accurate searches and analyses in Elasticsearch by running basic queries, understanding the underlying data structure, and applying common filters. Alongside an introduction to widely used filters, we also provide practical examples that help you quickly and easily extract and analyze the data you need.
One practical use of these queries—discussed throughout this article—is extracting log statistics for one or more specific statuses within a defined time range. These statistics can help you assess service health and can therefore be used for monitoring service status. You can also use these queries in the Moein Monitoring Platform, specifically in the Custom Indicators section, to record near-real-time log statistics in Moein and use them for monitoring and analysis.
A Brief Overview of Logs Stored in Elasticsearch
To query Elasticsearch correctly, you should first become familiar with the general structure of the data stored in your indices. Elasticsearch responses are typically returned in JSON format, and reviewing the full output helps you identify the fields you need for filtering, counting, and analysis.
A common first step is to run a general query that returns sample documents without applying any filters. This allows you to inspect the structure of the returned data and determine which fields should be used in more specific queries later.
To retrieve a sample JSON output and inspect the overall structure, you can use the following query and review all fields in the returned documents.
Example query to inspect JSON structure:
curl -u elastic:moein@123 -k -XPOST "http://localhost:9200/filebeat-*/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"size": 2
}' Running the command above returns a full response without any specific time range. To avoid heavy queries, a size limit is set to prevent retrieving too many documents unnecessarily.Note that the index name is specified in the request path after the port number. Therefore, you must set the correct index in the request URL.
Common Conditions and Filters in Elasticsearch Queries
Elasticsearch supports different query endpoints, but the most common ones are:
- /_count?pretty for count-only queries
- /_search?pretty for queries that return full results
To narrow down results and extract the data you need, you can use various conditions and filters. Below are some of the most common ones.
Term
Typically used inside the filter block. The value must exactly match the target field value:
{ "term": { "responseCode": 200 } }For non-numeric values or complex/composite strings, you should explicitly define the value using a value structure:
{"term": { "clientip": { "value": "130.246.123.197" } }}Terms
Similar to term, but allows checking multiple values at the same time:.
{ "terms": { "responseCode": [500, 404] } }Range
Used to evaluate a numeric range or a specific time window. The available operators are:
- gt: Greater than
- gte: Greater than or equal to
- lt: Less than
- lte: Less than or equal to
Example (last 5 minutes):
{
"range": {
"@timestamp": {
"gte": "now-5m",
"lt": "now"
}
}
}Exists
Checks whether a specific field exists in a document. A document matches if the field is present and has any value:
"exists": { "field": "userId" }There are also other filters such as prefix, wildcard, ids, and more, which you can use depending on your needs.
Practical Examples and Useful Elasticsearch Queries
- Count successful response codes (200) for in the last 5 minutes
curl -u elastic:moein@123 -k -XPOST "http://localhost:9200/filebeat-
*/_count?pretty"\
-H 'Content-Type: application/json' -d'
{ "query": { "bool": { "filter": [ { "term": { "responseCode": 200 } }, { "term": { "serviceName": "testservice" } }, { "range": { "@timestamp": { "gte": "now-5m", "lt": "now" } } } ] } } }'
- Count failed response codes (500 and 404) for in the last 15 minutes
curl -u elastic:moein@123 -k -XPOST "http://localhost:9200/filebeat-
*/_count?pretty"\
-H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"filter": [
{ "terms": { "responseCode": [500, 404] } },
{ "term": { "serviceName": "testservice" } },
{
"range": {
"@timestamp": {
"gte": "now-15m",
"lt": "now"
}
}
}
]
}
}'- Count successful response codes (200) for from 60 minutes ago to 15 minutes ago (excluding the last 15 minutes)
curl -u elastic:moein@123 -XPOST "http://localhost:9200/filebeat-
*/_count?pretty"\
-H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"filter": [
{ "term": { "responseCode": 200 } },
{ "term": { "serviceName.keyword": "testservice" } },
{ "range": {
"@timestamp": {
"gte": "now-60m",
"lt": "now-15m"
}
}
}
]
}
}
}'Conclusion
In this article, we reviewed the practical fundamentals of querying Elasticsearch logs—from inspecting JSON structure using a match_all query to narrowing results with common filters such as term, terms, range, and exists. We also walked through practical examples for counting results and applying different time windows With these techniques, you can build more advanced queries and extract more accurate, targeted insights from Elasticsearch. Finally, you can directly use the provided curl count queries in the Moein Monitoring Platform under Custom Indicators to continuously monitor log statistics based on specific conditions and time ranges. This integration enables proactive service monitoring, timely alerting, and data-driven insights into your application's health and performance.
The Latest Articles
