“An Elasticsearch datastream refers to a collection of automatically generated indices that are used to store logs, metrics, trace data, and other automatically generated data. A data stream requires a matching index template containing mappings and settings to configure the indices.
This article will cover the fundamentals of working with Elasticsearch data streams using the fetch data stream API.”
Let’s explore.
Request Syntax
The following shows the syntax for the get data stream API.
GET /_data_stream/<data-stream>
Path parameters:
- <data-stream> – a list of comma separated data stream names.
Example
Create data stream
PUT my_data_stream/_bulk
{ "create":{ } }
{ "@timestamp": "2022-09-06T16:21:15.000Z", "message": "192.168.0.101 - Accessed" }
{ "create":{ } }
{ "@timestamp": "2022-09-06T16:25:42.000Z", "message": "192.168.0.110 - Accessed" }
POST my_data_stream/_doc
{
"@timestamp": "2022-09-06T16:21:15.000Z",
"message": "192.168.0.101 - - Accessed"
}
{ "create":{ } }
{ "@timestamp": "2022-09-06T16:21:15.000Z", "message": "192.168.0.101 - Accessed" }
{ "create":{ } }
{ "@timestamp": "2022-09-06T16:25:42.000Z", "message": "192.168.0.110 - Accessed" }
POST my_data_stream/_doc
{
"@timestamp": "2022-09-06T16:21:15.000Z",
"message": "192.168.0.101 - - Accessed"
}
Output
{
"took": 3,
"errors": false,
"items": [
{
"create": {
"_index": "my-data-stream",
"_id": "oT6AXIMB1yF5Vqfa_Scu",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 3,
"_primary_term": 1,
"status": 201
}
},
{
"create": {
"_index": "my-data-stream",
"_id": "oj6AXIMB1yF5Vqfa_Scu",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 4,
"_primary_term": 1,
"status": 201
}
}
]
}
{
"_index": "my-data-stream",
"_id": "oz6BXIMB1yF5VqfaViem",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 5,
"_primary_term": 1
}
"took": 3,
"errors": false,
"items": [
{
"create": {
"_index": "my-data-stream",
"_id": "oT6AXIMB1yF5Vqfa_Scu",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 3,
"_primary_term": 1,
"status": 201
}
},
{
"create": {
"_index": "my-data-stream",
"_id": "oj6AXIMB1yF5Vqfa_Scu",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 4,
"_primary_term": 1,
"status": 201
}
}
]
}
{
"_index": "my-data-stream",
"_id": "oz6BXIMB1yF5VqfaViem",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 5,
"_primary_term": 1
}
Fetch the Data Stream
curl -XGET "http://localhost:9200/my_data_stream" -H "kbn-xsrf: reporting"
<h2>Output</h2>
{
"my_data_stream": {
"aliases": {},
"mappings": {
"properties": {
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"username": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "1",
"provided_name": "my_data_stream",
"creation_date": "1663703715012",
"number_of_replicas": "1",
"uuid": "OffUEhVkRn6RK1ql2RPReg",
"version": {
"created": "8040199"
}
}
}
}
}
<h2>Output</h2>
{
"my_data_stream": {
"aliases": {},
"mappings": {
"properties": {
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"username": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "1",
"provided_name": "my_data_stream",
"creation_date": "1663703715012",
"number_of_replicas": "1",
"uuid": "OffUEhVkRn6RK1ql2RPReg",
"version": {
"created": "8040199"
}
}
}
}
}
End
In this post, we discussed how to use the Elasticsearch get data stream API to fetch detailed information about the data streams available in your cluster.
Thanks for reading, and Happy coding!!
