Edit

Index Lifecycles

Learn How to to use Index Lifecycle Policies to transition time series data

For time-series data, such as logs or metrics, you may wish to transition or even delete older data after a period of time. Since Elasticsearch v7, ILM policies allow you to configure the transition and retention of your data. The ILMPolicyBuilder object helps facilitate the configuration of of ILM for your data streams or time-series indices. For an ILM policy to work, your documents must have a field of @timestamp. If your existing indices do not have this field, you can add it via an updateByQuery script using an existing timestamp field.

For more information, head to the ILM Overview in the Elasticsearch documentation.

Policy Creation

Let's create a simple policy to delete our data after 30 days:

getInstance( "ILMPolicyBuilder@cbelasticsearch" )
        .new( 
            "my-ilm-policy"
        ).withDeletion(
            age = 30
        ).save();

In order to attach the policy to an existing index you can specify the index.lifecycle.name in the index settings or in your Component or Index Templates;

getInstance( "IndexBuilder@cbelasticsearch" )
                .new( 
                    name="my-index-name", 
                    settings={ 
                        "index.lifecycle.name" : "my-ilm-policy" 
                    } 
                )
                .save();

Phased Rollover and Archival

You may also wish to transition data between phases, and consolidate or shrink your datasets as they age. This can be done by using lifeycle phases.

Let's create a more complex index lifecycle:

ILMPolicyBuilder Method Signatures

new

hotPhase

warmPhase

coldPhase

frozenPhase

Note that this phase may not be used without either the searchableSnapshot or unfollow arguments passed

withDeletion

PreviousManaging IndicesNextIndex Templates

Last updated

Was this helpful?