# Getting Started


# Installation

Get started installing CBElasticsearch

Via CommandBox:

```bash
install cbelasticsearch
```

## Requirements

* Coldbox >= v6
* Elasticsearch >= v6
* Lucee >= v5 or Adobe Coldfusion >= v2018

{% hint style="danger" %}
*Note: Most of the REST-based methods will work on Elasticsearch versions older than v5.0. A notable exception is the multi-delete methods, which use the* [*delete by query*](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-delete-by-query.html) *functionality of ES5. As such, Cachebox and Logbox functionality would be limited.*
{% endhint %}


# Configuration

Learn CBElasticsearch module config, environment variable support, and more.

Once you have installed the module, you may add a custom configuration, specific to your environment, by adding an `cbElasticsearch` configuration object to your `moduleSettings` inside your `Coldbox.cfc` configuration file.

By default the following are in place, without additional configuration:

```
moduleSettings = {
    "cbElasticsearch" = {
        //The native client Wirebox DSL for the transport client
        client="HyperClient@cbelasticsearch",
        // The default hosts - an array of host connections
        //  - REST-based clients (e.g. JEST):  round robin connections will be used
        //  - Socket-based clients (e.g. Transport):  cluster-aware routing used
        versionTarget = getSystemSetting( "ELASTICSEARCH_VERSION", '' ),
        hosts = [
            //The default connection is made to http://127.0.0.1:9200
            {
                serverProtocol: getSystemSetting( "ELASTICSEARCH_PROTOCOL", "http" ),
                serverName: getSystemSetting( "ELASTICSEARCH_HOST", "127.0.0.1" ),
                serverPort: getSystemSetting( "ELASTICSEARCH_PORT", 9200 )
            }
        ],
        // The default credentials for access, if any - may also be overridden when searching index collections
        defaultCredentials = {
            "username" : getSystemSetting( "ELASTICSEARCH_USERNAME", "" ),
            "password" : getSystemSetting( "ELASTICSEARCH_PASSWORD", "" )
        },
        // The default index
        defaultIndex           = getSystemSetting( "ELASTICSEARCH_INDEX", "cbElasticsearch" ),
        // The default number of shards to use when creating an index
        defaultIndexShards     = getSystemSetting( "ELASTICSEARCH_SHARDS", 5 ),
        // The default number of index replicas to create
        defaultIndexReplicas   = getSystemSetting( "ELASTICSEARCH_REPLICAS", 0 ),
        // Whether to use separate threads for client transactions
        multiThreaded          = true,
        // The maximum amount of time to wait until releasing a connection (in seconds)
        maxConnectionIdleTime = 30,
        // The maximum number of connections allowed per route ( e.g. search URI endpoint )
        maxConnectionsPerRoute = 10,
        // The maxium number of connections, in total for all Elasticsearch requests
        maxConnections         = getSystemSetting( "ELASTICSEARCH_MAX_CONNECTIONS", 100 ),
        // Read timeout - the read timeout in milliseconds
        readTimeout            = getSystemSetting( "ELASTICSEARCH_READ_TIMEOUT", 3000 ),
        // Connection timeout - timeout attempts to connect to elasticsearch after this timeout
        connectionTimeout      = getSystemSetting( "ELASTICSEARCH_CONNECT_TIMEOUT", 3000 )
    }
};
```

At the current time only the REST-based \[Hyper] native client is available. Support is in development for a socket based-client. For most applications, however the REST-based native client will be a good fit.

{% hint style="warning" %}
*Elasticsearch v8 Note: Elasticsearch version greater than 8.0.0 have* [*XPack security*](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html) *enabled by default. In order to disable security you must pass the `xpack.security.enabled=false` environment variable to the service, or add this configuration to your `elasticsearch.yml` file. Without security disabled, you will need to provide credentials.*
{% endhint %}

## Configuration via Environment Variables

Since the default settings will read from environment variables if they exist, we can easily configure cbElasticsearch from a `.env` file:

```bash
# .env

# Configure elasticsearch connection
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9222
ELASTICSEARCH_PASSWORD=B0xify_3v3ryth1ng

# Configure data storage and retrieval
ELASTICSEARCH_INDEX=books
ELASTICSEARCH_SHARDS=5
ELASTICSEARCH_REPLICAS=0
ELASTICSEARCH_MAX_CONNECTIONS=100
ELASTICSEARCH_READ_TIMEOUT=3000
ELASTICSEARCH_CONNECT_TIMEOUT=3000
```

You will need to read these settings into the coldfusion server upon server start via `commandbox-dotenv` or some other method.

{% hint style="warning" %}
For security reasons, make sure to add `.env` to your `.gitignore` file to avoid committing environment secrets to github/your git server.
{% endhint %}


# Secondary Cluster

Learn how to connect to a secondary Elasticsearch cluster using CBElasticsearch

As of the current version, the module conventions only allow for a default connection to one cluster. Multi-cluster native configuration is planned for a future major release, as it will be a breaking change. You may, however, create a separate instance of the client to connect to a different cluster. Since this needs to be accomplished after the module is loaded, the easiest way to do this is to create an application-specific module which is dedicated to connecting to that cluster. A major caveat, at this time, however is that native CRUD methods in the `Document`, `SearchBuilder`, `IndexBuilder`, and `ElasticsearchAppender` components will not work, as they are hard-wired to connect to the main client. As such, execution will need to be performed through the separate client instance. If you wish to use the secondary cluster for logging, a new Appender will also need to be created.

Below is an example of creating a secondary client connection to an alternate cluster.

1. First create a new application module

```js
box coldbox create module name=SecondaryCluster directory=modules_app dependencies=cbElasticsearch
```

*Note: the above command will also create `views`, `models` and `handlers` directories. These can be removed as they will not be used.*

2. Once your module is created, open up the `ModuleConfig.cfc` and add `cbElasticsearch` to `this.dependencies`
3. Now change the `settings` object in the `configure()` method to use your new configuration. Note that we have omitted the `client` key. We do this in order to prevent usage of member functions in the internal objects, by ensuring an error is thrown if we attempt to invoke them. All transactions need to pass through the client.

```js
settings = {
    versionTarget = '7.0.0',
    hosts = [
        //In this example, our secondary is on the same server, different port
        {
            serverProtocol: "http",
            serverName: "elasticsearch-cluster2",
            serverPort: 9200
        }
    ],
    // keep these credentials, but leave blank
    defaultCredentials = {
        "username" : "",
        "password" : ""
    },
    defaultIndex           = "otherData",
    // The default number of shards to use when creating an index
    defaultIndexShards     = getSystemSetting( "ELASTICSEARCH_SHARDS", 5 ),
    // The default number of index replicas to create
    defaultIndexReplicas   = getSystemSetting( "ELASTICSEARCH_REPLICAS", 0 ),
    // Whether to use separate threads for client transactions
    multiThreaded          = true,
    // The maximum amount of time to wait until releasing a connection (in seconds)
    maxConnectionIdleTime = 30,
    // The maximum number of connections allowed per route ( e.g. search URI endpoint )
    maxConnectionsPerRoute = 10,
    // The maxium number of connectsion, in total for all Elasticsearch requests
    maxConnections         = getSystemSetting( "ELASTICSEARCH_MAX_CONNECTIONS", 100 ),
    // Read timeout - the read timeout in milliseconds
    readTimeout            = getSystemSetting( "ELASTICSEARCH_READ_TIMEOUT", 3000 ),
    // Connection timeout - timeout attempts to connect to elasticsearch after this timeout
    connectionTimeout      = getSystemSetting( "ELASTICSEARCH_CONNECT_TIMEOUT", 3000 )
};
```

4. Now that we have our settings in place, add our new bindings to the internals `onLoad` method

```js
// map a new singleton instance of the config client
binder.map( "Config@SecondaryCluster" )
                .to( 'cbelasticsearch.models.Config' )
                .threadSafe()
                .asSingleton();

var secondaryConfig = wirebox.getInstance( "Config@SecondaryCluster" );

// override the module-injected config struct to our new configuration
// note that we need a full config structure passed in as an override to the coldbox settings
secondaryConfig.setConfigStruct( settings );

// note that we are using the native JEST client rather than Client@cbelasticsearch
binder.map( "Client@SecondaryCluster" )
                        .to( "cbElasticsearch.models.JestClient" )
                        .initWith( configuration=secondaryConfig )
                        .threadSafe()
                        .asSingleton();

```

5. After you have created your bindings, make sure you add a closing routine in your `onUnload` method for the client when the module is unloaded ( e.g. during a framework reinit ):

```js
// Close all active pool connections - necessary for native driver implementation
if( wirebox.containsInstance( "Client@SecondaryCluster" ) ){
    wirebox.getInstance( "Client@SecondaryCluster" ).close();
}
```

Now you may perform a search, considering the caveat that the search must now be executed through the client:

```js
var searchBuilder = getInstance( "SearchBuilder@cbelasticsearch" ).new( "myOtherIndex" );
searchBuilder.term( "foo", "bar" );

var searchResult = getInstance( "Client@SecondaryCluster" ).executeSearch( searchBuilder );
```

Document saves, retrievals, and deletions would need to be routed through the client, as well, rather than using the `save()` function:

```js
var newDocument = getInstance( "Document@cbelasticsearch" ).new( { "id" : createUUID(), "foo" : "bar" } );
getInstance( "Client@SecondaryCluster" ).save( newDocument );


var existingDocument = getInstance( "Client@SecondaryCluster" ).get( newDocument.getId() );
getInstance( "Client@SecondaryCluster" ).delete( existingDocument );
```

As you can see, connecting to a secondary Elasticsearch cluster, while not as fluent, is workable. Version 2.0 of this module has multi-cluster support planned via the native configuration.


# Indices


# Managing Indices

Learn how to create, update and delete indices with CBElasticsearch

Elasticsearch documents are stored in an "index", with the document structure defined by a "mapping". An Elasticsearch index is a JSON document store, and the mapping is a JSON configuration which defines the data type Elasticsearch should use for each document field.

By default, Elasticsearch will dynamically generate these index mapping when a document is saved to the index. See [Dynamic Mappings in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html) for more details.

## Retrieving information on Indices

To retrieve a list of all indices on the connected cluster, use the client `getIndices` method:

```js
var indexMap = getInstance( "Client@cbelasticsearch" ).getIndices();
```

This will return a struct of all indices ( with the names as keys ), which will provide additional information on each index, such as:

* Any assigned aliases
* The number of documents in the index
* The size of the storage space used for the index in bytes

## Creating an Index

The `IndexBuilder` model assists with the creation and mapping of indices. Mappings define the allowable data types within your documents and allow for better and more accurate search aggregations. Let's say we have a book model that we intend to make searchable via a `bookshop` index. Let's go ahead and create the index using the IndexBuilder:

```js
var indexBuilder = getInstance( "IndexBuilder@cbelasticsearch" ).new( "bookshop" ).save();
```

This will create an empty index which we can begin populating with documents.

## Creating an Explicit Index Mapping

To avoid the inherent troubles with dynamic mappings, you can define an explicit mapping using the `properties` argument:

```js
getInstance( "IndexBuilder@cbelasticsearch" )
    .new(
        name = "bookshop",
        properties = {
            "title" : { "type" : "text" },
            "summary" : { "type" : "text" },
            "description" : { "type" : "text" },
            // denotes a nested struct with additional keys
            "author" : { "type" : "object" },
            // date with specific format type
            "publishDate" : {
                "type" : "date",
                // our format will be = yyyy-mm-dd
                "format" = "strict_date"
            },
            "edition" : { "type" : "integer" },
            "ISBN" : { "type" : "integer" }
        }
    )
    .save();
```

{% hint style="info" %}
While it is not *required* that you explicitly define an index mapping, it is **highly recommended** since Elasticsearch's assumptions about the incoming document data may not always be correct. This leads to issues where the Elasticsearch-generated mapping is wrong and prevents further data from being indexed if it does not match the expected data type.
{% endhint %}

## Using Client.ApplyIndex

In the previous examples, we've created the index and mapping from the IndexBuilder itself. If we wish, we could instead pass the `IndexBuilder` object to the `Client@cbelasticsearch` instance's `applyIndex( required IndexBuilder indexBuilder )` method:

```js
var myNewIndex = indexBuilder.new( "bookshop" )
                    .populate( getInstance( "BookshopIndexConfig@myApp" ).getConfig() );
getInstance( "Client@cbelasticsearch" ).applyIndex( myNewIndex );
```

## Configuring Index Settings

So far we've passed a simple struct of field mappings in to the index properties. If we wanted to add additional settings or configure replicas and shards, we could pass a more comprehensive struct, including a [range of settings](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index-modules.html) to the `new()` method to do so:

```js
indexBuilder.new(
    "bookshop",
    {
        "settings" : {
            "number_of_shards" : 10,
            "number_of_replicas" : 2,
            "auto_expand_replicas" : true,
            "shard.check_on_startup" : "checksum"
        },
        "mappings" : {
            "properties" : {
                "title" : { "type" : "text" },
                "summary" : { "type" : "text" },
                "description" : { "type" : "text" },
                // denotes a nested struct with additional keys
                "author" : { "type" : "object" },
                // date with specific format type
                "publishDate" : {
                    "type" : "date",
                    // our format will be = yyyy-mm-dd
                    "format" : "strict_date"
                },
                "edition" : { "type" : "integer" },
                "ISBN" : { "type" : "integer" }
            }
        }
    }

);
```

## Updating an Existing Index

The `IndexBuilder` model also provides a `patch()` convenience method for updating the mapping or settings on an index:

```js
indexBuilder.patch(
    name = "bookshop",
    settings = {
        "number_of_shards"      : 10,
        "number_of_replicas"    : 2,
        "auto_expand_replicas"  : true,
        "shard.check_on_startup": "checksum"
    }
);
```

Here's a quick example of using `indexBuilder.patch()` to add two new fields to an existing `reviews` index:

```js
indexBuilder.patch(
    name = "reviews",
    properties = {
        "authorName"   : { "type" : "text" },
        "helpfulRating": { "type" : "integer" }
    }
);
```

## Retrieving Settings for an Index

To retreive a list of all settings for an index you may use the `getSettings` method on the client.

```js
var indexSettings = getInstance( "Client@cbelasticsearch" ).getSettings( "bookshop" )
```

## Retrieving Mappings for an Index

To retreive a list of the configured mappings for an index you may use the `getMappings` method on the client.

```js
var mappings = getInstance( "Client@cbelasticsearch" ).getMappings( "reviews" );
```

## Triggering an index refresh

On occasion, you may need to ensure the index is updated in real time (immediately and synchronously). This can be done via the `refreshIndex()` client method:

```js
var mappings = getInstance( "Client@cbelasticsearch" ).refreshIndex( "reviews" );
```

You can refresh multiple indices at once:

```js
var mappings = getInstance( "Client@cbelasticsearch" ).refreshIndex( [ "reviews", "books" ] );
// OR
var mappings = getInstance( "Client@cbelasticsearch" ).refreshIndex( "reviews,books" );
```

as well as pass [supported query parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html#refresh-api-query-params) to the refresh endpoint. This can be useful when using wildcards in the index/alias names:

```js
var mappings = getInstance( "Client@cbelasticsearch" ).refreshIndex(
    [ "reviews", "book*" ],
    { "ignore_unavailable" : true }
);
```

## Getting Index Statistics

To retrieve statistics on an index, use the `getIndexStats()` method:

```js
var mappings = getInstance( "Client@cbelasticsearch" ).getIndexStats( "reviews" );
```

You can retrieve particular statistics metrics:

```js
var mappings = getInstance( "Client@cbelasticsearch" )
                    .getIndexStats( "reviews", [ "indexing", "search" ] );
```

Or all metrics:

```js
var mappings = getInstance( "Client@cbelasticsearch" )
                    .getIndexStats( "reviews", [ "_all" ] );
```

You can even retrieve all metrics on all indices by skipping the `indexName` parameter entirely:

```js
var mappings = getInstance( "Client@cbelasticsearch" ).getIndexStats();
```

Finally, you can pass a struct of parameters to fine-tune the statistics result:

```js
var mappings = getInstance( "Client@cbelasticsearch" )
                    .getIndexStats(
                        "reviews",
                        [ "_all" ],
                        { "level" : "shards", "fields" : "title,createdTime" }
                    );
```

## Creating Runtime Fields

Elasticsearch allows [mapping runtime fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-mapping-fields.html), which are fields calculated at search time and returned in the `"fields"` array.

```js
var script = getInstance( "Util@cbelasticsearch" )
.formatToPainless("
  if( doc['summary'].value.contains('love') ){ emit('😍');}
  if( doc['summary'].value.contains('great') ){ emit('🚀');}
  if( doc['summary'].value.contains('hate') ){ emit('😡');}
  if( doc['summary'].value.contains('broke') ){ emit('💔');}
");
getInstance( "Client@cbelasticsearch" )
        .patch( "reviews", {
            "mappings" : {
                "runtime" : {
                    "summarized_emotions" : {
                        "type" : "text",
                        "script" : {
                            "source" : script
                        }
                    }
                }
            }
        } );
```

This `summarized_emotions` field [can then be retrieved during a search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html) to display an array of emotions matching the review summary.

## Opening or Closing an Index

Certain index-level settings can not be applied while the index is open. To solve this, CBElasticsearch offers the `.closeIndex()` and `.openIndex()` methods:

```js
var mappings = getInstance( "Client@CBElasticsearch" ).closeIndex( "reviews" );

// apply settings...

var mappings = getInstance( "Client@CBElasticsearch" ).openIndex( "reviews" );
```

Each of these methods accepts a struct of name/value (simple values only) arguments to pass in the query string:

```js
var mappings = getInstance( "Client@CBElasticsearch" ).closeIndex( "reviews", { "ignore_unavailable" : true } );
```

See [the Elasticsearch "Close Index" documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-close.html) for more information.

## Deleting an Index

All good things must come to an end, eh? You can use `Client.deleteIndex()` to delete an existing index:

```js
getInstance( "Client@cbelasticsearch" ).deleteIndex( "reviews" )
```

Or you can use `IndexBuilder.delete()`:

```js
IndexBuilder.new( "reviews" ).delete();
```

## Additional Reading

{% hint style="warning" %}
*Deprecation notice: Custom index types* [*are deprecated*](https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html) *since Elasticsearch v7.0, and should no longer be used. Only a single type will be accepted in future releases.*
{% endhint %}

* [Elasticsearch Mapping Guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html)
* [Index Settings Reference](https://www.elastic.co/guide/en/elasticsearch/guide/current/_index_settings.html)


# Index Lifecycles

Learn How to to use Index Lifecycle Policies to transition time series data

For time-series data, such as logs or metrics, you may wish to transition or even delete older data after a period of time. Since Elasticsearch v7, ILM policies allow you to configure the transition and retention of your data. The `ILMPolicyBuilder` object helps facilitate the configuration of of ILM for your data streams or time-series indices. For an ILM policy to work, your documents must have a field of `@timestamp`. If your existing indices do not have this field, you can add it [via an `updateByQuery` script](https://github.com/coldbox-modules/cbelasticsearch/blob/main/documents/README.md#update-by-query) using an existing timestamp field.

{% hint style="info" %}
For more information, head to the [ILM Overview in the Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html).
{% endhint %}

## Policy Creation

Let's create a simple policy to delete our data after 30 days:

```js
getInstance( "ILMPolicyBuilder@cbelasticsearch" )
        .new( 
            "my-ilm-policy"
        ).withDeletion(
            age = 30
        ).save();
```

In order to attach the policy to an existing index you can specify the `index.lifecycle.name` in the index settings or in your [Component or Index Templates](/indices/managing-indices/templates);

```js
getInstance( "IndexBuilder@cbelasticsearch" )
                .new( 
                    name="my-index-name", 
                    settings={ 
                        "index.lifecycle.name" : "my-ilm-policy" 
                    } 
                )
                .save();
```

## Phased Rollover and Archival

You may also wish to transition data between phases, and consolidate or shrink your datasets as they age. This can be done by using [lifeycle phases](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html).

Let's create a more complex index lifecycle:

```js
getInstance( "ILMPolicyBuilder@cbelasticsearch" )
        .new( "my-advanced-ilm-policy" )
        .hotPhase(
            // the number of shards to use in the initial phase ( overrides any index template settings )
            shards = 3
            // Forces a rollover of the oldest data, regardless of age, if the size of any shard is greater than 10GB
            rollover = 10
        ).warmPhase(
            // transition to this phase at 30 days old
            age = "30d",
            // shrink to 2 shards from 3 in the hot phase
            shards = 2,
            // Set to no replicas in this phase
            allocate = 0,
            // downsample our time series data to 1 hour intervals
            downsample = "1h"
        ).coldPhase(
            // transition to this phase at 30 days old
            age = "60d",
            // shrink to 2 shards from 3 in the hot phase
            shards = 1,
            // Set to no replicas in this phase
            allocate = 0,
            // downsample our time series data to 2 hour intervals
            downsample = "2h"
            // Make our index read only in this phase
            readOnly = true
        ).withDeletion(
            age = "120d"
        ).save();
```

## ILMPolicyBuilder Method Signatures

### `new`

```js
/**
     * Creates a new policy builder instance
     *
     * @policyName string
     * @phases a struct of phases ( optional )
     * @meta optional struct of meta
     */
    ILMPolicyBuilder function new(
        required string policyName,
        struct phases,
        struct meta
    )
```

### `hotPhase`

```js
    /**
     * Sets the configuration for the ILM Hot Phase
     * https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html
     *
     * @config a raw struct containing the phase configuration
     * @priority numeric a priority to set for this index during the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-set-priority.html
     * @rollover any either a raw rollover struct or a numeric (GB)/ string representing the size at which the index should rollover documents to the next phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html 
     * @shards numeric the number of shards to shrink to in the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-shrink.html
     * @searchableSnapshot string the name of a snapshot respository to create during this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html
     * @downsample any whether to downsample the repository. Either a numeric or string may be passed ( e.g. 1(days) or `1d` ) which denotes the fixed interval of the @timestamp to downsample to https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-downsample.html 
     * @forceMerge numeric The number of segments to force merge to during this phase.  This action makes the index read-only https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-forcemerge.html
     * @readOnly boolean  Whether to make the index read-only during the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-readonly.html
     * @unfollow boolean Whether to convert from a follower index ot a regular index at this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-unfollow.html
     */
    ILMPolicyBuilder function hotPhase(
        struct config,
        numeric priority,
        any rollover,
        numeric shards,
        string searchableSnapshot,
        any downsample,
        numeric forceMerge,
        boolean readOnly,
        boolean unfollow
    )
```

### `warmPhase`

```js
    /**
     * Sets the configuration for the ILM Warm Phase
     * https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html
     *
     * @config a raw struct containing the phase configuration
     * @age any Either a numeric of the number of days or a string interval to use as the threshold at which data is transitioned to this tier 
     * @priority numeric a priority to set for this index during the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-set-priority.html
     * @shards numeric the number of shards to shrink to in the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-shrink.html
     * @downsample any whether to downsample the repository. Either a numeric or string may be passed ( e.g. 1(days) or `1d` ) which denotes the fixed interval of the @timestamp to downsample to https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-downsample.html
     * @allocate any if a numeric is provided it is applied as the number of replicas.  Otherwise a struct config may be provided https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-allocate.html 
     * @migrate boolean moves the data to the phase-configured tier. Defaults to true so only use this argument if disabling migration https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html
     * @forceMerge numeric The number of segments to force merge to during this phase.  This action makes the index read-only https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-forcemerge.html
     * @readOnly boolean  Whether to make the index read-only during the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-readonly.html
     * @unfollow boolean Whether to convert from a follower index ot a regular index at this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-unfollow.html
     */
    ILMPolicyBuilder function warmPhase(
        struct config,
        any age,
        numeric priority,
        numeric shards,
        any downsample,
        any allocate,
        boolean migrate,
        numeric forceMerge,
        boolean readOnly,
        boolean unfollow
    )
```

### `coldPhase`

```js
    /**
     * Sets the configuration for the ILM Cold Phase
     * https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html
     *
     * @config a raw struct containing the phase configuration
     * @age any Either a numeric of the number of days or a string interval to use as the threshold at which data is transitioned to this tier 
     * @priority numeric a priority to set for this index during the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-set-priority.html
     * @searchableSnapshot string the name of a snapshot respository to create during this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html
     * @downsample any whether to downsample the repository. Either a numeric or string may be passed ( e.g. 1(days) or `1d` ) which denotes the fixed interval of the @timestamp to downsample to https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-downsample.html
     * @allocate any if a numeric is provided it is applied as the number of replicas.  Otherwise a struct config may be provided https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-allocate.html 
     * @migrate boolean moves the data to the phase-configured tier. Defaults to true so only use this argument if disabling migration https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html
     * @readOnly boolean  Whether to make the index read-only during the phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-readonly.html
     * @unfollow boolean Whether to convert from a follower index ot a regular index at this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-unfollow.html
     */
    ILMPolicyBuilder function coldPhase(
        struct config,
        any age,
        numeric priority,
        string searchableSnapshot,
        any downsample,
        any allocate,
        boolean migrate,
        boolean readOnly,
        boolean unfollow
    )

```

### `frozenPhase`

Note that this phase may not be used without either the `searchableSnapshot` or `unfollow` arguments passed

```js
    /**
     * Sets the configuration for the ILM Freeze Phase
     * https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html
     *
     * @config a raw struct containing the phase configuration
     * @age any Either a numeric of the number of days or a string interval to use as the threshold at which data is transitioned to this tier 
     * @searchableSnapshot string the name of a snapshot respository to create during this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html
     * @unfollow boolean Whether to convert from a follower index ot a regular index at this phase https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-unfollow.html
     */
    ILMPolicyBuilder function frozenPhase(
        struct config,
        any age,
        string searchableSnapshot,
        boolean unfollow
    )
```

### `withDeletion`

```js
    /**
     * Sets the configuration for the deletion phase
     * 
     * @config a raw struct containing the phase configuration
     * @age any Either a numeric of the number of days or a string interval to use as the threshold at which data is transitioned to this tier 
     * @waitForSnapshot string the name of the SLM policy to execute that the delete action should wait for https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-wait-for-snapshot.html
     * @deleteSnapshot boolean Whether to delete the snapshot created in the previous phase
     * 
     */
    ILMPolicyBuilder function withDeletion(
        struct config,
        any age,
        string waitForSnapshot,
        boolean deleteSnapshot
    )
```


# Index Templates

Learn How to Create Index and Component templates to Ensure Data Mappings

Index templates provide a way to ensure your indices are mapped correctly upon creation. You may control both settings and individual field mappings within your documents.

{% hint style="info" %}
Check out Elasticsearch's [Index Templates documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) for further reading.
{% endhint %}

## Component Templates

In order to map your indices, you must first create a [component template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-component-template.html) which includes your mappings and (optionally) settings for the index. You may use the [Mapping Builder](/indices/mapping-builder) to create your mapping DSL or provide it in the form of a DSL struct.

```js
var mappings = getInstance( "MappingBuilder@cbelasticsearch" )
                    .create( function( mapping ){
                        mapping.keyword( "id" );
                        mapping.text( "description" );
                        mapping.date( "@timestamp" );
                        mapping.object( "meta", function( mapping ){
                            mapping.date( "createdTime" ).format( "date_time_no_millis" );
                            mapping.date( "modifiedTime" ).format( "date_time_no_millis" );
                            mapping.text( "changelog" );
                            mapping.keyword( "createdBy" );
                            mapping.keyword( "modifiedBy" );
                        } );
                    } );

getInstance( "Client@cbelasticsearch" ).applyComponentTemplate(
    "my-component-template",
    { 
        "template" :{
            "settings" : {
                    "index.refresh_interval" : "5s",
                    "number_of_replicas"     : 0,
                    "number_of_shards"       : 1,
                    "index.lifecycle.name"   : "my-lifecycle-policy"
            },
            "mappings" : mappings.toDSL()
        }
    }
);
```

Additional methods are available for managing component templates:

* `getInstance( "Client@cbelasticsearch" ).componentTemplateExists( [ template name] )`
* `getInstance( "Client@cbelasticsearch" ).getComponentTemplate( [ template name] )`
* `getInstance( "Client@cbelasticsearch" ).deleteComponentTemplate( [ template name] )`

Note that a component template may not be deleted if it is in use by an Index template. In order to delete it, you must first delete the index template.

## Index Templates

Now that we have created a component template, we can use it in an [Index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html). Index templates define a matching index ( or Data Stream ) name pattern to which any new indexes created with this naming pattern will have the template applied. Multiple component templates may be used in an array.

Note, that if a `data_stream` key is provided in the definition, any writes to a matching index pattern will be created as a data stream.

```js
getInstance( "Client@cbelasticsearch" ).applyIndexTemplate(
    // the index template name
    "my-index-template",
    {
        // The pattern of the inbound index to apply this template. Only applies the template to newly created indices
        "index_patterns" : [ "my-index-*" ],
        "composed_of" : [
            // some other component template
            "other-component-template",
            "my-component-template" 
        ],
        // The presence of this key creates a data stream for any matching index pattern. If it is absent an index will be created when data is received
        "data_stream" : {},
        // A priority - allows you to prioritize the order in which templates are applied with similar patterns
        "priority" : 150,
        // An optional struct of arbitrary meta information for the template
        "_meta" : {
            "description" : "My global index template"
        }
    }
);
```

All Component and Index template updates are applied as *upsert* operations - meaning that the `@version` key will be incremented if a new version of a template is applied.

Additional methods are available for managing idnex templates:

* `getInstance( "Client@cbelasticsearch" ).indexTemplateExists( [ template name] )`
* `getInstance( "Client@cbelasticsearch" ).getIndexTemplate( [ template name] )`
* `getInstance( "Client@cbelasticsearch" ).deleteIndexTemplate( [ template name] )`

Note that an index template may not be deleted if it is in use by a data stream. In order to delete it, you must first delete any data streams which use the template or modify the template used in that data stream.


# Mapping Builder

Learn how to define an Elasticsearch index mapping using the fluent MappingBuilder syntax in CBElasticsearch

Introduced in `v1.0.0`, the MappingBuilder model provides a fluent closure-based sytax for defining and mapping indexes. This builder can be accessed by injecting it into your components:

```js
component {
    property name="builder" inject="MappingBuilder@cbElasticSearch";
}
```

The `new` method of the `IndexBuilder` also accepts a closure as the second (`properties`) argument. If a closure is passed, a `MappingBuilder` instance is passed as an argument to the closure:

```js
indexBuilder.new( "elasticsearch", function( builder ) {
    return {
        "_doc" = builder.create( function( mapping ) {
            mapping.text( "title" );
            mapping.date( "createdTime" ).format( "date_time_no_millis" );
        } )
    };
} );
```

The `MappingBuilder` has one primary method: `create`. `create` takes a callback with a `MappingBlueprint` object, usually aliased as `mapping`.

## Mapping Blueprint

The `MappingBlueprint` gives a fluent api to defining a mapping. It has methods for all the ElasticSearch mapping types:

```js
builder.create( function( mapping ) {
    mapping.text( "title" );
    mapping.date( "createdTime" ).format( "date_time_no_millis" );
    mapping.object( "user", function( mapping ) {
        mapping.keyword( "gender" );
        mapping.integer( "age" );
        mapping.object( "name", function( mapping ) {
            mapping.text( "first" );
            mapping.text( "last" );
        } );
    } );
} )
```

As seen above, `object` expects a closure which will be provided another `MappingBlueprint`. The results will be set as the `properties` of the `object` call.

## Parameters

Parameters can be chained on to a mapping type. Parameters are set using `onMissingMethod` and will use the method name (as snake case) as the parameter name and the first argument passed as the parameter value.

```js
builder.create( function( mapping ) {
    mapping.text( "title" ).fielddata( true );
    mapping.date( "createdTime" ).format( "date_time_no_millis" );
} )
```

> You can also add parameters using the `addParameter( string name, any value )` or `setParameters( struct map )` methods.

The only exception to the parameters functions is `fields` which expects a closure argument and allows you to create multiple field definitions for a mapping.

```js
builder.create( function( mapping ) {
    mapping.text( "city" ).fields( function( mapping ) {
        mapping.keyword( "raw" );
    } );
} );
```

## Reuse Mapping bits with "Partials"

The Mapping Blueprint also has a way to reuse mappings. Say for instance you have a `user` mapping that gets repeated for managers as well.

The partial method accepts three different kinds of arguments:

1. A closure
2. A component with a `getPartial` method
3. A WireBox mapping to a component with a `getPartial` method

The first approach is a simple way to reuse a mapping partials in the same index. The second two approaches work better for partials that are reused across multiple indices.

```js
var partialFn = function( mapping ) {
    return mapping.object( "user", function( mapping ) {
        mapping.integer( "age" );
        mapping.object( "name", function( mapping ) {
            mapping.text( "first" );
            mapping.text( "last" );
        } );
    } );
};

builder.create( function( mapping ) {
    mapping.partial( "manager", partialFn );
    mapping.partial( definition = partialFn ); // uses the partial's defined name, `user` in this case
} );
```


# Aliases

Learn how to create and manage index aliases with CBElasticsearch

cbElasticSearch offers the `AliasBuilder` for assistance in adding and removing index aliases.

For creating an alias:

```js
getWireBox().getInstance( "AliasBuilder@cbElasticSearch" )
    .add( indexName = "myIndex", aliasName = "newAlias" )
    .save();
```

For removing an alias:

```js
getWireBox().getInstance( "AliasBuilder@cbElasticSearch" )
    .remove( indexName = "otherIndex", aliasName = "randomAlias" )
    .save();
```

For bulk operations, use the cbElasticSearch client's `applyAliases` method. These operations are performed in the same transaction (i.e. atomic), so it's safe to use for switching the alias from one index to another.

```js
var removeAliasAction = getWireBox().getInstance( "AliasBuilder@cbElasticSearch" )
    .remove( indexName = "testIndexName", aliasName = "aliasNameOne" );
var addNewAliasAction = getWireBox().getInstance( "AliasBuilder@cbElasticSearch" )
    .add( indexName = "testIndexName", aliasName = "aliasNameTwo" );

variables.client.applyAliases(
    // a single alias action can also be provided
    aliases = [ removeAliasAction, addNewAliasAction ]
);
```

## Retrieving Aliases

The client's `getAliases` method allows you to retrieve a map containing information on aliases in use in the connected cluster.

```js
var aliasMap = getInstance( "Client@cbelasticsearch" ).getAliases();
```

The corresponding object will have two keys: `aliases` and `unassigned`. The former is a map of aliases with their corresponding index, the latter is an array of indexes which are unassigned to any alias.


# Reindexing

Sometimes indices get messy. Learn how to reindex your data in CBElasticsearch.

On occasion, due to a mapping or settings change, you will need to reindex data from one index (the "source") to another (the "destination"). You can do this by calling the `reindex` method on CBElasticsearch's `Client` component.

```js
getInstance( "Client@cbelasticsearch" )
    .reindex( "oldIndex", "newIndex" );
```

## Asynchronous Reindexing

If you want the work to be done asynchronusly, you can pass `false` to the `waitForCompletion` flag. When this flag is set to false the method will return a [`Task` instance](/tasks), which can be used to follow up on the completion status of the reindex process.

```js
getInstance( "Client@cbelasticsearch" )
    .reindex(
        source = "oldIndex",
        destination = "newIndex",
        waitForCompletion = false
    );
```

## Additional Reindex Options

If you have more settings or constraints for the reindex action, you can pass a struct containing valid options to `source` and `destination`.

```js
getInstance( "Client@cbelasticsearch" )
    .reindex(
        source = {
            "index": "oldIndex",
            "type": "testdocs",
            "query": {
                "term": {
                    "active": true
                }
            }
        },
        destination = "newIndex"
    );
```

## Transforming Documents via a Reindex Script

You may also pass a script in to the `reindex` method to transform objects as they are being transferred from one index to another:

```js
getInstance( "Client@cbelasticsearch" )
    .reindex(
        source = {
            "index": "oldIndex",
            "type": "testdocs",
            "query": {
                "term": {
                    "active": true
                }
            }
        },
        destination = "newIndex",
        script = {
            "lang" : "painless",
            "source" : "if( ctx._source.foo != null && ctx._source.foo == 'baz' ){ ctx._source.foo = 'bar'; }"
        }
    );
```

Note that a Painless script containing newlines, tabs, or space indentation will throw a parsing error. To work around this limitation, use CBElasticsearch's `Util.formatToPainless( string script )` method to remove newlines and indentation:

```js
getInstance( "Client@cbelasticsearch" )
    .reindex(
        // ...
        script = {
            "lang" : "painless",
            "source" : getInstance( "Util@cbelasticsearch" )
                        .formatToPainless( getReindexScript() )
        }
    );
```

## Handling Reindex Errors

If you `waitForCompletion` and the reindex action fails, a `cbElasticsearch.HyperClient.ReindexFailedException` will be thrown. You can disable the exception by passing `false` to the `throwOnError` parameter:

```js
getInstance( "Client@cbelasticsearch" )
    .reindex(
        source = "oldIndex",
        destination = "newIndex",
        waitForCompletion = false,
        throwOnError = false
    );
```

{% hint style="info" %}
As always, check out the Elasticsearch documentation for more specifics on [what reindexing is and how it works](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).
{% endhint %}


# Data Streams

Learn How to use Data Streams for time series data

Since v7 Elasticsearch offers the ability to use [Data streams](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/data-streams.html) for time series or rotational data. A data stream uses an [Index Template](/indices/managing-indices/templates) to automatically create backing indices for the data. Depending on the [lifecycle configuration](/indices/managing-indices/index-lifecycles), data may be rotated out to separate indices, snapshots or deleted altogether. Because the creation of a data stream requires a number of dependencies to be created, the process of implementing is a bit more complex. The steps, in order would be:

1. Create a lifecycle policy for your data stream
2. Create one or more [component templates](/indices/managing-indices/templates) for your data stream index mappings/settings. You may also use pre-configured system templates, like log settings, when applicable.
3. Create an [index template](/indices/managing-indices/templates) using your component template(s)
4. Create your data stream - either manually, or by pushing data

Let's take a look at what this might look like for a new logging data stream. For a more detailed example, you can take a look at the [built in Log appender](/logging) for this module.

## Create a lifecycle policy

We will create a policy to:

1. Start with 2 shards in the "hot" phase ( new data ) and force rollover of old data if any shard grows to more than 1GB
2. Transition to the "warm" phase at 7 days, shrink to 1 shard and make the backing index for the phase read-only.
3. Delete the data after 60 days

```js
var policyBuilder = getInstance( "ILMPolicyBuilder@cbelasticsearch" )
                        .new( "my-new-policy" )
                        .hotPhase( shards = 2, rollover = "1gb" )
                        .warmPhase( age=7, shards = 1, readOnly = true )
                        .withDeletion( age = 60 )
                        .save();
```

## Create a component template

Next we'll create a component template to

1. handle custom fields and settings in our logs
2. assign future index templates which use it to the lifecycle policy we created

```js
var mappings = getInstance( "MappingBuilder@cbelasticsearch" )
                    .create( function( mapping ){
                        mapping.date( "@timestamp" );
                        mapping.object( "event", function( mapping ){
                            mapping.text( "message" );
                            mapping.keyword( "application" );
                            mapping.keyword( "version" );
                        } );
                    } );

getInstance( "Client@cbelasticsearch" ).applyComponentTemplate(
    "my-component-template",
    { 
        "template" :{
            "settings" : {
                    "index.lifecycle.name"   : "my-new-policy"
            },
            "mappings" : mappings.toDSL()
        }
    }
);
```

## Create an index template

Now we'll create an index template to use our component template in addition to Elasticsearch's built-in logging templates:

```js
getInstance( "Client@cbelasticsearch" ).applyIndexTemplate(
    // the index template name
    "my-index-template",
    {
        // The pattern of the inbound index to apply this template. Only applies the template to newly created indices
        "index_patterns" : [ "my-index-*" ],
        "composed_of" : [
            // built-in templates
            "logs-mappings",
            "data-streams-mappings",
            "logs-settings", 
            // custom template
            "my-component-template" 
        ],
        // The presence of this key creates a data stream for any matching index pattern.
        "data_stream" : {},
        // A priority - allows you to prioritize the order in which templates are applied with similar patterns
        "priority" : 150,
        // An optional struct of arbitrary meta information for the template
        "_meta" : {
            "description" : "My data stream index template"
        }
    }
);
```

### Create our data stream

Now we can create our data stream in one of two ways. We can either send data to an index matching the pattern or we can create it manually. Let's do both:

Create a data stream manually without data:

```js
getInstance( "Client@cbelasticsearch" ).ensureDataStream( "my-index-foo" );
```

This will create the data stream and backing indices for the data stream named `my-index-foo`.

Create a data stream by adding data:

```js
getInstance( "Document@cblasticsearch" )
        .new( "my-index-bar" )
        .populate(
            {
                "@timestamp" : now(),
                "event" : {
                    "message" : "This is a new event!",
                    "application" : "MyApplicationName",
                    "version" : 1.0.0
                }
            }
        ).create();
```

This will create the data stream and backing indices for the data stream named `my-index-bar`.

Data streams can be a powerful way to ensure that time series data remains relevant and purges itself when there is no longer a need for it. You can also use data streams for a mapping change between versions of your application indices.


# Searching


# Search

Learn how to search documents with CBElasticsearch

The `SearchBuilder` object offers an expressive syntax for crafting detailed searches with ranked results. To perform a simple search for matching documents documents, using Elasticsearch's automatic scoring, we would use the `SearchBuilder` like so:

```js
var searchResults = getInstance( "SearchBuilder@cbelasticsearch" )
    .new( index="bookshop", type="books" )
    .match( "name", "Elasticsearch" )
    .execute();
```

By default this search will return an array of `Document` objects ( or an empty array if no results are found ), with a descending match score as the sort.

To output the results of our search, we would use a loop, accessing the `Document` methods:

```js
for( var resultDocument in searchResults.getHits() ){
    var resultScore     = resultDocument.getScore();
    var documentMemento = resultDocument.getMemento();
    var bookName        = documentMemento.name;
    var bookDescription = documentMemento.description;
}
```

The "memento" is our structural representation of the document. We can also use the built-in method of the Document object:

```js
for( var resultDocument in searchResults.getHits() ){
    var resultScore     = resultDocument.getScore();
    var bookName        = resultDocument.getValue( "name" );
    var bookDescription = resultDoument.getValue( "description" );
}
```

## Search matching

### Exact matching

The `term()` method allows a means of specifying an exact match of all documents in the search results. An example use case might be only to search for active documents:

```js
searchBuilder.term( "isActive", 1 );
```

Or a date:

```js
searchBuilder.term( "publishDate", "2017-05-13" );
```

### Boosting individual matches

The `match()` method of the `SearchBuilder` also allows for a `boost` argument. When provided, results which match the term will be ranked higher in the results:

```js
searchBuilder
    .match( "shortDescription", "Elasticsearch" )
    .match( "description", "Elasticsearch" )
    .match(
        name = "name",
        value = "Elasticsearch",
        boost = 0.5
    );
```

In the above example, documents with a `name` field containing "Elasticsearch" would be boosted in score higher than those which only find the value in the short or long description.

## Wildcards

There are times when you want to be able to match a portion of a `keyword`-mapped field in elasticsearch. The `wildcard` method allows you to do this. Let's say I wanted to match any documents with a `name` key containing `Elastic`. I could use the following method to match those documents:

```js
searchBuilder.keyword( "name", "Elastic" );
```

This would match any documents with a `name` keyword field containing `Elasticsearch` or `Elasticache`. It is important to note that wildcard queries are exceptionally slow, compared to `term`/`must`/`should` queries, as they require recursion through the entire index of document values to obtain their matches.

We can also boost matches and make this a conditional to an existing query:

```js
searchBuilder.shouldMatch( "shortDescription", "Elastic", 1 )
             .wildcard( "name", "Elastic", 5, "should" );
```

In the above query we change the `operator` argument for the wildcard query to "should" to ensure that the match becomes an "or" for the short description or the wildcard. In addition, we boost the wildcard results 5 times above the short description matched results.

## Sorting Results

The `sort()` method also allows you to specify custom sort options. To sort by author last name, instead of score, use:

```js
searchBuilder.sort( "author.lastName", "asc" );
// OR 
searchBuilder.sort( "author.lastName ASC" );
```

While our documents would still be scored, the results order would be changed to the specified alphabetical order on the author's last name.

The `sort()` method also accepts a full sort config:

```js
searchBuilder.sort( "post_date", {
    "order" : "asc",
    "format": "strict_date_optional_time_nanos"
} );
```

Calling `.sort()` multiple times will append the sort configurations to allow fine-tuning the sort order:

```js
searchBuilder.sort( "author.lastName", "asc" );
searchBuilder.sort( "author.age", "DESC" );
```

{% hint style="info" %}
For more information on sorting search results, check out [Elasticsearch: Sort search results](https://www.elastic.co/guide/en/elasticsearch/reference/8.1/sort-search-results.html#sort-search-results)
{% endhint %}

### Paging Through Query Results

The `size` and `from` search options allow adjusting the page size and start row, respectively, of the configured search:

```js
searchBuilder.setFrom( 11 );
searchBuilder.setSize( 10 );
```

The number of matched documents will be in the `SearchResult`'s `getHitCount()` value:

```js
var totalRows = result.getHitCount();
```

{% hint style="info" %}
Be sure to read the [Elasticsearch "Paginate Search Results" documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/paginate-search-results.html), as paging too deeply can adversely affect CPU and memory usage.
{% endhint %}

### Script Fields

SearchBuilder also supports [Elasticsearch script fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#script-fields), which allow you to evaluate field values at search time for each document hit:

```js
searchBuilder.addScriptField( "interestCost",{
    "script": {
        "lang": "painless",
        "source": "return doc['price'].size() != 0 ? doc['price'].value * (params.interestRate/100): null",
        "params": { "interestRate": 5.5 }
    }
} );
```

This will result in an `"interestCost"` field in the `fields` property on the `Document` object:

```js
var interest = searchBuilder.execute().getHits().map( (document) => document.getFields()["interestCost"] ); // 5.50
```

### Runtime Fields

Elasticsearch also supports defining runtime fields, which are fields defined in the index mapping but populated at search time via a script. You can [define these in the index mapping](/indices/managing-indices#creating-runtime-fields), or [define them at search time](#define-runtime-fields-at-search-time).

{% hint style="info" %}
See [Managing-Indices](/indices/managing-indices#creating-runtime-fields) for more information on creating runtime fields.
{% endhint %}

Runtime fields can be fetched via the `setFields()` or `addField()` methods, and will appear in the `Document` object's `fields` struct. This example retrieves the `"fuel_mpg"` runtime field as well as the indexed `"make"` and `"model"` fields:

```js
var hits = searchBuilder.new( "itinerary" )
             .setFields( [ "fuel_mpg", "make", "model" ] )
             .execute()
             .getHits();
// OR
var hits = searchBuilder.new( "itinerary" )
             .addField( "fuel_mpg" )
             .addField( "make" )
             .addField( "model" )
             .execute()
             .getHits();
```

Once you have a search response, you can use the `.getFields()` method to retrieve the specified fields from the search document:

```js
for( hit in hits ){
    var result = hit.getFields();
    writeOutput( "This #result.make# #result.model# gets #fuel_mpg#/gallon" );
}
```

To access document `fields` as well as the `_source` properties, use`hit.getDocument( includeFields = true)`:

```js
var result = searchBuilder.execute();
for( hit in result.getHits() ){
    var document = document.getDocument( includeFields = true );
    writeOutput( "This #document.make# #document.model# gets #fuel_mpg#/gallon" );
}
```

### Define Runtime Fields At Search Time

Elasticsearch also allows you to [define runtime fields at search time](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-search-request.html), and unlike [script fields](#script-fields) these runtime fields are available to use in aggregations, search queries, and so forth.

```js
searchBuilder.addRuntimeMapping( "hasPricing", {
	"type" : "boolean",
	"script": {
		"source": "doc.containsKey( 'price' )"
	}
} );
```

Using `.addField()` ensures the field is returned with the document upon query completion:

```js
searchBuilder.addRuntimeMapping( "hasPricing", ... ).addField( "hasPricing" );
```

We can then retrieve the result field via the `getFields()` method:

```js
var documentsWithPricing = searchBuilder.execute()
	.getHits()
	.filter( (document) => document.getFields()["hasPricing"] );
```

or inlined with the document mento using `hit.getDocument( includeFields = true )`.

### Advanced Query DSL

The SearchBuilder also allows full use of the [Elasticsearch query language](https://www.elastic.co/guide/en/elasticsearch/reference/current/_introducing_the_query_language.html), allowing full configuration of your search queries. There are several methods to provide the raw query language to the Search Builder. One is during instantiation.

In the following we are looking for matches of active records with "Elasticsearch" in the `name`, `description`, or `shortDescription` fields. We are also looking for a phrase match of "is awesome" and are boosting the score of the applicable document, if found.

```js
var search = getInstance( "SearchBuilder@cbelasticsearch" )
    .new(
        index = "bookshop",
        type = "books",
        properties = {
            "query" = {
                "term" = {
                    "isActive" = 1
                },
                "match" = {
                    "name" = "Elasticsearch",
                    "description" = "Elasticsearch",
                    "shortDescription" = "Elasticsearch"
                },
                "match_phrase" = {
                    "description" = {
                        "query" = "is awesome",
                        "boost" = 2
                    }
                }
            }
        }
    )
    .execute();
```

After instantion, you can use the `.param()` and `.bodyParam()` methods to set [query parameters](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/search-search.html#search-search-api-query-params) and [body parameters](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/search-search.html#search-search-api-request-body), respectively.

```js
var response = getInstance( "SearchBuilder@cbelasticsearch" )
    .new( "bookshop" )
    .sort( "publishDate DESC" )
    // match everything
    .setQuery( { "match_all": {} } )
    // Query parameter: return the document version with each hit
    .param( "version", true )
    // Body parameter: return a relevance score for each document, despite our custom sort
    .bodyParam( "track_scores", true );
    // Body parameter: filter by minimum relevance score
    .bodyParam( "min_score", 3 )
    // run the search
    .execute();
```

{% hint style="info" %}
For more information on Elasticsearch query DSL, the [Search in Depth Documentation](https://www.elastic.co/guide/en/elasticsearch/guide/current/search-in-depth.html) is an excellent starting point.
{% endhint %}

## Collapsing Results

The `collapseToField` allows you to collapse the results of the search to a specific field. The data return includes the first matched, most relevant, document found with the collapsed field. When field collapsing is specified, an automatic aggregation will be run, which provides a pagination total for the collapsed document counts. When paginating collapsed fields, you will want to use the `SearchResult` method `getCollapsedCount()` as your total record count rather than the usual `getHitCount()` - which returns all documents matched to the query.

Let's say, for example, we want to find the most recent version of a book in our index, for all books matching the phrase "Elasticsearch". In this case, we can group on the `title` field ( or, in this case `title.keyword`, which is a dynamic keyword-typed field in our index ) to retrieve the most recent version of the book.

```js
var searchResults = getInstance( "SearchBuilder@cbelasticsearch" )
                                .new( index="bookshop" )
                                .mustMatch( "description", "Elasticsearch" )
                                .collapseToField( "title.keyword" )
                                .sort( "publishDate DESC" )
                                .execute()
```

There is also an option to include the number of ocurrences of each collapsed field in the results. When the argument `includeOccurrences=true` is passed to `collapseToField` you can retrieve a map of all collapsed key values and their corresponding document count by calling `searchResult.getCollapsedOccurrences()`.

For more information on field collapsing, see the [Collapse Search Results Documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.html).

### Get Collapsed Ocurrences

`collapseToField()` also supports an `includeOccurrences` option. By passing `includeOccurrences=true` to `collapseToField`, you can retrieve a map of all collapsed key values and their corresponding document count by calling `searchResult.getCollapsedOccurrences()`:

```js
var elasticsearchBookTitles = getInstance( "SearchBuilder@cbelasticsearch" )
                                .new( index="bookshop" )
                                .mustMatch( "description", "Elasticsearch" )
                                .collapseToField( field = "title.keyword", includeOccurrences = {} )
                                .sort( "publishDate DESC" )
                                .execute()
                                .getCollapsedOccurrences();
```

For more information on field collapsing, see the \[Collapse Search Results Documentation]\(<https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.html>).

## Counting Documents

Sometimes you only need a count of matching documents, rather than the results of the query. When this is the case, you can call the `count()` method from the search builder ( or using the client ) to only return the number of matched documents and omit the result set and metadata:

```js
var docCount = getInstance( "SearchBuilder@cbelasticsearch" )
    .new(
        index = "bookshop",
        type = "books",
        properties = {
            "query" = {
                "term" = {
                    "isActive" = 1
                },
                "match" = {
                    "name" = "Elasticsearch",
                    "description" = "Elasticsearch",
                    "shortDescription" = "Elasticsearch"
                },
                "match_phrase" = {
                    "description" = {
                        "query" = "is awesome",
                        "boost" = 2
                    }
                }
            }
        }
    )
    .count();
```

## Highlights

ElasticSearch has the ability to highlight the portion of a document that matched. This is useful for showing context on why certain search results were returned. You can add an ElasticSearch highlight struct to your `SearchBuilder` using the `highlight` method. The struct should take the shape outlined on the [ElasticSearch website](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/search-request-body.html#request-body-search-highlighting).

```js
SearchBuilder.highlight( {
    "fields" : {
        "body" : {}
    }
})
```

## Terms Enum

On occasion, you may wish to show a set of terms matching a partial string. This is similar to aggregations, only filtered by the provided string and intended for autocompletion.

To retrieve this data, you can use the client's `getTermsEnum()` method:

```js
var terms = getInstance( "HyperClient@cbelasticsearch" )
            .getTermsEnum(
                indexName  = "hotels",
                field = "city",
                match = "alb",
                size = 50,
                caseInsensitive = true
            );
```

For advanced lookups, you can use the second argument to pass a struct of custom options:

```js
var terms = getInstance( "HyperClient@cbelasticsearch" )
            .getTermsEnum( ["cities","towns"], {
                "field" : "name",
                "string" : "west",
                "size" : 50,
                "timeout" : "10s"
            } );
```

## Term Vectors

The ["Term Vectors" Elasticsearch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-termvectors.html) allows you to retrieve information and statistics for terms in a specific document field. This could be useful for finding the most common term in a book description, or retrieving all terms with a minimum word length from the book title.

### Retrieving Term Vectors By Document ID

To retrieve term vectors for a known document ID, pass the index name, id, and an array or list of fields to pull from:

```js
var result = getInstance( "HyperClient@cbelasticsearch" ).getTermVectors(
    "books",
    "book_12345",
    [ "title" ]
);
```

You can fine-tune the request using the `options` argument:

```js
var result = getInstance( "HyperClient@cbelasticsearch" ).getTermVectors(
    indexName = "books",
    id = "book_12345",
    options = {
        "fields" : "title",
        "min_word_length" : 4
    }
);
```

See the [query parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-termvectors.html#docs-termvectors-api-query-params) documentation for more configuration options.

### Retrieving Term Vectors By Payload

If you wish to analyze a payload (not an existing document) you can pass a `"doc"` payload in the `options` argument:

```js
var result = getInstance( "HyperClient@cbelasticsearch" ).getTermVectors(
    indexName = "books",
    fields = [ "title" ],
    options = {
      "doc" : {
        "title" : "The Lord of the Rings: The Fellowship of the Ring"
      }
    }
);
```

### SearchBuilder Term Vector Fetch

The SearchBuilder object also offers a `getTermVectors()` method for convenience:

```js
var result = getInstance( "SearchBuilder@cbelasticsearch" )
                .new( "books" )
                .getTermVectors(
                    myDocument._id,
                    [ "title,author.name" ]
                );
```

## `SearchBuilder` Function Reference

* `new([string index], [string type], [struct properties])` - Populates a new SearchBuilder object.
* `reset()` - Clears the SearchBuilder and resets the DSL
* `deleteAll()` - Deletes all documents matching the currently built search query.
* `execute()` - Executes the built search
* `getDSL()` - Returns a struct containing the assembled Elasticsearch query DSL
* `match(string name, any value, [numeric boost], [struct options], [string matchType='any'])` - Applies a match requirement to the search builder query.
* `multiMatch( array names, any value, [numeric boost], [type="best_fields"])` - Search an array of fields with a given search value.
* `dateMatch( string name, string start, string end, [numeric boost])` - Adds a date range match.
* `mustMatch(string name, any value, [numeric boost])` - `must` query alias for match().
* `mustNotMatch(string name, any value, [numeric boost])` - `must_not` query alias for match().
* `shouldMatch(string name, any value, [numeric boost])` - `should` query alias for match().
* `sort(any sort, [any sortConfig])` - Applies a custom sort to the search query.
* `term(string name, any value, [numeric boost])` - Adds an exact value restriction ( elasticsearch: term ) to the query.
* `aggregation(string name, struct options)` - Adds an aggregation directive to the search parameters.
* `collapseToField( string field, struct options, boolean includeOccurrences = false )` - Collapses the results to the single field and returns only the most relevant/ordered document matched on that field.


# Aggregations

Learn how to summarize or "aggregate" data with cbElasticsearch

In some cases, you aren't interested in searching documents as you are in retrieving specific information stored within each document. It is for such a purpose that Elasticsearch provides the ability to aggregate, or summarize, index data.

## Creating an Aggregation

cbElasticsearch's `SearchBuilder` provides an `aggregation()` method for simple aggregation definitions:

```js
searchBuilder.aggregation( string name, struct options )
```

## Max Aggregation

Here's an example of a [max aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-max-aggregation.html) using SearchBuilder:

```js
searchBuilder.aggregation( "last_updated", { "max": { "field": "meta.timestamp" } } )
```

This aggregation will retrieve the most recent date value stored in `meta.timestamp`.

## Terms Aggregation

Use a terms aggregation to return an array of term "buckets", one per value:

```js
searchBuilder.aggregation( "movie_genres", { "terms": { "field": "genre" } } )
```

## Working with Aggregations

To run the query and retrieve aggregations, call `searchBuilder.execute()` followed by `getAggregations()`. `getAggregations()` will return a key/value struct where the key is your provided aggregation name:

```js
var data = mySearch.aggregation( "last_updated", { "max": { "field": "meta.timestamp" } } )
                    .execute()
                    .getAggregations()[ "last_updated" ];
```

For a simple metrics aggregation, you should be able to use the `value` or `value_as_string` keys of the returned aggregation:

```js
function getLastUpdateTime(){
    var aggregation = getSearchBuilder()
                        .new( "exams" )
                        .aggregation( "last_updated", { "max": { "field": "meta.timestamp" } } )
                        .execute()
                        .getAggregations();
                
    return aggregation[ "last_updated" ][ "value_as_string" ];
}
```

In contrast, bucket aggregations return a `buckets` array which can be used as-is or mapped into a separate result entirely:

```js
function getMovieGenres(){
    var aggregations = getSearchBuilder()
                        .new( "exams" )
                        .aggregation( "genres", { "terms": { "field": "genre" } })
                        .execute()
                        .getAggregations();
                
    return aggregations[ "genres" ].buckets.map( ( term ) => term.key );
}
```

{% hint style="info" %}
For a full break down on aggregations, check out the [ElasticSearch aggregation reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html).
{% endhint %}


# Suggestions

Provide "Did you mean \_\_?" or autocomplete functionality to your app's search form with CBElasticsearch

Suggestors are ElasticSearch's way of providing similar looking terms. They fall into two different use cases: "Did you mean...?" or spell check functionality and autocomplete functionality.

You can access the suggestions on the `SearchResult` component using the `getSuggestions` method.

## Did you mean...?

cbElasticSearch can provide a spell-checked or "Did you mean...?" suggestions using either the `suggestTerm` or `suggestPhrase` methods.

* `suggestTerm(string text, string name, string field = arguments.name, struct options = {})` - Adds a term suggestion to the query.
* `suggestPhrase(string text, string name, string field = arguments.name, struct options = {})` - Adds a phrase suggestion to the query.

Term suggestions suggestors on a single word at a time, while phrase suggestors operate on an entire phrase. Any additional options shown on the website can be passed in as the `options` struct.

Term and phrase suggestors are usually added to an existing query. The results will appear in a `suggest` property on the `SearchResult`.

{% hint style="info" %}
More information on how to optimize term and phrase suggestors can be found in the [ElasticSearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html).
{% endhint %}

## Autocomplete

cbElasticSearch can also provide autocomplete behavior using the `suggestCompletion` method. This adds a `completion` block to the `suggest` query.

* `suggestCompletion(string text, string name, string field = arguments.name, struct options = {})` - Adds a completion suggestion to the query.

Completion suggestors can only operate against a mapping of type `completion`. This can be built either using the struct notation or the `MappingBuilder#completion` method.

Completion suggestors usually operate without a query. It is recommended you also only bring back the `_source` fields that you need. If you do not need any of the `_source` fields, you can `setSource( false )` to not bring back `_source` at all.


# Documents

Learn how to create and manage Elasticsearch document records using CBElasticsearch

Documents are the searchable, serialized objects within your indexes. As noted above, documents may be assigned a type, allowing separation of schema, while still maintaining searchability across all documents in the index. Within an index, each document is referenced by an `_id` value. This `_id` may be set manually ( `document.setId()` ) or, if not provided will be auto-generated when the record is persisted. Note that, if using numeric primary keys for your `_id` value, they will be cast as strings on serialization.

* [Managing Documents](#managing-documents)
  * [Creating Documents](#creating-documents)
  * [Retrieving Documents](#retrieving-documents)
  * [Updating Documents](#updating-documents)
    * [Refreshing the Index At Save Time](#refreshing-the-index-at-save-time)
    * [Updating Individual Document Fields](#updating-individual-document-fields)
  * [Deleting Documents](#deleting-documents)
  * [Bulk Operations](#bulk-operations)
    * [Bulk Saving of Documents](#bulk-saving-of-documents)
    * [Update by Query](#update-by-query)
    * [Bulk Operation Parameters](#bulk-operation-parameters)
    * [Asynchronous Bulk Operations](#asynchronous-bulk-operations)

## Creating Documents

The `Document` model is the primary object for creating and working with Documents. Let's say, again, we were going to create a new document in our index. We would do so, by first creating a `Document` object.

```js
var book = getInstance( "Document@cbelasticsearch" ).new(
    index = "bookshop",
    type = "_doc",
    properties = {
        "title" = "Elasticsearch for Coldbox",
        "summary" = "A great book on using Elasticsearch with the Coldbox framework",
        "description" = "A long descriptio with examples on why this book is great",
        "author" = {
            "id" = 1,
            "firstName" = "Jon",
            "lastName" = "Clausen"
        },
        // date with specific format type
        "publishDate" = dateTimeFormat( now(), "yyyy-mm-dd'T'HH:nn:ssZZ" ),
        "edition" = 1,
        "ISBN" = 123456789054321
    }
);

book.save();
```

In addition to population during the new method, we could also populate the document schema using other methods:

```js
document.populate( myBookStruct )
```

or by individual setters:

```js
document.setValue(
    "author",
    {
        "firstName" = "Jon",
        "lastName" = "Clausen"
    }
);
```

If we want to manually assign the `_id` value, we would need to explicitly call `setId( myCustomId )` to do so, or would need to provide an `_id` key in the struct provided to the `new()` or `populate()` methods.

## Retrieving Documents

To retrieve an existing document, we must first know the `_id` value. We can either retrieve using the `Document` object or by interfacing with the `Client` object directly. In either case, the result returned is a `Document` object, i f found, or null if not found.

Using the `Document` object's accessors:

```js
var existingDocument = getInstance( "Document@cbelasticsearch" )
    .setIndex( "bookshop" )
    .setType( "_doc" )
    .setId( bookId )
    .get();
```

Calling the `get()` method with explicit arguments:

```js
var existingDocument = getInstance( "Document@cbelasticsearch" )
    .get(
        id = bookId,
        index = "bookshop",
        type = "_doc"
    );
```

Calling directly, using the same arguments, from the client:

```js
var existingDocument = getInstance( "Client@cbelasticsearch" )
    .get(
        id = bookId,
        index = "bookshop",
        type = "_doc"
    );
```

The `get` method also accepts a struct of [query parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html#docs-get-api-query-params) to pass to the document retrieval request. For example, we only want certain items returned in the document JSON, we can pass a `_source_includes` query parameter:

```js
var minimal = getInstance( "Client@cbelasticsearch" )
    .get(
        id = bookId,
        index = "bookshop",
        type = "_doc",
        params = {
            "_source_includes" : "_id,title"
        }
    );
```

This will bring back only the identifier and title in the retrieved document. [A list of available query parameters may be found here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html#docs-get-api-query-params).

## Updating Documents

Once we've retrieved an existing document, we can simply update items through the `Document` instance and re-save them.

```js
existingDocument.populate( properties = myUpdatedBookStruct ).save()
```

You can also pass Document objects to the `Client`'s `save()` method:

```js
getInstance( "Client@cbelasticsearch" ).save( existingDocument );
```

### Refreshing the Index At Save Time

If you need your document available immediately (such as during a test or pipeline), you can pass `refresh = true` to [instruct Elasticsearch to refresh the relevant index shard immediately and synchronously](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html):

```js
getInstance( "Client@cbelasticsearch" ).save(
    document = existingDocument,
    refresh = true
);
```

The refresh parameter also accepts a `wait_for` option, which tells Elasticsearch to wait until the next index refresh:

```js
getInstance( "Client@cbelasticsearch" ).save(
    document = existingDocument,
    refresh = "wait_for"
);
```

### Updating Individual Document Fields

The `patch` method of the Client allows a user to update select fields, bypassing the need for a fully retrieved document. This is similar to an `UPDATE foo SET bar = 'xyz' WHERE id = :id` query on a relational database. The method requires an index name, identifier and a struct containing the keys to be updated:

```js
getInstance( "Client@cbelasticsearch" ).patch( 
    "bookshop",
    bookId,
    {
        "title"  : "My Book Title - 1st Edition"
    }
);
```

Nested keys can also be updated using dot-notation:

```js
getInstance( "Client@cbelasticsearch" ).patch(
    "bookshop",
    bookId,
    {
        "author.firstName"  : "Jonathan"
    }
);
```

## Deleting Documents

Deleting documents is similar to the process of saving. The `Document` object may be used to delete a single item.

```js
var document = getInstance( "Document@cbelasticsearch" )
    .get(
        id = documentId,
        index = "bookshop",
        type = books
    );
if( !isNull( document ) ){
    document.delete();
}
```

Documents may also be deleted by passing a `Document` instance to the client:

```js
getInstance( "Client@cbelasticsearch" ).delete( myDocument );
```

Finally, documents may also be deleted by query, using the `SearchBuilder` ( more below ):

```js
getInstance( "SearchBuilder@cbelasticsearch" )
    .new( index="bookshop", type="books" )
    .match( "name", "Elasticsearch for Coldbox" )
    .deleteAll();
```

## Bulk Operations

Elasticsearch allows to you perform [bulk operations](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html), which allows a developer to queue multiple backend operations on the search indices and send them all at once. The `processBulkOperation` method allows you to send a payload of operations in one batch. Note that create, update, and index actions require a `source` key, where as `delete` methods only require an `operation` key. The schema of the `source` key follows the same schema's described in the [Bulk API Documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html):

```js
var ops = [
    {
        "operation" : { "update" :  { "_index" : "bookshop", "_id" : "xyz" } },
        "source" : {
            "doc" : { "title" : "My Book Title - 1st Edition" }
        }
    },
    {
        "operation" : { "delete" : { "_index" : "otherindex", "_id" : "abc" } }
    },
    {
        "operation" : { "update" : { "_index" : "bookshop", "_id" : "efg" } }
        "source" : {
            "doc" : {
                "title" = "Elasticsearch for Coldbox - 2nd Edition",
                "summary" = "A new version of the original great book on using Elasticsearch with the Coldbox framework",
                "description" = "A long description with examples on why this book is great",
                "author" = {
                    "id" = 1,
                    "firstName" = "Jon",
                    "lastName" = "Clausen"
                },
                // date with specific format type
                "publishDate" = dateTimeFormat( now(), "yyyy-mm-dd'T'HH:nn:ssZZ" ),
                "edition" = 1,
                "ISBN" = 123456789054321
            },
            "doc_as_upsert" : true
        }
    }
];

getInstance( "Client@cbelasticsearch" ).processBulkOperation( ops, { "refresh" : true } );
```

### Bulk Saving of Documents

Builk inserts and updates can be peformed by passing an array of `Document` objects to the Client's `saveAll()` method:

```js
var documents = [];

for( var myStruct in myArray ){
    var document = getInstance( "Document@cbelasticsearch" ).new(
        index = myIndex,
        type = myType,
        properties = myStruct
    );

    arrayAppend( documents, document );
}

getInstance( "Client@cbelasticsearch" ).saveAll( documents );
```

### Update by Query

For advanced updates to documents in the index, the `updateByQuery` method can provide a powerful way to make bulk transformations on documents in your index. The `updateByQuery` method requires the passing of a "script" argument, which is a struct containing two strings - the language and the script. Elasticsearch [supports a number of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) however, most of the time, the "painless" language is the easiest choice.

Let's say, for example, that you need to add a new key, with a default value, to every document in your index where the key does not already exist:

```js
var searchBuilder = getInstance( "SearchBuilder@cbelasticsearch" )
                        .mustNotExist( "isInPrint" );
getInstance( "Client@cbelasticsearch" )
            .updateByQuery(
                searchBuilder,
                {
                    "lang" : "painless"
                    "script" : "ctx._source.isInPrint = true"
                }
            );
```

In the above case, we queried for a lack of existence on the `isInPrint` key and created all documents which matched to use a default value of `false`.

Note the variable `ctx._source` used in the script, which is a reference to the document being iterated in the update loop. More information on crafting complex, scripted, query-based updates can be found in [the official elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html).

Note that a Painless script containing newlines, tabs, or space indentation will throw a parsing error. To work around this limitation, use CBElasticsearch's `Util.formatToPainless( string script )` method to remove newlines and indentation:

```js
getInstance( "Client@cbelasticsearch" )
            .updateByQuery(
                searchBuilder,
                {
                    "lang" : "painless",
                    "script" : getInstance( "Util@cbelasticsearch" )
                                .formatToPainless( getReindexScript() )
                }
            );
```

### Bulk Operation Parameters

The search builder also supports the addition of URL parameters, which may be used to transform or modify the behavior of bulk document actions. Comprehensive lists of these parameters may be found at the official Elasticsearch docs:

* [Update by Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#_url_parameters)
* [Delete by Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html)

Of note are the throttling parameters, which are useful in dealing with large documents and/or indices. By default elasticsearch processes batch operations in groups of 1000 documents. Depending on the size of your documents and the collection, it may be preferable to throttle the batch to a smaller number of documents per batch:

```js
getInstance( "SearchBuilder@cbelasticsearch" )
    .new( index="bookshop", type="books" )
    .match( "name", "Elasticsearch for Coldbox" )
    .param( "scroll_size", 100 )
    .deleteAll();
```

### Asynchronous Bulk Operations

Both the `updateByQuery` and `deleteByQuery` methods support a `waitForCompletion` argument. By default, this is set to `true`. When passed as false, however, the method will return a [`Task` instance](/tasks), which can be used to follow up on the completion status of the action process.

{% hint style="info" %}
You may also provide this argument in the SearchBuilder Params ( see "Parameters" above ): `searchBuilder.param( 'wait_for_completion', false )`, in lieu of providing the argument to the action. The end result is the same.
{% endhint %}


# Logging

Learn how to transform incoming data in an Elasticsearch Ingest Pipeline.

## Logging

cbElasticsearch comes pre-packaged with a logging appenders which can be configured in your Coldbox application to capture log messages and store them for later search and retrieval.

The `LogstashAppender` uses a time-series data stream to cycle log data through a configured lifecycle policy. By default data is retained for 365 days. If you wish to provide a different configuration or retention period, you can do so by specifying a custom `lifeCyclePolicy` setting to the appender. [More on Index LifeCycles here](/indices/managing-indices/index-lifecycles).

Appenders may be configured in your Coldbox configuration. Alternately, you can [install the `logstash` module ](https://logstash.ortusbooks.com/getting-started/introduction), which will auto-register appenders for you. Note that the Logstash module already installs with `cbElasticsearch` and will register it so you only need one module or the other.

```js
logBox = {
    // Define Appenders
    appenders = {
        console = {
            class="coldbox.system.logging.appenders.ConsoleAppender"
        },
        logstash = {
            class="cbelasticsearch.models.logging.LogstashAppender",
            // The log level to use for this appender - in this case only errors and above are logged to Elasticsearch
            levelMax = "ERROR",
            // Appender configuration
            properties = {      
                // The pattern used for the data stream configuration.  All new indices with this pattern will be created as data streams        
                "dataStreamPattern" : "logs-coldbox-*",
                // The data stream name to use for this appenders logs
                "dataStream" : "logs-coldbox-logstash-appender",
                // The ILM policy name to create for transitioning/deleting data
                "ILMPolicyName"   : "cbelasticsearch-logs",
                // The name of the component template to use for the index mappings
                "componentTemplateName" : "cbelasticsearch-logs-mappings",
                // The name of the index template whic the data stream will use
                "indexTemplateName" : "cbelasticsearch-logs",
                // Retention of logs in number of days
                "retentionDays"   : 365,
                // an optional lifecycle full policy https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-put-lifecycle.html
                "lifecyclePolicy" : javacast( "null", 0 ),
                // The name of the application which will be transmitted with the log data and used for grouping
                "applicationName" : "My Application name",
                // A release version to use for your logs
                "releaseVersion"  : "1.0.0",
                // The number of shards for the backing data stream indices
                "indexShards"     : 1,
                // The number of replicas for the backing indices
                "indexReplicas"   : 0,
                // The max shard size at which the hot phase will rollover data
                "rolloverSize"    : "10gb",
                // Whether to migrate any indices used in v2 of this module over to data streams - only used if an `index` key ( v2 config ) is provided to the properties
                "migrateIndices"  : false,
                // Whether to allow log events to fail quietly.  When turned on, any errors received when saving log entries will not throw but will be logged out to other appenders
                "throwOnError"    : true
            }
        }

    },
    // Root Logger - appends into messages to all appenders - except those with a specified `levelMax` like above
    root = { levelmax="INFO", appenders="*" }
};
```

For more information on configuring LogBox log appenders for your application, see the [Coldbox documentation](https://coldbox.ortusbooks.com/getting-started/configuration/coldbox.cfc/configuration-directives/logbox)

## Detached Appenders

The logging capbilities of the Elasticsearch module extend beyond the framework LogBox appenders. In a era of big data an analytics, developers also have the ability to create custom appenders appenders for ad-hoc use in storing messages, collecting metrics, or for use in aggregations. Simple messages can be logged or even raw messages which adhere to the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html). By using detached appenders, you can capture custom information for later reference.

### Creating a Detached Appender

To create a detached appender, use the `AppenderService` method `createDetachedAppender( string name, struct properties )`. The properties passed can be any of the above, or you can omit those and the default properties will be used:

```
getInstance( "AppenderService@cbelasticsearch" )
    .createDetachedAppender(
        "myCustomAppender",
        {
                        
            "retentionDays"         : 30,
            "applicationName"       : "Custom Appender Logs",
            "rolloverSize"          : "1gb"
        }
    );
```

Now we can log messages to this appender on an ad-hoc basis by calling the methods in the Appender service.

#### Logging a single message

We can log a traditional single message by using the `logToAppender` method of the Appender service. This is familiar to many Coldbox developers:

```java
getInstance( "AppenderService@cbelasticsearch" )
    .logToAppender(
        "myCustomAppender",
        "This is my custom log message which contains information I need to search later",
        "info",
        {
            // labels are stored as exact match keywords which allow you aggregate and filter the log messages
            // These are promoted to the root labels object
            "labels" : {
                "manager" : "Jim Leyland",
                "team" : "Detroit Tigers"
            },
            // Any other key value pairs become part of the log entry extra info, which is searchable but not filterable
            "person" : {
                "firstName" : "Jim",
                "lastName" : "Leyland",
                "teams" : [
                    "Detroit Tigers",
                    "Pittsburg Pirates",
                    "Colorado Rockies"
                ],
                "hallOfFamer" : true,
                "inductionYear" : 2024
            }
        }
    );
```

#### Logging one or more raw formatted messages

If you are comfortable assembling your own JSON and want to ship those log entries to elasticsearch raw, you can do so by usin the `logRawToAppender` method of the Appender service. This functionality can also allow you to assemble a series of logs and ship them all off in one bulk operation. The `messages` argument to the function may be a single entry struct, or it may be an array of multiple message which adhere to the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html).

```java
getInstance( "AppenderService@cbelasticsearch" )
    .logRawToAppender(
        "myCustomAppender",
        [
            {
                "@timestamp" : now(),
                "log"        : {
                    "level"    : "info",
                    "logger"   : "myCustomLogger",
                    "category" : "CustomEvents"
                },
                "message" : "This is my custom log message which contains information I need to search later",
                "event"   : {
                    "action" : event.getCurrentAction(),
                    "duration" : myProcessingDurationNanos,
                    "created"  : now(),
                    "severity" : 4,
                    "category" : "myCustomLogger",
                    "dataset"  : "cfml",
                    "timezone" : createObject( "java", "java.util.TimeZone" ).getDefault().getId()
                },
                "file" : { "path" : CGI.CF_TEMPLATE_PATH },
                "url"  : {
                    "domain" : CGI.SERVER_NAME,
                    "path"   : CGI.PATH_INFO,
                    "port"   : CGI.SERVER_PORT,
                    "query"  : CGI.QUERY_STRING,
                    "scheme" : lCase( listFirst( CGI.SERVER_PROTOCOL, "/" ) )
                },
                "http"    : {
                    "request" : { "referer" : CGI.HTTP_REFERER },
                },
                "labels" :  {
                    "manager" : "Jim Leyland",
                    "team" : "Detroit Tigers",
                    "hallOfFameYear" : "2024"
                },
                "package" : {
                    "name"    : getProperty( "applicationName" ),
                    "version" : "1.1.0",
                    "type"    : "cfml",
                    "path"    : expandPath( "/" )
                },
                "host"       : { "name" : CGI.HTTP_HOST, "hostname" : CGI.SERVER_NAME },
                "client"     : { "ip" : CGI.REMOTE_ADDR },
                "user"       : {},
                "user_agent" : { "original" : CGI.HTTP_USER_AGENT },
                "error" : {
                    "type"      : "message",
                    "level"     : level,
                    "message"   : loge.getMessage(),
                    "extrainfo" : serializeJSON(
                        {
                            "person" : {
                                "firstName" : "Jim",
                                "lastName" : "Leyland",
                                "teams" : [
                                    "Detroit Tigers",
                                    "Pittsburg Pirates",
                                    "Colorado Rockies"
                                ],
                                "hallOfFamer" : true,
                                "inductionYear" : 2024
                            }
                        }
                    )
                }
            },
            ... and so on ...
        ]
    );
```

See our docs [on search](https://cbelasticsearch.ortusbooks.com/searching/search) and [aggregations](https://cbelasticsearch.ortusbooks.com/searching/aggregations) for more information on how to assemble custom reports and aggregations of your logged data.


# Pipelines

Learn how to transform incoming data in an Elasticsearch Ingest Pipeline.

Elasticsearch allows you to create pipelines which pre-process inbound documents and data. Methods are available to create, read, update and delete pipelines. For more information on defining processors, conditionals and options see the [PUT Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/master/put-pipeline-api.html) and [Processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-processors.html) documentation.

## Creating a Pipeline

Let's say we want to automatically set a field on a document when we save it. We can add a processor on the ingest of documents like so:

```js
var myPipeline = getInstance( "Pipeline@cbelasticsearch" ).new( {
                        "id" : "foo-pipeline",
                        "description" : "A test pipeline",
                        "version" : 1,
                        "processors" : [
                            {
                                "set" : {
                                    "if" : "ctx.foo == null",
                                    "field" : "foo",
                                    "value" : "bar"
                                }
                            }
                        ]
                    } );
```

With this pipeline, if a value of `foo` is not defined ( note that `ctx` is the document reference in the `if` conditional ) in the inbound document, then the value of that field will automatically be set to `'bar'`.

We can save/apply this pipeline in one of two ways.

Through the pipeline object:

```js
myPipeline.save();
```

Or through the client:

```js
getInstance( "Client@cbelasticsearch" ).applyPipeline( myPipeline );
```

Note that if you are using a [secondary cluster](https://github.com/coldbox-modules/cbelasticsearch/blob/main/docs/Configuration.md), you will need to perform your CRUD operations through the client, as the `save` method in the pipeline object will route through the top level client.

## Retrieving Pipeline Definitions

If we know the name of our pipeline, we can retreive the definition from Elasticsearch by using the `getPipeline` method of the client:

```js
getInstance( "Client@cbelasticsearch" ).getPipeline( "foo-pipeline" );
```

If we need to retreive the definitions of all configured pipelines we can call the `getPipelines` method:

```js
getInstance( "Client@cbelasticsearch" ).getPipelines();
```

## Updating a Pipeline

We can modify pipelines using the pipeline object, as well. Let's do this by retrieving the existing pipeline, updating and then saving it:

```js
var pipeline = getInstance( "Pipeline@cbelasticsearch" )
                .new( getInstance( "Client@cbelasticsearch" )
                .getPipeline( "foo-pipeline" ) );

pipeline.addProcessor(
    {
        "set" : {
            "if" : "ctx.foo == 'baz'",
            "field" : "foo",
            "value" : "bar"
        }
    }
).save();
```

Now we've added a processor stating that if our `foo` value is `baz`, set it to `bar`. Newly saved/ingested documents using this pipeline will never have a value of `baz` for the `foo` key.

## Deleting a Pipeline

We can delete a pipeline by using the identifier, or by passing the wildcard `*`, which delete all configured ingest pipelines on the server.

```js
getInstance( "Client@cbElastisearch" )
	.deletePipeline( "foo-pipeline" );
```

## Using Pipelines When Saving Documents

Pipelines may be used when saving individual or multiple documents. See the [Documents](/documents) section for more information on document creation.

To save an individual document, with pipeline processing:

```js
myDocument.setPipeline( 'foo-pipeline' ).save();
```

For multiple documents, the pipeline may be set in the document, prior to the `saveAll` call. Note, however, that all documents provided in the bulk save must share the same pipeline, as elasticsearch does not support multiple pipelines in bulk saves. Attempting to save multiple documents with different pipelines will throw an error. Alternately, you may pass the pipeline in as a param to the `saveAll` call:

```js
getInstance( "Client@cbelasticsearch" )
	.saveAll( documents=myDocuments, params={ "pipeline" : "foo-pipeline" } );
```


# Tasks

Learn how to work with asynchronous Elasticsearch operations to process documents in a non-blocking manner

When performing bulk operations - [reindexing](https://github.com/coldbox-modules/cbelasticsearch/blob/main/docs/Indexes.md), [query-based updating or deletions](/documents) - a parameter may be provided which allows the job to run in a non-blocking manner. The method that Elasticsearch uses to monitor the completion of these jobs is called a [task](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html). In the `reindex`, `updateByQuery`, and `deleteByQuery` methods of the client, the argument `waitForCompletion` may be passed. When set to false a `Task` object can be returned which will provide the status of the task and allow you to refresh through completion.

An example, using the reindex method and flushing the status output to the browser, might look something like:

```js
var oldIndex = "books_v1";
var newIndex = "books_v2";
var reindexTask = getInstance( "Client@cbelasticsearch" )
                        .reindex(
                            source = oldIndex,
                            destination = newIndex,
                            waitForCompletion = false,
                            params = {
                                // throttle our reindexing tasks to ensure we don't kill the ES server on a big index - https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-throttle
                                "requests_per_second" : 50,
                                // slice the reindex to concurrent 5 batches at a time - https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-automatic-slice
                                "slices" : 5
                            }
                        );

while( !reindexTask.isComplete() ){
    var status = reindexTask.getStatus();
    writeOutput( "Waiting for task to complete. #status.created# documents of #status.total# documents have been migrated to #newIndex#" );
    flush();
}
```

The `isComplete` method also accepts an argument of `delay` to slow down the rate at which it re-checks the completion of the task. Using the above example, we could check only every 5 seconds by passing 5000 milliseconds as the `delay` argument:

```js
while( !reindexTask.isComplete( delay=5000 ) ){
    var status = reindexTask.getStatus();
    writeOutput( "<p>Waiting for task to complete. #status.created# documents of #status.total# documents have been migrated to #newIndex#.</p>" );
    flush();
}
```

At times a reindex process may be marked as complete, but the documents were not transfered. In CBElasticsearch v2.2.5+, you can use `reindexTask.getError()` to check for errors running the reindex script.

```
if ( !isNull( task.getError() ) ){
  throw(
    message = "Error running task",
    extendedInfo = serializeJSON( error )
  );
}
```

For older CBElasticsearch versions, inspect the document counts to determine if a reindex or update script failed:

```
if ( getStatus().total != getStatus().created ){
  throw(
    message = "Failed to create #getStatus().total-getStatus().created# documents",
    extendedInfo = serializeJSON( getStatus() )
  );
}
```

{% hint style="info" %}
`getResponse()` will contain the error details when a script fails on specific documents. For more broad syntax or script-level errors, the error may only be contained in the newer `getError()` payload in newer versions of CBElasticsearch.
{% endhint %}

When managing large indexes, a task-based approach to bulk operations can allow you to optimize the resource usage of both your Elasticsearch and CFML Application servers.


# Contributing

We're open source! Get started hacking on CBElasticsearch to add a new feature, fix the docs, or prove a regression.

Follow these steps to get started hacking on CBElasticsearch:

1. Clone the module - `git clone git@github.com:coldbox-modules/cbox-elasticsearch.git`
2. Install dependencies - `box install`
3. Start a [new Elasticsearch instance](#Running-Elasticsearch)
4. Start the cbelasticsearch server - `box start`
5. Run tests - `box testbox run`

## Running Elasticsearch

To run the test suite you need a running instance of ElasticSearch. We have provided a `docker-compose.yml` file in the root of the repo to make this easy as possible. Run `docker-compose up --build` ( omit the `--build` after the first startup ) in the root of the project and open `http://localhost:8080/tests/runner.cfm` to run the tests.

If you would prefer to set up Elasticsearch yourself, make sure you start this app with the correct environment variables set:

```ini
ELASTICSEARCH_PROTOCOL=http
ELASTICSEARCH_HOST=127.0.0.1
ELASTICSEARCH_PORT=9200
```

## Releases

To issue a new release:

1. Update (and commit) `changelog.md` with each addition, bugfix, or security issue.
   1. These should be placed under the version number heading: `## [x.y.z] - dd-mm-yyyy`. Later this will be automated to use the `## Unreleased` section.
2. Set and commit the new version number in `box.json`, following semantic versioning format.
3. Run the release script: `box recipe build/release.boxr`