Logsene, our hosted ELK stack now part of Sematext Cloud, exposes the Elasticsearch API. As a Logsene user you may or may not be aware of this.  If you have experience with Elasticsearch though, if you’re aware of its rich ecosystem of tools, you know how valuable this is.  Besides using the Logsene UI, Kibana, or Grafana to search your logs or build charts and dashboards, you can also query all your data via Logsene’s Elasticsearch API.  You can search your logs, filter them, run aggregation queries, use scan and scroll to retrieve large ranges of your data, etc.

Index Templates

When you send your logs to a Logsene app their structure (their mapping in Elasticsearch parlance) is automatically detected.  Log event fields are created and their types are deduced from the received data.  The mapping and the index template are also created.  Under the hood Logsene maintains a sequence of indices for each Logsene app.  Every so often a new index is added and its mapping is based on the index template for that Logsene app.  If you are familiar with how Elasticsearch works you may have already created custom Elasticsearch index templates for your Logsene apps.  This is useful when you want to change the structure of your index, change the type of a field, and so on.  Until recently, after changing your Logsene app’s index template you’d have to wait until next index rotation in your Logsene app for this change to apply.  You couldn’t force index rotation to happen right there and then.  Bummer!

We’re happy to report that this annoying behaviour is now history!  No more waiting!  From now, when you push a new template to Logsene it will be applied immediately. Also, all templates are automatically verified, in real-time, in addition to verification that Elasticsearch does. Logsene checks all field definitions and, if any issues are found, reports them in the response. Valid templates are applied immediately, immediately visible, and ready for new data!  We hope you like this!

How do I Create Custom Index Templates?

This is easy.  We’ll borrow from the Custom Elasticsearch Index Templates in Logsene:

curl -XPUT 'logsene-receiver.sematext.com/_template/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate' -d '{
 "template" : "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*",
 "order" : 21,
 "mappings" : {
  "message" : {
   "properties" : {
    "message" : { "type" : "string" },
    "tags" : { "type" : "string", "analyzer" : "my_own_lowercase" },
    "nick" : { "type" : "string", "analyzer" : "my_own_lowercase" }
   }
  }
 }
}'

Please see Custom Elasticsearch Index Templates in Logsene for a few more important details.

What About The Mappings?

Logsene will let you set your index mappings, too.  Note that you will need to know the name of the underlying index in order to change the mapping.  You can get the index name by searching your Logsene app via the API and looking for index names in the response.  That said, we suggest you simply set the template instead of the mapping.  Setting the template provides you a superset of functionality.  It doesn’t change the mapping of the current index, but instead creates the mapping/template for your future indices and immediately creates a new index for you.

What’s Next?

In the next Logsene release you will be able to modify your mapping via a UI wizard that will make index mapping changes as easy as click-click-click. You’ll be able to add or remove fields, change their types or properties, etc.  The wizard will also warn you about any incompatible changes you try to make and let you optionally reindex (or delete) historical data.  More on that soon!