Blog Feed Post

Introducing Instant Templates & Mappings

Logsene, our hosted ELK stack now part of Sematext Cloud, exposes the Elasticsearch API. As a Logsene user you may or may not be aware of this.  If you have experience with Elasticsearch though, if you’re aware of its rich ecosystem of tools, you know how valuable this is.  Besides using the Logsene UI, Kibana, or Grafana to search your logs or build charts and dashboards, you can also query all your data via Logsene’s Elasticsearch API.  You can search your logs, filter them, run aggregation queries, use scan and scroll to retrieve large ranges of your data, etc.

Index Templates

When you send your logs to a Logsene app their structure (their mapping in Elasticsearch parlance) is automatically detected.  Log event fields are created and their types are deduced from the received data.  The mapping and the index template are also created.  Under the hood Logsene maintains a sequence of indices for each Logsene app.  Every so often a new index is added and its mapping is based on the index template for that Logsene app.  If you are familiar with how Elasticsearch works you may have already created custom Elasticsearch index templates for your Logsene apps.  This is useful when you want to change the structure of your index, change the type of a field, and so on.  Until recently, after changing your Logsene app’s index template you’d have to wait until next index rotation in your Logsene app for this change to apply.  You couldn’t force index rotation to happen right there and then.  Bummer!

We’re happy to report that this annoying behaviour is now history!  No more waiting!  From now, when you push a new template to Logsene it will be applied immediately. Also, all templates are automatically verified, in real-time, in addition to verification that Elasticsearch does. Logsene checks all field definitions and, if any issues are found, reports them in the response. Valid templates are applied immediately, immediately visible, and ready for new data!  We hope you like this!

How do I Create Custom Index Templates?

This is easy.  We’ll borrow from the Custom Elasticsearch Index Templates in Logsene:

curl -XPUT 'logsene-receiver.sematext.com/_template/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate' -d '{
 "template" : "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*",
 "order" : 21,
 "mappings" : {
  "message" : {
   "properties" : {
    "message" : { "type" : "string" },
    "tags" : { "type" : "string", "analyzer" : "my_own_lowercase" },
    "nick" : { "type" : "string", "analyzer" : "my_own_lowercase" }

Please see Custom Elasticsearch Index Templates in Logsene for a few more important details.

What About The Mappings?

Logsene will let you set your index mappings, too.  Note that you will need to know the name of the underlying index in order to change the mapping.  You can get the index name by searching your Logsene app via the API and looking for index names in the response.  That said, we suggest you simply set the template instead of the mapping.  Setting the template provides you a superset of functionality.  It doesn’t change the mapping of the current index, but instead creates the mapping/template for your future indices and immediately creates a new index for you.

What’s Next?

In the next Logsene release you will be able to modify your mapping via a UI wizard that will make index mapping changes as easy as click-click-click. You’ll be able to add or remove fields, change their types or properties, etc.  The wizard will also warn you about any incompatible changes you try to make and let you optionally reindex (or delete) historical data.  More on that soon!

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
"This week we're really focusing on scalability, asset preservation and how do you back up to the cloud and in the cloud with object storage, which is really a new way of attacking dealing with your file, your blocked data, where you put it and how you access it," stated Jeff Greenwald, Senior Director of Market Development at HGST, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Rodrigo Coutinho is part of OutSystems' founders' team and currently the Head of Product Design. He provides a cross-functional role where he supports Product Management in defining the positioning and direction of the Agile Platform, while at the same time promoting model-based development and new techniques to deliver applications in the cloud.
In a world where the internet rules all, where 94% of business buyers conduct online research, and where e-commerce sales are poised to fall between $427 billion and $443 billion by the end of this year, we think it's safe to say that your website is a vital part of your business strategy. Whether you're a B2B company, a local business, or an e-commerce site, digital presence is key to maintain in your drive towards success. Digital Performance will take priority in 2018 for the following reason...
All organizations that did not originate this moment have a pre-existing culture as well as legacy technology and processes that can be more or less amenable to DevOps implementation. That organizational culture is influenced by the personalities and management styles of Executive Management, the wider culture in which the organization is situated, and the personalities of key team members at all levels of the organization. This culture and entrenched interests usually throw a wrench in the work...
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
In his session at Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to maximize project result...
"Software-defined storage is a big problem in this industry because so many people have different definitions as they see fit to use it," stated Peter McCallum, VP of Datacenter Solutions at FalconStor Software, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
"Our strategy is to focus on the hyperscale providers - AWS, Azure, and Google. Over the last year we saw that a lot of developers need to learn how to do their job in the cloud and we see this DevOps movement that we are catering to with our content," stated Alessandro Fasan, Head of Global Sales at Cloud Academy, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Personalization has long been the holy grail of marketing. Simply stated, communicate the most relevant offer to the right person and you will increase sales. To achieve this, you must understand the individual. Consequently, digital marketers developed many ways to gather and leverage customer information to deliver targeted experiences. In his session at @ThingsExpo, Lou Casal, Founder and Principal Consultant at Practicala, discussed how the Internet of Things (IoT) has accelerated our abilit...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
As organizations shift towards IT-as-a-service models, the need for managing and protecting data residing across physical, virtual, and now cloud environments grows with it. Commvault can ensure protection, access and E-Discovery of your data – whether in a private cloud, a Service Provider delivered public cloud, or a hybrid cloud environment – across the heterogeneous enterprise. In his general session at 18th Cloud Expo, Randy De Meno, Chief Technologist - Windows Products and Microsoft Part...
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with extensive global expertise as a strategist, technologist, innovator, marketer, and communicator. For over 30 years across five continents, he has built success with Fortune 500 corporations, vendors, governments, and as a leading research analyst and consultant.