Blog Feed Post

2016 Year in Review: Monitoring and Logging Highlights

2017 is almost here and, like last year, we thought we’d share how 2016 went for us.  We remain committed to be your “one-stop shop” for all things Elasticsearch and Solr: from Consulting, Production Support, and Training, to complementing that with our Logsene for all your logs, and SPM for all your monitoring needs.


It’s safe to say 2016 was the year of Docker and by extension Kubernetes, Mesos, Docker Swarm, among others, too.  They stopped being just early adopters’ toys and have become production-ready technologies used by many. This year we’ve added excellent support for Docker monitoring with SPM and logging with Logsene via the open-source Sematext Docker Agent.

But, let’s walk through 2016 highlights in the chronological order, month by month.


We started the year fresh and published a lot of great content on our blog: from a bunch of useful articles about handling log management (such as: How to forward CloudTrail (or other logs from AWS S3) to Logsene, announcing PagerDuty and Logsene integration or using Filebeat to send Elasticsearch logs to Logsene) to hot topics about monitoring key metrics (such as: introducing NetMaps, a new feature in SPM or presenting the Sematext Docker Agent as a turnkey-solution for Docker Logs, Metrics and Events). If you, like many of our readers, run Elasticsearch and are considering running it inside Docker, have a look at how to run high performance and fault tolerant Elasticsearch Clusters on Docker.


We continued our journey through the logging field by describing how to send your Windows Event Logs to Logsene using NxLog and Logstash. In February we introduced Logagent – a log parser and shipper with log patterns for a number of popular log formats – from various Docker images including Nginx, Apache, Linux and Mac system logs, to Elasticsearch, Redis, Solr, MongoDB and more. Logagent detects the log format automatically using the built-in pattern definitions (also lets you provide your own, custom patterns). You may read more here. Also, we thought about our team needs too, as Sematext team is highly distributed and we prepared a insightful blog post showing how to index your Slack messages in Logsene for the purpose of archiving, searching, and analytics.


Spring was good to us. We announced the launch of on-demand Java Profiling that lets you profile any app running on top of JVM, even in production. If you are using AWS like us and want to collect all your application logs, you may want to start sending AWS CloudWatch Logs (e.g. VPC logs) to Logsene. As Elasticsearch is our expertise, March ended with two related and popular topics: reindexing data with Elasticsearch and documents update by query with Elasticsearch.


Docker, Solr and Elasticsearch were our focus in April. We went to London to give several public Elasticsearch and Solr classes: Core Solr Training, Elasticsearch for Developers; Elasticsearch for Logging and Elasticsearch Operations. While not in London we found time to write about Docker Cloud: Monitoring & Logging, Automatic Geo-IP Enrichment for Docker Logs and Monitoring Kafka on Docker Cloud (just to name a few).


We were recognized by Docker as the Ecosystem Technology Partner (ETP) for monitoring and logging. This designation indicates that our products: SPM Performance Monitoring and Logsene have demonstrated working integration with the Docker platform via the Docker API and are available to users and organizations that seek solutions to monitor their Dockerized distributed applications. Just take a look at Sematext Docker Agent, which is extremely easy to deploy with Kubernetes, Docker Swarm, Docker Cloud and Docker Datacenter.

We covered a few more interesting Solr and Elasticsearch topics, such as: Solr streaming expressions for collection auto-updating, DocValues reindexing with Solr streaming expressions, or scalable and flexible Elasticsearch reindexing via rsyslog.


The middle of the year found us in New York, holding public Elasticsearch / Elastic Stack Training and Apache Solr Training. While some of our colleagues were in New York, others were giving talks about running Elasticsearch in Docker at Berlin Buzzwords. Indeed, monitoring of Docker environments is challenging enough, so we shared our Docker DevOps experience in Top Docker Metrics to Watch.


We created a handy mindmap showing a number of  open source Docker Monitoring & Logging tools and we published a reference guide for Monitoring Docker Datacenter Logs & Metrics.

As the ecosystem of DevOps tools and resources – for monitoring, for logging, for alerting, for continuous integration and deployment, configuration management, etc. –  is growing, finding answers as quickly and efficiently as possible becomes extremely important. We are DevOps like many of our followers and often need to research specific DevOps tools. To help with that we launched search-devops.com, which aggregates, indexes and makes searchable all content repositories (mailing lists, source code, wikis, issue trackers, etc.) for a number of open source DevOps projects.


August was about logging all the way. If you are interested in Elastic Stack Import-Export with Logstash & Logsene or RancherOS Monitoring and Logging Support these are the how-to articles to read.


When it comes to centralizing logs to Elasticsearch the first log shipper that comes to mind is Logstash. However, Logstash is far from being the only game in town. In 5 Logstash Alternatives we covered other log shippers you may want to consider. We couldn’t let September pass without covering Docker, of course.  Here’s how to set up full Docker Swarm Cluster Monitoring & Logging with 1 Command.


We participated in an exciting community-organized Swarm3K project by providing Docker Swarm monitoring tooling.You can read more about that in taming SwarmZilla: 150k Containers in 3K+ Docker Swarm Nodes. We flew to  San Francisco where we held public Elasticsearch trainings and Solr trainings.


This was a conference month for us. We talked about tuning Solr & pipeline for Logs and about running Solr in Docker at Lucene/Solr Revolution conference in Boston. Then, we presented a deep dive into using Elasticsearch for logs and metrics at O’REILLY Velocity 2016. November brought us two great guest posts: one about Docker Swarm lessons from Swarm3K by Prof. Chanwit Kaewkasi (Docker Captain who organized Swarm3K – the largest Docker Swarm cluster to date) and the other about exploring Windows Kernel with Fibratus and Logsene by Nedim Šabić, developer of Fibratus, a tool for exploration and tracing of the Windows kernel.


Still thinking about Logging Libraries vs Log Shippers? You may see some pros and cons here.  We were a bit disappointed when we couldn’t find any good, modern DataTable alternative that works well with React and Redux, so we built Sematable – ReactJS & Redux Table, a clean and easy to use React and Redux friendly data table.

Again some Docker knowledge that we shared, such as: how and why running Solr in Docker, logging and monitoring support for Kubernetes containers or making Elasticsearch in Docker Swarm Elastic.

And that was it with 2016 highlights!                 </div>
                                  <p class=Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex software systems for startups and enterprises. Since 2009 it has grown from a small group of passionate engineers and business...
SYS-CON Events announced today that GrapeUp, the leading provider of rapid product development at the speed of business, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market acr...
SYS-CON Events announced today that Ayehu will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara California. Ayehu provides IT Process Automation & Orchestration solutions for IT and Security professionals to identify and resolve critical incidents and enable rapid containment, eradication, and recovery from cyber security breaches. Ayehu provides customers greater control over IT infras...
What's the role of an IT self-service portal when you get to continuous delivery and Infrastructure as Code? This general session showed how to create the continuous delivery culture and eight accelerators for leading the change. Don Demcsak is a DevOps and Cloud Native Modernization Principal for Dell EMC based out of New Jersey. He is a former, long time, Microsoft Most Valuable Professional, specializing in building and architecting Application Delivery Pipelines for hybrid legacy, and cloud ...
Automation is enabling enterprises to design, deploy, and manage more complex, hybrid cloud environments. Yet the people who manage these environments must be trained in and understanding these environments better than ever before. A new era of analytics and cognitive computing is adding intelligence, but also more complexity, to these cloud environments. How smart is your cloud? How smart should it be? In this power panel at 20th Cloud Expo, moderated by Conference Chair Roger Strukhoff, pane...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...
Join us at Cloud Expo June 6-8 to find out how to securely connect your cloud app to any cloud or on-premises data source – without complex firewall changes. More users are demanding access to on-premises data from their cloud applications. It’s no longer a “nice-to-have” but an important differentiator that drives competitive advantages. It’s the new “must have” in the hybrid era. Users want capabilities that give them a unified view of the data to get closer to customers and grow business. The...
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
In this presentation, Striim CTO and founder Steve Wilkes will discuss practical strategies for counteracting fraud and cyberattacks by leveraging real-time streaming analytics. In his session at @ThingsExpo, Steve Wilkes, Founder and Chief Technology Officer at Striim, will provide a detailed look into leveraging streaming data management to correlate events in real time, and identify potential breaches across IoT and non-IoT systems throughout the enterprise. Strategies for processing massive ...
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud com...
Cloud Expo, Inc. has announced today that Andi Mann and Aruna Ravichandran have been named Co-Chairs of @DevOpsSummit at Cloud Expo Silicon Valley which will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. "DevOps is at the intersection of technology and business-optimizing tools, organizations and processes to bring measurable improvements in productivity and profitability," said Aruna Ravichandran, vice president, DevOps product and solutions marketing...
In his session at Cloud Expo, Alan Winters, an entertainment executive/TV producer turned serial entrepreneur, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to ma...