Welcome!

Blog Feed Post

2016 Year in Review: Monitoring and Logging Highlights

2017 is almost here and, like last year, we thought we’d share how 2016 went for us.  We remain committed to be your “one-stop shop” for all things Elasticsearch and Solr: from Consulting, Production Support, and Training, to complementing that with our Logsene for all your logs, and SPM for all your monitoring needs.

Docker

It’s safe to say 2016 was the year of Docker and by extension Kubernetes, Mesos, Docker Swarm, among others, too.  They stopped being just early adopters’ toys and have become production-ready technologies used by many. This year we’ve added excellent support for Docker monitoring with SPM and logging with Logsene via the open-source Sematext Docker Agent.

But, let’s walk through 2016 highlights in the chronological order, month by month.

January

We started the year fresh and published a lot of great content on our blog: from a bunch of useful articles about handling log management (such as: How to forward CloudTrail (or other logs from AWS S3) to Logsene, announcing PagerDuty and Logsene integration or using Filebeat to send Elasticsearch logs to Logsene) to hot topics about monitoring key metrics (such as: introducing NetMaps, a new feature in SPM or presenting the Sematext Docker Agent as a turnkey-solution for Docker Logs, Metrics and Events). If you, like many of our readers, run Elasticsearch and are considering running it inside Docker, have a look at how to run high performance and fault tolerant Elasticsearch Clusters on Docker.

February

We continued our journey through the logging field by describing how to send your Windows Event Logs to Logsene using NxLog and Logstash. In February we introduced Logagent – a log parser and shipper with log patterns for a number of popular log formats – from various Docker images including Nginx, Apache, Linux and Mac system logs, to Elasticsearch, Redis, Solr, MongoDB and more. Logagent detects the log format automatically using the built-in pattern definitions (also lets you provide your own, custom patterns). You may read more here. Also, we thought about our team needs too, as Sematext team is highly distributed and we prepared a insightful blog post showing how to index your Slack messages in Logsene for the purpose of archiving, searching, and analytics.

March

Spring was good to us. We announced the launch of on-demand Java Profiling that lets you profile any app running on top of JVM, even in production. If you are using AWS like us and want to collect all your application logs, you may want to start sending AWS CloudWatch Logs (e.g. VPC logs) to Logsene. As Elasticsearch is our expertise, March ended with two related and popular topics: reindexing data with Elasticsearch and documents update by query with Elasticsearch.

April

Docker, Solr and Elasticsearch were our focus in April. We went to London to give several public Elasticsearch and Solr classes: Core Solr Training, Elasticsearch for Developers; Elasticsearch for Logging and Elasticsearch Operations. While not in London we found time to write about Docker Cloud: Monitoring & Logging, Automatic Geo-IP Enrichment for Docker Logs and Monitoring Kafka on Docker Cloud (just to name a few).

May

We were recognized by Docker as the Ecosystem Technology Partner (ETP) for monitoring and logging. This designation indicates that our products: SPM Performance Monitoring and Logsene have demonstrated working integration with the Docker platform via the Docker API and are available to users and organizations that seek solutions to monitor their Dockerized distributed applications. Just take a look at Sematext Docker Agent, which is extremely easy to deploy with Kubernetes, Docker Swarm, Docker Cloud and Docker Datacenter.

We covered a few more interesting Solr and Elasticsearch topics, such as: Solr streaming expressions for collection auto-updating, DocValues reindexing with Solr streaming expressions, or scalable and flexible Elasticsearch reindexing via rsyslog.

June

The middle of the year found us in New York, holding public Elasticsearch / Elastic Stack Training and Apache Solr Training. While some of our colleagues were in New York, others were giving talks about running Elasticsearch in Docker at Berlin Buzzwords. Indeed, monitoring of Docker environments is challenging enough, so we shared our Docker DevOps experience in Top Docker Metrics to Watch.

July

We created a handy mindmap showing a number of  open source Docker Monitoring & Logging tools and we published a reference guide for Monitoring Docker Datacenter Logs & Metrics.

As the ecosystem of DevOps tools and resources – for monitoring, for logging, for alerting, for continuous integration and deployment, configuration management, etc. –  is growing, finding answers as quickly and efficiently as possible becomes extremely important. We are DevOps like many of our followers and often need to research specific DevOps tools. To help with that we launched search-devops.com, which aggregates, indexes and makes searchable all content repositories (mailing lists, source code, wikis, issue trackers, etc.) for a number of open source DevOps projects.

August

August was about logging all the way. If you are interested in Elastic Stack Import-Export with Logstash & Logsene or RancherOS Monitoring and Logging Support these are the how-to articles to read.

September

When it comes to centralizing logs to Elasticsearch the first log shipper that comes to mind is Logstash. However, Logstash is far from being the only game in town. In 5 Logstash Alternatives we covered other log shippers you may want to consider. We couldn’t let September pass without covering Docker, of course.  Here’s how to set up full Docker Swarm Cluster Monitoring & Logging with 1 Command.

October

We participated in an exciting community-organized Swarm3K project by providing Docker Swarm monitoring tooling.You can read more about that in taming SwarmZilla: 150k Containers in 3K+ Docker Swarm Nodes. We flew to  San Francisco where we held public Elasticsearch trainings and Solr trainings.

November

This was a conference month for us. We talked about tuning Solr & pipeline for Logs and about running Solr in Docker at Lucene/Solr Revolution conference in Boston. Then, we presented a deep dive into using Elasticsearch for logs and metrics at O’REILLY Velocity 2016. November brought us two great guest posts: one about Docker Swarm lessons from Swarm3K by Prof. Chanwit Kaewkasi (Docker Captain who organized Swarm3K – the largest Docker Swarm cluster to date) and the other about exploring Windows Kernel with Fibratus and Logsene by Nedim Šabić, developer of Fibratus, a tool for exploration and tracing of the Windows kernel.

December

Still thinking about Logging Libraries vs Log Shippers? You may see some pros and cons here.  We were a bit disappointed when we couldn’t find any good, modern DataTable alternative that works well with React and Redux, so we built Sematable – ReactJS & Redux Table, a clean and easy to use React and Redux friendly data table.

Again some Docker knowledge that we shared, such as: how and why running Solr in Docker, logging and monitoring support for Kubernetes containers or making Elasticsearch in Docker Swarm Elastic.

And that was it with 2016 highlights!                 </div>
      
                                  <p class=Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
What if you could build a web application that could support true web-scale traffic without having to ever provision or manage a single server? Sounds magical, and it is! In his session at 20th Cloud Expo, Chris Munns, Senior Developer Advocate for Serverless Applications at Amazon Web Services, will show how to build a serverless website that scales automatically using services like AWS Lambda, Amazon API Gateway, and Amazon S3. We will review several frameworks that can help you build serverle...
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, will provide a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services ...
VeriStor Systems has announced that CRN has named VeriStor to its 2017 Managed Service Provider (MSP) 500 list in the Elite 150 category. This annual list recognizes North American solution providers with cutting-edge approaches to delivering managed services. Their offerings help companies navigate the complex and ever-changing landscape of IT, improve operational efficiencies, and maximize their return on IT investments. In today’s fast-paced business environments, MSPs play an important role...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
SYS-CON Events announced today that Juniper Networks (NYSE: JNPR), an industry leader in automated, scalable and secure networks, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Juniper Networks challenges the status quo with products, solutions and services that transform the economics of networking. The company co-innovates with customers and partners to deliver automated, scalable and secure network...
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In his Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, will explore t...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Have you ever noticed how some IT people seem to lead successful, rewarding, and satisfying lives and careers, while others struggle? IT author and speaker Don Crawley uncovered the five principles that successful IT people use to build satisfying lives and careers and he shares them in this fast-paced, thought-provoking webinar. You'll learn the importance of striking a balance with technical skills and people skills, challenge your pre-existing ideas about IT customer service, and gain new in...
"When you think about the data center today, there's constant evolution, The evolution of the data center and the needs of the consumer of technology change, and they change constantly," stated Matt Kalmenson, VP of Sales, Service and Cloud Providers at Veeam Software, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...