Welcome!

Blog Feed Post

Docker Log Management & Enrichment

Over the last several months we’ve made all kinds of improvements to Sematext Docker Agent (SDA).  If you’re not familiar with SDA yet, here it is in a nutshell

Sematext Docker Agent is a modern, open-source, Docker-native monitoring and log collection agent. It runs as a tiny container on each Docker host and provides automatic collection and processing of Docker Metrics, Events and Logs for all cluster nodes and all auto-discovered containers. It works with Kubernetes, Mesos, Docker Swarm, Docker Datacenter, Docker Cloud, as well as Amazon EC2, RancherOS, and CoreOS.

By default SDA collects all logs from all containers, but we’ve recently added LOGSENE_ENABLED_DEFAULT, a new flat that lets you control if log collection should be enabled by default or not. When set to false,  SDA will collect logs only from containers labelled with a LOGSENE_TOKEN=<Logsene app token> that, in addition to enabling log collection, also specifies where logs should be shipped. This is really useful because it:

  • Gives each team in an organization full control over log collection for their containers and the routing to their Logsene Apps or Elasticsearch indices. As you can imagine, this is very handy for larger organisations where e.g. one Swarm or Kubernetes cluster is shared by several teams, and each team needs to enable/disable log collection however they see fit and ship logs to different destinations.
  • Lets one exclude collection of logs from infrastructure containers whose logs may not contain enough value to be worth collecting.

The figure below illustrates this.  There are 3 containers running Nginx, Rancher agent, and MySQL.  We want to have explicit control over which containers logs are collected and where they are shipped.  We want to collect only Nginx logs, and we want to ship it to a specific Logsene app, not the default one set in SDA. We can accomplish that by doing the following

  • Set LOGSENE_ENABLED_DEFAULT=false flag in SDA config
  • Set Docker label LOGSENE_ENABLED=true for the Nginx container to enable its log collection
  • Set Docker label LOGSENE_TOKEN=…. for the Nginx container to specify which Logsene app we want to ship logs

https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 300w, https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 768w" sizes="(max-width: 960px) 100vw, 960px" />

Sematext Docker Agent log routing via container labels – disable log collection by default and enable it only for the Nginx container

 

We can also inverse the setup and enable log collection for all containers by default, as illustrated in the following figure.

https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 300w, https://sematext.com/wp-content/uploads/2017/05/LOGSENE_ENABLED_DEFAULT_... 768w" sizes="(max-width: 960px) 100vw, 960px" />

Sematext Docker Agent log routing via container labels (default settings) – collect and ship all logs, either to default Logsene app or to container specific apps.

We’ve also introduced TAGGING_LABELS.  This flag lets you enrich your container logs with data extracted from your existing Docker environment variables or labels.  Just specify patterns for values to extract, e.g. TAGGING_LABELS=”com.docker.,com.myorg.,role*”. This will add extracted values to your log events as additional fields (you can also think of this as meta-data) and let you easily slice and dice your logs by using values from labels or environment variables.  You can, of course, also build custom reports using this data, thus extracting more value and operational insight from your existing data in Sematext Cloud.

https://sematext.com/wp-content/uploads/2017/05/LOGS_WITH_DOCKER_LABELS-... 300w, https://sematext.com/wp-content/uploads/2017/05/LOGS_WITH_DOCKER_LABELS-... 768w, https://sematext.com/wp-content/uploads/2017/05/LOGS_WITH_DOCKER_LABELS-... 1024w" sizes="(max-width: 2419px) 100vw, 2419px" />Enriched logs with Docker labels

Need a tool that collects your containers Metrics + Events + Logs?  Try Sematext Docker Agent, it’s all open-source and can send your data to Sematext Cloud so you don’t have to manage or build out the backend for storing all the monitoring data, alerting, etc.  For feature requests, bugs, or PRs, see sematext/sematext-agent-docker.

 

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
Enterprises are moving to the cloud faster than most of us in security expected. CIOs are going from 0 to 100 in cloud adoption and leaving security teams in the dust. Once cloud is part of an enterprise stack, it’s unclear who has responsibility for the protection of applications, services, and data. When cloud breaches occur, whether active compromise or a publicly accessible database, the blame must fall on both service providers and users. In his session at 21st Cloud Expo, Ben Johnson, C...
Most of the time there is a lot of work involved to move to the cloud, and most of that isn't really related to AWS or Azure or Google Cloud. Before we talk about public cloud vendors and DevOps tools, there are usually several technical and non-technical challenges that are connected to it and that every company needs to solve to move to the cloud. In his session at 21st Cloud Expo, Stefano Bellasio, CEO and founder of Cloud Academy Inc., will discuss what the tools, disciplines, and cultural...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In his session at @DevOpsSummit at 21st Cloud Expo, Chris Van Tuin, Chief Technologist, NA West at Red Hat, will discuss: The top security r...
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
IBM helps FinTechs and financial services companies build and monetize cognitive-enabled financial services apps quickly and at scale. Hosted on IBM Bluemix, IBM’s platform builds in customer insights, regulatory compliance analytics and security to help reduce development time and testing. In his session at 21st Cloud Expo, Lennart Frantzell, a Developer Advocate with IBM, will discuss how these tools simplify the time-consuming tasks of selection, mapping and data integration, allowing devel...
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capabi...
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, vi...
There is huge complexity in implementing a successful digital business that requires efficient on-premise and cloud back-end infrastructure, IT and Internet of Things (IoT) data, analytics, Machine Learning, Artificial Intelligence (AI) and Digital Applications. In the data center alone, there are physical and virtual infrastructures, multiple operating systems, multiple applications and new and emerging business and technological paradigms such as cloud computing and XaaS. And then there are pe...
Today traditional IT approaches leverage well-architected compute/networking domains to control what applications can access what data, and how. DevOps includes rapid application development/deployment leveraging concepts like containerization, third-party sourced applications and databases. Such applications need access to production data for its test and iteration cycles. Data Security? That sounds like a roadblock to DevOps vs. protecting the crown jewels to those in IT.
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.i...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.