Welcome!

Blog Feed Post

What is Syslog: Daemons, Message Formats and Protocols

Pretty much everyone’s heard about syslog: with its roots in the 80s, it’s still used for a lot of the logging done today. Mostly because of its long history, syslog is quite a vague concept, referring to many things. Which is why you’ve probably heard:

  • Check syslog, maybe it says something about the problem – referring to /var/log/messages.
  • Syslog doesn’t support messages longer than 1K – about message format restrictions.
  • Syslog is unreliable – referring to the UDP protocol.

In this post, we’ll explain the different facets by being specific: instead of saying “syslog”, you’ll read about syslog daemons, about syslog message formats and about syslog protocols.

Note the plurals: there are multiple options for each. We’ll show the important ones here to shed some light over the vague (and surprisingly rich) concept. Along the way, we’ll debunk some of the myths surrounding syslog. For example, you can choose to limit messages to 1K and you can choose to send them via UDP, but you don’t have to – it’s not even a default in modern syslog daemons.

Syslog daemons

A syslog daemon is a program that:

  • can receive local syslog messages. Traditionally /dev/log UNIX socket and kernel logs.
  • can write them to a file. Traditionally /var/log/messages or /var/log/syslog will receive everything, while some categories of messages go to specific files, like /var/log/mail.
  • can forward them to the network or other destinations. Traditionally, via UDP. Usually, the daemon also implements equivalent network listeners (UDP in this case).

This is where syslog is often referring to syslogd or sysklogd, the original BSD syslog daemon. Development for it stopped for Linux since 2007, but continued for BSDs and OSX. There are alternatives, most notably:
* rsyslog. Originally a fork of syslogd, it still can be used as a drop in replacement for it. Over the years, it evolved into a performance-oriented, multipurpose logging tool, that can read data from multiple sources, parse and enrich logs in various ways, and ship to various destinations
* syslog-ng. Unlike rsyslog, it used a different configuration format from the start (rsyslog eventually got to the same conclusion, but still supports the BSD syslog config syntax as well – which can be confusing at times). You’d see a similar feature set to rsyslog, like parsing unstructured data and shipping it to Elasticsearch or Kafka. It’s still fast and light, and while it may not have the ultimate performance of rsyslog (depends on the use-case, see the comments section), it has better documentation and it’s more portable
* nxlog. Yet another syslog daemon which evolved into a multi-purpose log shipper, it sets itself apart by working well on Windows

In essence, a modern syslog daemon is a log shipper that works with various syslog message formats and protocols. If you want to learn more about log shippers in general, we wrote a side-by-side comparison of Logstash and 5 other popular shippers, including rsyslog and syslog-ng.

Myths about syslog daemons

The one we come across most often is that syslog daemons are no good if you log to files or if you want to parse unstructured data. This used to be true years ago, but then so was Y2K. Things changed in the meantime. In the myth’s defense, some distributions ship with old versions of rsyslog and syslog-ng. Plus, the default configuration often only listens for /dev/log and kernel messages (it doesn’t need more), so it’s easy to generalize.

Syslog message formats

You’ll normally find syslog messages in two major formats:

RFC3164 a.k.a “the old format”

Although RFC suggests it’s a standard, RFC3164 was more of a collection of what was found in the wild at the time (2001), rather than a spec that implementations will adhere to. As a result, you’ll find slight variations of it. That said, most messages will look like the RFC3164 example:

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

This is how the application should log to /dev/log, and you can see some structure:

  • <34> is a priority number. It represents the facility number multiplied by 8, to which severity is added. In this case, facility=4 (Auth) and severity=2 (Critical).
  • Oct 11 22:14:15 is commonly known as syslog timestamp. It misses the year, the time-zone and doesn’t have sub-second information. For those reasons, rsyslog also parses RFC3164-formatted messages with an ISO-8601 timestamp instead
  • mymachine is a host name where the message was written.
  • su: is a tag. Typically this is the process name – sometimes having a PID (like su[1234]:). The tag typically ends in a colon, but it may end up just with the square brackets or with a space.
  • the message (MSG) is everything after the tag. In this example, since we have the colon to separate the tag and the message, the message actually starts with a space. This tiny detail often gives a lot of headache when parsing.

In /var/log/messages, you’ll often see something like this:

Oct 11 22:14:15 su: 'su root' failed for lonvick on /dev/pts/8

This isn’t a syslog message format, it’s just how most syslog deamons write messages to files by default. Usually, you can choose how the output data looks like, for example rsyslog has templates.

RFC5424 a.k.a. “the new format”

RFC5424 came up in 2009 to deal with the problems of RFC3164. First of all, it’s an actual standard, that daemons and libraries chose to implement. Here’s an example message:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - - - 'su root' failed for lonvick on /dev/pts/8

Now we get an ISO-8601 timestamp, amongst other improvements. We also get more structure: the dashes you can see there are places for PID, message ID and other structured data you may have. That said, RFC5424 structured data never really took off, as people preferred to put JSON in the syslog message (whether it’s the old or the new format). Finally, the new format supports UTF8 and other encodings, not only ASCII, and it’s easier to extend because it has a version number (in this example, the 1 after the priority number).

Myths around the syslog message formats

The ones we see more often are:

Syslog protocols

Originally, syslog messages were sent over the wire via UDP – which was also mentioned in RFC3164. It was later standardized in RFC5426, after the new message format (RFC5424) was published.

Modern syslog daemons support other protocols as well. Most notably:

  • TCP. Just like the UDP, it was first used in the wild and then documented. That documentation finally came with RFC6587, which describes two flavors:
    • messages are delimited by a trailer character, typically a newline
    • messages are framed based on an octet count
  • TLS. Standardized in RFC5425, which allows for encryption and certificate-based authorization
  • RELP. Unlike plain TCP, RELP adds application-level acknowledgements, which provides at-least-once guarantees on delivering messages. You can also get RELP with TLS if you need encryption and authorization

Besides writing to files and communicating to each other, modern syslog daemons can also write to other destinations. For example, datastores like MySQL or Elasticsearch or queue systems such as Kafka and RabbitMQ. Each such destination often comes with its own protocol and message format. For example, Elasticsearch uses JSON over HTTP (though you can also secure it and send syslog messages over HTTPS).

Myths around syslog protocols

The ones we hear most come from the assumption that UDP is the only option, implying there’s no reliability, authorization or encryption.

The other frequent one is that you can’t send multiline messages, like stack traces. This is only true for TCP syslog, if newlines are used for delimiting. Then, a stacktrace will end up as multiple messages at the destination – unless its newlines are escaped at the source and reverted at the destination. With UDP, multiline logs work out of the box, because you have one message per datagram. Other protocols (TLS, RELP and octet-delimited TCP) also handle multiline logs well, by framing messages.

What’s next?

Hopefully this post helped clear the fog around syslog. If you’re looking for tips on how to configure your syslog daemon, you can find a lot of them on this blog. We especially love the topic of centralizing logs with Elasticsearch. That’s because we run Logsene, our logging SaaS that exposes the Elasticsearch API which supports all the syslog we discussed here in terms of message formats (including JSON over syslog) and protocols (UDP, TCP, TLS, RELP).

You can sign up and get a free Logsene trial here. You’ll find configuration samples for all major log shippers (including the syslog daemons discussed in this post).

If you’re looking to build a log-centralization solution for yourself, we can help: either through logging consulting or trainings on Elasticsearch and its logging ecosystem. If you’re into logging in general and want to build such solutions for others, we’re hiring worldwide.

SIGN UP – FREE TRIAL

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists looked at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deliver...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ago and we implemented this knowledge into our software," explained Jakub Ratajczak, Business Development Manager at MooseFS, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Tintri focuses on the Ops side of the DevOps, which basically is pushing more and more of the accessibility of the infrastructure to the developers and trying to get behind the scenes," explained Dhiraj Sehgal of Tintri in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Cloud applications are seeing a deluge of requests to support the exploding advanced analytics market. “Open analytics” is the emerging strategy to deliver that data through an open data access layer, in the cloud, to be directly consumed by external analytics tools and popular programming languages. An increasing number of data engineers and data scientists use a variety of platforms and advanced analytics languages such as SAS, R, Python and Java, as well as frameworks such as Hadoop and Spark...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cl...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 21st Int\ernational Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their ...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), provided an overview of various initiatives to certify the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldwide re...