|By Trevor Parsons||
|August 12, 2014 01:01 PM EDT||
I was very interested to see that Manning Publications have just announced the Early Access release of a new book called Unified Log Management. I looked through the table of contents and it was what I expected - a roll your own log management cookbook with a list of technologies that might help (Kinesis, Redshift, Kafka etc.). Presumably it will also include Google DataFlow by the time the book ships. So it's only really for those super-talented all-rounders with a lot of time on their hands or companies with the budget to hire a team of software developers and data scientists. Not really for the very busy DevOps or IT Ops crew with systems to keep available and the need for powerful yet easy to use tools to help them distill the hosepipe of data being generated.
What really peaked my interest was the blog post by the book's author, Alex Dean, here. It turns out the "Unified" bit in the title was influenced by Jay Kreps article on the log. In my opinion, this is one of those articles that comes around once a generation! I don't think it's overstating the case to say that this captures all the most important lessons about big data, Internet of Things, massively scaled-out services in the cloud, growth hacking etc. It's up there with the 1988 paper that introduced the data warehouse concept and it does capture the future of enterprise architecture in a narrative that is refreshingly philosophical and erudite (at least for a tech blog post).
- Capture every event that happens in your business - particularly every customer interaction
- Store it multiple times because this stuff is gold
- Pick the right tools to view this data from the perspectives that make sense for your business
- Need better data - feed more into the log
- Need a new perspective - add a new tool
The perspectives are visualizations of the output data. The tools are getting more specialized every day... time-series databases for metrics, complex event processing engines for security threats, distributed graph databases for social interactions, search indexing etc. If you know what you are looking for then use real-time pattern matching. If you don't know what you are looking for but want to be told when things start to look different then look for anomalies.
The log, according to Kreps, is at the heart of all these data transformations.
Stream data into the system as fast as your customers can generate it. On the other end you get insights out of the system. And in the middle you painstakingly, hand-craft your log transformation algorithms (with your team of superstar coders and data scientists).
We are busy working on new perspectives and visualizations. New tools and transformations. New insights and new alerts. The book is not out until Spring 2015 so we'll have a ton more features out before then.
Seeing as we started talking about unified logs - how about this for unified?
SYS-CON Events announced today that Pulzze Systems will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Pulzze Systems, Inc. provides infrastructure products for the Internet of Things to enable any connected device and system to carry out matched operations without programming. For more information, visit http://www.pulzzesystems.com.
Aug. 31, 2016 10:45 AM EDT Reads: 431
Aug. 31, 2016 10:11 AM EDT
Aug. 31, 2016 09:30 AM EDT Reads: 1,088
Aug. 31, 2016 09:00 AM EDT Reads: 939
Aug. 31, 2016 09:00 AM EDT Reads: 506
Aug. 31, 2016 09:00 AM EDT Reads: 1,014
Aug. 31, 2016 08:45 AM EDT Reads: 922
Aug. 31, 2016 08:30 AM EDT Reads: 913
Aug. 31, 2016 07:45 AM EDT Reads: 960
Aug. 31, 2016 07:45 AM EDT Reads: 985
Aug. 31, 2016 04:45 AM EDT Reads: 2,312
Aug. 31, 2016 02:45 AM EDT Reads: 1,902
Aug. 31, 2016 02:15 AM EDT Reads: 2,295
Aug. 31, 2016 02:00 AM EDT Reads: 2,031
Aug. 31, 2016 01:45 AM EDT Reads: 3,115