|By Trevor Parsons||
|August 12, 2014 01:01 PM EDT||
I was very interested to see that Manning Publications have just announced the Early Access release of a new book called Unified Log Management. I looked through the table of contents and it was what I expected - a roll your own log management cookbook with a list of technologies that might help (Kinesis, Redshift, Kafka etc.). Presumably it will also include Google DataFlow by the time the book ships. So it's only really for those super-talented all-rounders with a lot of time on their hands or companies with the budget to hire a team of software developers and data scientists. Not really for the very busy DevOps or IT Ops crew with systems to keep available and the need for powerful yet easy to use tools to help them distill the hosepipe of data being generated.
What really peaked my interest was the blog post by the book's author, Alex Dean, here. It turns out the "Unified" bit in the title was influenced by Jay Kreps article on the log. In my opinion, this is one of those articles that comes around once a generation! I don't think it's overstating the case to say that this captures all the most important lessons about big data, Internet of Things, massively scaled-out services in the cloud, growth hacking etc. It's up there with the 1988 paper that introduced the data warehouse concept and it does capture the future of enterprise architecture in a narrative that is refreshingly philosophical and erudite (at least for a tech blog post).
- Capture every event that happens in your business - particularly every customer interaction
- Store it multiple times because this stuff is gold
- Pick the right tools to view this data from the perspectives that make sense for your business
- Need better data - feed more into the log
- Need a new perspective - add a new tool
The perspectives are visualizations of the output data. The tools are getting more specialized every day... time-series databases for metrics, complex event processing engines for security threats, distributed graph databases for social interactions, search indexing etc. If you know what you are looking for then use real-time pattern matching. If you don't know what you are looking for but want to be told when things start to look different then look for anomalies.
The log, according to Kreps, is at the heart of all these data transformations.
Stream data into the system as fast as your customers can generate it. On the other end you get insights out of the system. And in the middle you painstakingly, hand-craft your log transformation algorithms (with your team of superstar coders and data scientists).
We are busy working on new perspectives and visualizations. New tools and transformations. New insights and new alerts. The book is not out until Spring 2015 so we'll have a ton more features out before then.
Seeing as we started talking about unified logs - how about this for unified?
“The Internet of Things transforms the way organizations leverage machine data and gain insights from it,” noted Splunk’s CTO Snehal Antani, as Splunk announced accelerated momentum in Industrial Data and the IoT. The trend is driven by Splunk’s continued investment in its products and partner ecosystem as well as the creativity of customers and the flexibility to deploy Splunk IoT solutions as software, cloud services or in a hybrid environment. Customers are using Splunk® solutions to collect ...
Oct. 6, 2015 01:00 PM EDT Reads: 580
Oct. 6, 2015 01:00 PM EDT Reads: 200
SYS-CON Events announced today that ProfitBricks, the provider of painless cloud infrastructure, will exhibit at SYS-CON's 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. ProfitBricks is the IaaS provider that offers a painless cloud experience for all IT users, with no learning curve. ProfitBricks boasts flexible cloud servers and networking, an integrated Data Center Designer tool for visual control over the...
Oct. 6, 2015 01:00 PM EDT Reads: 733
Oct. 6, 2015 01:00 PM EDT Reads: 216
Oct. 6, 2015 01:00 PM EDT Reads: 230
Oct. 6, 2015 12:45 PM EDT Reads: 456
Oct. 6, 2015 12:30 PM EDT Reads: 278
Oct. 6, 2015 12:30 PM EDT Reads: 572
Oct. 6, 2015 12:15 PM EDT
Oct. 6, 2015 12:00 PM EDT Reads: 405
Oct. 6, 2015 12:00 PM EDT Reads: 433
Oct. 6, 2015 12:00 PM EDT Reads: 282
Oct. 6, 2015 12:00 PM EDT Reads: 241
Oct. 6, 2015 12:00 PM EDT Reads: 274
Oct. 6, 2015 12:00 PM EDT Reads: 333