|By Trevor Parsons||
|August 12, 2014 01:01 PM EDT||
I was very interested to see that Manning Publications have just announced the Early Access release of a new book called Unified Log Management. I looked through the table of contents and it was what I expected - a roll your own log management cookbook with a list of technologies that might help (Kinesis, Redshift, Kafka etc.). Presumably it will also include Google DataFlow by the time the book ships. So it's only really for those super-talented all-rounders with a lot of time on their hands or companies with the budget to hire a team of software developers and data scientists. Not really for the very busy DevOps or IT Ops crew with systems to keep available and the need for powerful yet easy to use tools to help them distill the hosepipe of data being generated.
What really peaked my interest was the blog post by the book's author, Alex Dean, here. It turns out the "Unified" bit in the title was influenced by Jay Kreps article on the log. In my opinion, this is one of those articles that comes around once a generation! I don't think it's overstating the case to say that this captures all the most important lessons about big data, Internet of Things, massively scaled-out services in the cloud, growth hacking etc. It's up there with the 1988 paper that introduced the data warehouse concept and it does capture the future of enterprise architecture in a narrative that is refreshingly philosophical and erudite (at least for a tech blog post).
- Capture every event that happens in your business - particularly every customer interaction
- Store it multiple times because this stuff is gold
- Pick the right tools to view this data from the perspectives that make sense for your business
- Need better data - feed more into the log
- Need a new perspective - add a new tool
The perspectives are visualizations of the output data. The tools are getting more specialized every day... time-series databases for metrics, complex event processing engines for security threats, distributed graph databases for social interactions, search indexing etc. If you know what you are looking for then use real-time pattern matching. If you don't know what you are looking for but want to be told when things start to look different then look for anomalies.
The log, according to Kreps, is at the heart of all these data transformations.
Stream data into the system as fast as your customers can generate it. On the other end you get insights out of the system. And in the middle you painstakingly, hand-craft your log transformation algorithms (with your team of superstar coders and data scientists).
We are busy working on new perspectives and visualizations. New tools and transformations. New insights and new alerts. The book is not out until Spring 2015 so we'll have a ton more features out before then.
Seeing as we started talking about unified logs - how about this for unified?
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
Oct. 25, 2016 08:45 PM EDT Reads: 2,685
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
Oct. 25, 2016 08:45 PM EDT Reads: 3,239
Established in 1998, Calsoft is a leading software product engineering Services Company specializing in Storage, Networking, Virtualization and Cloud business verticals. Calsoft provides End-to-End Product Development, Quality Assurance Sustenance, Solution Engineering and Professional Services expertise to assist customers in achieving their product development and business goals. The company's deep domain knowledge of Storage, Virtualization, Networking and Cloud verticals helps in delivering ...
Oct. 25, 2016 08:30 PM EDT Reads: 1,095
Oct. 25, 2016 08:15 PM EDT Reads: 1,369
Oct. 25, 2016 07:30 PM EDT Reads: 1,107
Oct. 25, 2016 07:30 PM EDT Reads: 1,536
Oct. 25, 2016 07:15 PM EDT Reads: 284
Oct. 25, 2016 07:00 PM EDT Reads: 1,024
Oct. 25, 2016 07:00 PM EDT Reads: 342
Oct. 25, 2016 06:45 PM EDT Reads: 1,241
Oct. 25, 2016 06:45 PM EDT Reads: 1,439
Oct. 25, 2016 06:15 PM EDT Reads: 376
Oct. 25, 2016 06:15 PM EDT Reads: 2,141
Oct. 25, 2016 06:00 PM EDT Reads: 2,705
Oct. 25, 2016 05:45 PM EDT Reads: 2,892