Welcome!

Article

Making the World a Better Place with Big Data

This is just a summary of some of the great content at the conference

I spent last week at the Strata + Hadoop World Conference in New York City with 5000 other “big data” customers, vendors, and enthusiasts. In the last 6 months we’ve seen demand for a “big data” based network infrastructure really start to take off, and I’ve spent a lot of time recently trying to better understand the evolving market and technology landscape and use cases. I’m particularly interested in how network infrastructure can drive a better experience for users of big data applications, or networking/infrastructure teams that need to support these applications, but ultimately I want to know what do businesses get out of these investments in data, analytics, and infrastructure.

[On a related note, as part of our efforts to provide the best “Big Data Fabric” we recently brought on @networkn3rd (Ed Henry) to Plexxi to fully define our reference architecture. Ed will be demo’ing the first fruits of his labor this Friday on SDN’s Central’s Demo Friday - Register Here].

Hadoop World was a really great experience. As a relative newbie to Big Data, I have a lot to learn and this was a great place to soak up actual customer use cases. While there was certainly much feel-good hyperbole about the “making the world a better place” (if you haven’t seen HBO’s Silicon Valley, please watch this!) nature of big data, that was more than offset with actual real-world details of how data was being used to solve more day-to-day business problems. Here’s a quick synopsis of some of my personal highlights:

  • Finance People have their Sh*t Together In “Why Marketing Suck at Big Data” Jennifer Zeszut (@jenniferland) from Beckon pleaded with Marketers to learn from other functional areas that actually make formal use of data in pretty structured ways (e.g. finance people). Her message was a bit contrarian to the whole Big Data notion of data exploration – she talked about structuring the data on the way in, storing only the data that matters, and avoiding the data “spelunking” approach. She used some great examples of what Finance people would never do (like throw all receipts into one big data warehouse without any input classification!).
  • Data is Intrinsically Worthless In “Do you Know What Your Data is Worth?” Brian d’Alessandro from Dstillery (@delbrians)  talked about how you can easily double your data, which often doubles the investment (cost to acquire the data) but doesn’t double the benefit. He captured the “Value of Data (VOD)” in a handy equation that looked at understanding the value of an application with data minus the value of an application without data. A key lesson I learned here was that data has no intrinsic value – rather it is tied to the applications and actions derived from the data.
  • Surprise leads to Innovation In “The Sound of Data Silence” Jana Eggers from Nara Logics (@jeggers) talked about how to better listen to the non-obvious signals in the data. She gave some very practical exercises on how to do this including going beyond the ‘show me state’ to the ‘curiosity state’ – being hyper curious by channelling your inner Steve Jobs and remembering not to rationalize surprises in the data as this is the true source of innovation.
  • The Data Natives Generation In “The Future of Data” Kim Rees of Periscopic (@krees) gave a fascinating talk on just that – the future of data. To demonstrate how data is much more scalable than algorithms, she used an example of a robot that crowd sources its knowledge on how to handle objects from other robots’ data. Then she led us to the “data natives” phenomenon – using kids’ familiarity with gadgets as an analogy – data natives speaks to how we’ll very quickly have a generation that will be born with the universe of data at their fingertips and from birth will never need to remember or figure. This marks a new state in our evolution.
  • The Kevin Bacon Game for Banking In “How Goldman Sachs is Using Knowledge to Create an Information Edge” Peter Ferns talked about the GS “Big Graph” application and how it is used to build a relationship graph of people, legal entities, organizational entities, transactional data, and banking transactions like M&A. He then detailed how this information is used for compliance (surveillance, investigations and analytics), information security, technology infrastructure management, and customer relationship management. The best thing is that they put this info out in the public at http://www.gs.com/engineering!
  • Big People In “Building with Data: Lessons from Etsy” Nellwyn Thomas talked about how she built the data organization at Etsy.  She covered how they have 3 groups within the “Data Org” – Data engineering / hadoop team, data science team, and analysts. Then she zoomed in on the specific skill sets that are required for analysts – not just analytical skills (which she defined as the ability to understand the problem and the opportunity), but also math/statistics skills (to understand the data), technical skills (to get, parse, and visualize the data), and communication skills (to communicate what matters, and more importantly to not communicate what doesn’t matter).
  • Bad Recommendations Make Angry Customers In “Learning About Music and Listeners” Brian Whitman (@bwhitman) from Spotify gave what was probably my favorite session of the week. He talked about the company he started called The Echo Nest (that was acquired by Spotify) and the work they are doing to move beyond simple collaborative filtering engines (the ‘other customers bought this’ type). Spotify’s goal is to make users loyal by encouraging discovery, understanding that giving bad recommendations really makes music fans angry! They are doing this with content-based recommendations. He talked about the progress they have made in helping computers understand enough music to make recommendations, and the obvious but almost always overlooked basic human fact that we can actually have different “modes” that determine what we might want to listen too! He also went through some fascinating data analysis derived solely from user usage data – like predicting political affiliations. Pretty cool stuff. Best of all was the fact that the entire Echo Nest API and a million song data set are available for anyone to use for research purposes on http://developer.spotify.com.
  • Knowledge is Dangerous Last but not least, the venerable Shankar Vedantam (@HiddenBrain), author of “The Hidden Brain” and the Social Science Correspondent on NPR warned us all that more data doesn’t necessarily make us smarter or better. In fact, what the data shows is that the more knowledge we have, the more we amplify our own existing biases into stronger positions because we ultimately are really good at cherry picking what we want to believe. It was a bit sobering given the tone of the conference, but a practical message nonetheless.

This is just a summary of some of the great content at the conference, and I’m leaving out a great deal. I also spent a ton of time in our booth talking to some really fascinating customers and learning about what they are doing (and of course doing a bit of selling). Bottom line is that Big Data is not only big, but it has real/broad applications beyond the typical web crawling for the search crowd -including customer profiling for marketing/sales across a variety of industries, content and goods recommendations for eCommerce and online media, fraud detection/compliance for banking, resource allocation / inventory planning for retailers/manufacturers, and of course solving world hunger and making the world a better place!

All of this will ultimately drive new infrastructure designs and decisions as the data sets get larger, the users get more diverse and more demanding, and the expectations to provide real-time analysis across many data sets becomes more possible. Don’t forget to tune into our SDN Central Demo Friday to see how we’re starting to tackle these infrastructure challenges from the network side!

[Today’s fun fact: 111,111,111 x 111,111,111 = 12,345,678,987,654,321. That's big data for you.]

The post Making the World a Better Place with Big Data appeared first on Plexxi.

More Stories By Mat Mathews

Visionary solutions are built by visionary leaders. Plexxi co-founder and Vice President of Product Management Mat Mathews has spent 20 years in the networking industry observing, experimenting and ultimately honing his technology vision. The resulting product — a combination of traditional networking, software-defined networking and photonic switching — represents the best of Mat's career experiences. Prior to Plexxi, Mat held VP of Product Management roles at Arbor Networks and Crossbeam Systems. Mat began his career as a software engineer for Wellfleet Communications, building high speed Frame Relay Switches for the carrier market. Mat holds a Bachelors of Science in Computer Systems Engineering from the University of Massachusetts at Amherst.

Latest Stories
Cloud-based disaster recovery is critical to any production environment and is a high priority for many enterprise organizations today. Nearly 40% of organizations have had to execute their BCDR plan due to a service disruption in the past two years. Zerto on IBM Cloud offer VMware and Microsoft customers simple, automated recovery of on-premise VMware and Microsoft workloads to IBM Cloud data centers.
SYS-CON Events announced today that Daiya Industry will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Daiya Industry specializes in orthotic support systems and assistive devices with pneumatic artificial muscles in order to contribute to an extended healthy life expectancy. For more information, please visit https://www.daiyak...
The Internet giants are fully embracing AI. All the services they offer to their customers are aimed at drawing a map of the world with the data they get. The AIs from these companies are used to build disruptive approaches that cannot be used by established enterprises, which are threatened by these disruptions. However, most leaders underestimate the effect this will have on their businesses. In his session at 21st Cloud Expo, Rene Buest, Director Market Research & Technology Evangelism at Ara...
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
One of the biggest challenges with adopting a DevOps mentality is: new applications are easily adapted to cloud-native, microservice-based, or containerized architectures - they can be built for them - but old applications need complex refactoring. On the other hand, these new technologies can require relearning or adapting new, oftentimes more complex, methodologies and tools to be ready for production. In his general session at @DevOpsSummit at 20th Cloud Expo, Chris Brown, Solutions Marketi...
In the world of DevOps there are ‘known good practices’ – aka ‘patterns’ – and ‘known bad practices’ – aka ‘anti-patterns.' Many of these patterns and anti-patterns have been developed from real world experience, especially by the early adopters of DevOps theory; but many are more feasible in theory than in practice, especially for more recent entrants to the DevOps scene. In this power panel at @DevOpsSummit at 18th Cloud Expo, moderated by DevOps Conference Chair Andi Mann, panelists discussed...
Elon Musk is among the notable industry figures who worries about the power of AI to destroy rather than help society. Mark Zuckerberg, on the other hand, embraces all that is going on. AI is most powerful when deployed across the vast networks being built for Internets of Things in the manufacturing, transportation and logistics, retail, healthcare, government and other sectors. Is AI transforming IoT for the good or the bad? Do we need to worry about its potential destructive power? Or will we...
Many organizations adopt DevOps to reduce cycle times and deliver software faster; some take on DevOps to drive higher quality and better end-user experience; others look to DevOps for a clearer line-of-sight to customers to drive better business impacts. In truth, these three foundations go together. In this power panel at @DevOpsSummit 21st Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, industry experts will discuss how leading organizations build application success from all...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
The last two years has seen discussions about cloud computing evolve from the public / private / hybrid split to the reality that most enterprises will be creating a complex, multi-cloud strategy. Companies are wary of committing all of their resources to a single cloud, and instead are choosing to spread the risk – and the benefits – of cloud computing across multiple providers and internal infrastructures, as they follow their business needs. Will this approach be successful? How large is the ...
Your clients expect transactions to never fail, cloud access to be fast and always on, and their data to be protected - no exceptions. Hear about how Secure Service Container (SSC), an IBM-exclusive open technology, enables secure building and hosting of next-generation applications, both cloud and on-premises. SSC protects the full stack from external and insider threats, allows automatic encryption of data in-flight and at-rest, and is tamper-resistant during installation and runtime – with no...