Welcome!

News Feed Item

Databricks Launches “Certified Spark Distribution” Program to Recognize Vendors Committed to Supporting the Apache Spark Application Ecosystem

Databricks, the company founded by the creators of Apache Spark, the next generation Big Data engine, today announced the “Certified Spark Distribution” program for vendors with a commercial Spark distribution. Certification indicates that the vendor’s Spark distribution is compatible with the open source Apache Spark distribution, enabling “Certified on Spark” applications - certified to work with Apache Spark - to run on the vendor’s Spark distribution out-of-the-box.

“One of Databricks’ goals is to ensure users have a fantastic experience. Our belief is that having the community work together to maintain compatibility and therefore facilitate a vibrant application ecosystem is crucial to this vision,” said Ion Stoica, Databricks CEO. “We first launched the ‘Certified on Spark’ program to help build a robust ecosystem of innovative applications on top of Apache Spark. The ‘Certified Spark Distribution’ program is the other half of the equation, recognizing vendors that are committed to providing a home for these applications to allow the ecosystem to flourish.”

In keeping with the open source nature of Spark, the certification process is fully transparent with open-source tests, lightweight, and 100% free - a mirror image of the “Certified on Spark” process for Spark applications. Vendors fill out a short questionnaire and then simply execute a set of open-source tests - developed and maintained by the community and used to test each release of Apache Spark - against their build of Spark to demonstrate compatibility.

“Certification shouldn’t be used as a tool for lock-in: Certified Spark Distributions are not required to ship all the bits of Apache Spark, or be open source, or prevented from innovating significantly within and around Spark,” said Arsalan Tavakoli-Shiraji, Business Development Lead at Databricks. “They simply need to maintain compatibility with Apache Spark to provide support for the application ecosystem.”

As part of the certification program launch, five vendors have completed the certification process: DataStax, Hortonworks, IBM, Oracle, and Pivotal - industry leaders that have recognized and embraced the power of Spark when integrated with their respective platforms. Each of these vendors put their distributions through the certification process, which included a host of integration tests to ensure full compatibility with the latest Apache Spark release.

“One of the big risks faced by open source projects is fragmentation among distributors. Fragmentation is bad for both users and application developers, and ultimately for the growth of the project,” said Matei Zaharia, Databricks CTO and VP of the Spark project at Apache. “We are delighted that these partners - along with others in the certification pipeline - share our vision of an undivided Spark platform based directly around Apache, and will ensure that all applications built on Apache Spark run on their distributions.”

Vendors interested in certifying their Spark distribution should visit www.databricks.com and select "Apply for Certification." Enterprise users can also visit the Databricks site regularly to see the latest set of certified distributions and applications, and read “spotlight” blog articles that provide deep-dives on the Spark ecosystem by newly certified vendors.

All the inaugural members will be on hand at the upcoming Spark Summit from June 30th to July 2nd in San Francisco to provide greater information on the role of Spark in helping better serve their customers. Additionally, there will be an “Application Spotlight” segment that will highlight innovative “Certified on Spark” applications.

Supporting Quotes:

"DataStax is strongly committed to making Cassandra and Spark the best combination for today's online applications," said Robin Schumacher, VP of products at DataStax. "We have demonstrated that commitment with the integration work we have contributed back to both open source communities as well as the certified versions of Spark and Cassandra we provide in DataStax Enterprise for production environments."

“We support the fact that Apache Spark project provides enterprises with an additional processing engine in Hadoop to execute in-memory algorithms for advanced analytics,” said John Kreisa, vice president of strategic marketing at Hortonworks. “We applaud Databricks’ vision to ensure Spark is fully integrated on YARN, which enterprises have adopted as the data OS for Hadoop.”

"Pivotal's open source credentials are quite extensive - Apache-compatible Hadoop, MADLib, RabbitMQ, CloudFoundry - and now we've added Spark to that set," said Sarabjeet Chugh, Head of Hadoop Product Management at Pivotal. "Additionally, we recognize the importance of a unified community to enable the ecosystem to grow and so are thrilled to back this effort."

About Databricks

Databricks was founded by the creators of Apache Spark, who have been working for the past six years on cutting-edge systems to analyze and process Big Data. They believe that Big Data is a tremendous opportunity that is still largely untapped, and are actively working to revolutionize what enterprises can do with it. Databricks is venture-backed by Andreessen Horowitz. For more information, visit http://www.databricks.com.

More Stories By Business Wire

Copyright © 2009 Business Wire. All rights reserved. Republication or redistribution of Business Wire content is expressly prohibited without the prior written consent of Business Wire. Business Wire shall not be liable for any errors or delays in the content, or for any actions taken in reliance thereon.

Latest Stories
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
In IT, we sometimes coin terms for things before we know exactly what they are and how they’ll be used. The resulting terms may capture a common set of aspirations and goals – as “cloud” did broadly for on-demand, self-service, and flexible computing. But such a term can also lump together diverse and even competing practices, technologies, and priorities to the point where important distinctions are glossed over and lost.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Personalization has long been the holy grail of marketing. Simply stated, communicate the most relevant offer to the right person and you will increase sales. To achieve this, you must understand the individual. Consequently, digital marketers developed many ways to gather and leverage customer information to deliver targeted experiences. In his session at @ThingsExpo, Lou Casal, Founder and Principal Consultant at Practicala, discussed how the Internet of Things (IoT) has accelerated our abilit...
As organizations shift towards IT-as-a-service models, the need for managing and protecting data residing across physical, virtual, and now cloud environments grows with it. Commvault can ensure protection, access and E-Discovery of your data – whether in a private cloud, a Service Provider delivered public cloud, or a hybrid cloud environment – across the heterogeneous enterprise. In his general session at 18th Cloud Expo, Randy De Meno, Chief Technologist - Windows Products and Microsoft Part...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessio...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
In his session at 20th Cloud Expo, Scott Davis, CTO of Embotics, discussed how automation can provide the dynamic management required to cost-effectively deliver microservices and container solutions at scale. He also discussed how flexible automation is the key to effectively bridging and seamlessly coordinating both IT and developer needs for component orchestration across disparate clouds – an increasingly important requirement at today’s multi-cloud enterprise.
In his session at @DevOpsSummit at 20th Cloud Expo, Kelly Looney, director of DevOps consulting for Skytap, showed how an incremental approach to introducing containers into complex, distributed applications results in modernization with less risk and more reward. He also shared the story of how Skytap used Docker to get out of the business of managing infrastructure, and into the business of delivering innovation and business value. Attendees learned how up-front planning allows for a clean sep...
In his session at 20th Cloud Expo, Brad Winett, Senior Technologist for DDN Storage, will present several current, end-user environments that are using object storage at scale for cloud deployments including private cloud and cloud providers. Details on the top considerations of features and functions for selecting object storage will be included. Brad will also touch on recent developments in tiering technologies that deliver single solution and an end-user view of data across files and objects...
In his session at @ThingsExpo, Arvind Radhakrishnen discussed how IoT offers new business models in banking and financial services organizations with the capability to revolutionize products, payments, channels, business processes and asset management built on strong architectural foundation. The following topics were covered: How IoT stands to impact various business parameters including customer experience, cost and risk management within BFS organizations.
Given the popularity of the containers, further investment in the telco/cable industry is needed to transition existing VM-based solutions to containerized cloud native deployments. The networking architecture of the solution isolates the network traffic into different network planes (e.g., management, control, and media). This naturally makes support for multiple interfaces in container orchestration engines an indispensable requirement.
"I will be talking about ChatOps and ChatOps as a way to solve some problems in the DevOps space," explained Himanshu Chhetri, CTO of Addteq, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.