News Feed Item
Databricks Launches “Certified Spark Distribution” Program to Recognize Vendors Committed to Supporting the Apache Spark Application Ecosystem
|By Business Wire
|June 26, 2014 12:11 PM EDT
Databricks, the company founded by the creators of Apache Spark, the
next generation Big Data engine, today announced the “Certified Spark
Distribution” program for vendors with a commercial Spark distribution.
Certification indicates that the vendor’s Spark distribution is
compatible with the open source Apache Spark distribution, enabling
“Certified on Spark” applications - certified to work with Apache Spark
- to run on the vendor’s Spark distribution out-of-the-box.
“One of Databricks’ goals is to ensure users have a fantastic
experience. Our belief is that having the community work together to
maintain compatibility and therefore facilitate a vibrant application
ecosystem is crucial to this vision,” said Ion Stoica, Databricks CEO.
“We first launched the ‘Certified on Spark’ program to help build a
robust ecosystem of innovative applications on top of Apache Spark. The
‘Certified Spark Distribution’ program is the other half of the
equation, recognizing vendors that are committed to providing a home for
these applications to allow the ecosystem to flourish.”
In keeping with the open source nature of Spark, the certification
process is fully transparent with open-source tests, lightweight, and
100% free - a mirror image of the “Certified on Spark” process for Spark
applications. Vendors fill out a short questionnaire and then simply
execute a set of open-source tests - developed and maintained by the
community and used to test each release of Apache Spark - against their
build of Spark to demonstrate compatibility.
“Certification shouldn’t be used as a tool for lock-in: Certified Spark
Distributions are not required to ship all the bits of Apache Spark, or
be open source, or prevented from innovating significantly within and
around Spark,” said Arsalan Tavakoli-Shiraji, Business Development Lead
at Databricks. “They simply need to maintain compatibility with Apache
Spark to provide support for the application ecosystem.”
As part of the certification program launch, five vendors have completed
the certification process: DataStax,
Oracle, and Pivotal
- industry leaders that have recognized and embraced the power of Spark
when integrated with their respective platforms. Each of these vendors
put their distributions through the certification process, which
included a host of integration tests to ensure full compatibility with
the latest Apache Spark release.
“One of the big risks faced by open source projects is fragmentation
among distributors. Fragmentation is bad for both users and application
developers, and ultimately for the growth of the project,” said Matei
Zaharia, Databricks CTO and VP of the Spark project at Apache. “We are
delighted that these partners - along with others in the certification
pipeline - share our vision of an undivided Spark platform based
directly around Apache, and will ensure that all applications built on
Apache Spark run on their distributions.”
Vendors interested in certifying their Spark distribution should visit www.databricks.com
and select "Apply for Certification." Enterprise users can also visit
the Databricks site regularly to see the latest set of certified
distributions and applications, and read “spotlight” blog articles that
provide deep-dives on the Spark ecosystem by newly certified vendors.
All the inaugural members will be on hand at the upcoming Spark
Summit from June 30th to July 2nd in San Francisco to provide
greater information on the role of Spark in helping better serve their
customers. Additionally, there will be an “Application
Spotlight” segment that will highlight innovative “Certified on
"DataStax is strongly committed to making Cassandra and Spark the best
combination for today's online applications," said Robin Schumacher, VP
of products at DataStax. "We have demonstrated that commitment with the
integration work we have contributed back to both open source
communities as well as the certified versions of Spark and Cassandra we
provide in DataStax Enterprise for production environments."
“We support the fact that Apache Spark project provides enterprises with
an additional processing engine in Hadoop to execute in-memory
algorithms for advanced analytics,” said John Kreisa, vice president of
strategic marketing at Hortonworks. “We applaud Databricks’ vision to
ensure Spark is fully integrated on YARN, which enterprises have adopted
as the data OS for Hadoop.”
"Pivotal's open source credentials are quite extensive -
Apache-compatible Hadoop, MADLib, RabbitMQ, CloudFoundry - and now we've
added Spark to that set," said Sarabjeet Chugh, Head of Hadoop Product
Management at Pivotal. "Additionally, we recognize the importance of a
unified community to enable the ecosystem to grow and so are thrilled to
back this effort."
Databricks was founded by the creators of Apache Spark, who have been
working for the past six years on cutting-edge systems to analyze and
process Big Data. They believe that Big Data is a tremendous opportunity
that is still largely untapped, and are actively working to
revolutionize what enterprises can do with it. Databricks is
venture-backed by Andreessen Horowitz. For more information, visit http://www.databricks.com.
For IoT to grow as quickly as analyst firms’ project, a lot is going to fall on developers to quickly bring applications to market. But the lack of a standard development platform threatens to slow growth and make application development more time consuming and costly, much like we’ve seen in the mobile space.
In his session at @ThingsExpo, Mike Weiner, Product Manager of the Omega DevCloud with KORE Telematics Inc., discussed the evolving requirements for developers as IoT matures and conducte...
Jul. 29, 2015 07:30 AM EDT Reads: 229
SYS-CON Events announced today that MobiDev, a software development company, will exhibit at the 17th International Cloud Expo®, which will take place November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
MobiDev is a software development company with representative offices in Atlanta (US), Sheffield (UK) and Würzburg (Germany); and development centers in Ukraine. Since 2009 it has grown from a small group of passionate engineers and business managers to a full-scale mobi...
Jul. 29, 2015 07:00 AM EDT Reads: 174
One of the hottest areas in cloud right now is DRaaS and related offerings.
In his session at 16th Cloud Expo, Dale Levesque, Disaster Recovery Product Manager with Windstream's Cloud and Data Center Marketing team, will discuss the benefits of the cloud model, which far outweigh the traditional approach, and how enterprises need to ensure that their needs are properly being met.
Jul. 29, 2015 06:00 AM EDT Reads: 1,661
The time is ripe for high speed resilient software defined storage solutions with unlimited scalability. ISS has been working with the leading open source projects and developed a commercial high performance solution that is able to grow forever without performance limitations.
In his session at Cloud Expo, Alex Gorbachev, President of Intelligent Systems Services Inc., shared foundation principles of Ceph architecture, as well as the design to deliver this storage to traditional SAN storage co...
Jul. 29, 2015 04:00 AM EDT Reads: 1,721
The speed of software changes in growing and large scale rapid-paced DevOps environments presents a challenge for continuous testing. Many organizations struggle to get this right. Practices that work for small scale continuous testing may not be sufficient as the requirements grow.
In his session at DevOps Summit, Marc Hornbeek, Sr. Solutions Architect of DevOps continuous test solutions at Spirent Communications, explained the best practices of continuous testing at high scale, which is rele...
Jul. 28, 2015 11:00 PM EDT Reads: 1,352
"We got started as search consultants. On the services side of the business we have help organizations save time and save money when they hit issues that everyone more or less hits when their data grows," noted Otis Gospodnetić, Founder of Sematext, in this SYS-CON.tv interview at @DevOpsSummit, held June 9-11, 2015, at the Javits Center in New York City.
Jul. 28, 2015 10:45 PM EDT Reads: 1,006
"We've just seen a huge influx of new partners coming into our ecosystem, and partners building unique offerings on top of our API set," explained Seth Bostock, Chief Executive Officer at IndependenceIT, in this SYS-CON.tv interview at 16th Cloud Expo, held June 9-11, 2015, at the Javits Center in New York City.
Jul. 28, 2015 08:00 PM EDT Reads: 584
Learn how to solve the problem of keeping files in sync between multiple Docker containers.
In his session at 16th Cloud Expo, Aaron Brongersma, Senior Infrastructure Engineer at Modulus, discussed using rsync, GlusterFS, EBS and Bit Torrent Sync. He broke down the tools that are needed to help create a seamless user experience.
In the end, can we have an environment where we can easily move Docker containers, servers, and volumes without impacting our applications? He shared his results so yo...
Jul. 28, 2015 07:15 PM EDT Reads: 715
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy.
How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Arch...
Jul. 28, 2015 06:30 PM EDT Reads: 1,371
Chuck Piluso presented a study of cloud adoption trends and the power and flexibility of IBM Power and Pureflex cloud solutions.
Prior to Secure Infrastructure and Services, Mr. Piluso founded North American Telecommunication Corporation, a facilities-based Competitive Local Exchange Carrier licensed by the Public Service Commission in 10 states, serving as the company's chairman and president from 1997 to 2000.
Between 1990 and 1997, Mr. Piluso served as chairman & founder of International Te...
Jul. 28, 2015 05:30 PM EDT Reads: 253
It is one thing to build single industrial IoT applications, but what will it take to build the Smart Cities and truly society-changing applications of the future? The technology won’t be the problem, it will be the number of parties that need to work together and be aligned in their motivation to succeed.
In his session at @ThingsExpo, Jason Mondanaro, Director, Product Management at Metanga, discussed how you can plan to cooperate, partner, and form lasting all-star teams to change the world...
Jul. 28, 2015 04:30 PM EDT Reads: 1,752
Public Cloud IaaS started its life in the developer and startup communities and has grown rapidly to a $20B+ industry, but it still pales in comparison to how much is spent worldwide on IT: $3.6 trillion. In fact, there are 8.6 million data centers worldwide, the reality is many small and medium sized business have server closets and colocation footprints filled with servers and storage gear. While on-premise environment virtualization may have peaked at 75%, the Public Cloud has lagged in adop...
Jul. 28, 2015 04:00 PM EDT Reads: 2,183
The Internet of Everything (IoE) brings together people, process, data and things to make networked connections more relevant and valuable than ever before – transforming information into knowledge and knowledge into wisdom. IoE creates new capabilities, richer experiences, and unprecedented opportunities to improve business and government operations, decision making and mission support capabilities.
Jul. 28, 2015 04:00 PM EDT Reads: 217
How do you securely enable access to your applications in AWS without exposing any attack surfaces? The answer is usually very complicated because application environments morph over time in response to growing requirements from your employee base, your partners and your customers.
In his session at @DevOpsSummit, Haseeb Budhani, CEO and Co-founder of Soha, shared five common approaches that DevOps teams follow to secure access to applications deployed in AWS, Azure, etc., and the friction an...
Jul. 28, 2015 03:30 PM EDT Reads: 486
Digital Transformation is the ultimate goal of cloud computing and related initiatives. The phrase is certainly not a precise one, and as subject to hand-waving and distortion as any high-falutin' terminology in the world of information technology.
Yet it is an excellent choice of words to describe what enterprise IT—and by extension, organizations in general—should be working to achieve.
Digital Transformation means:
handling all the data types being found and created in the organizat...
Jul. 28, 2015 03:00 PM EDT Reads: 1,062