Welcome!

News Feed Item

Databricks Expands Platform for Turnkey Production Apache Spark Deployments in the Cloud

Company Launches Enhanced Reliability and Security Capabilities for Data Engineering on its Managed Spark Platform

LAS VEGAS, NV -- (Marketwired) -- 11/30/16 -- Databricks®, the company founded by the team that created the popular Apache® Spark™ project, announced new capabilities to its platform that further simplify the production deployment of Spark in the cloud. The production enhancements complement the existing Databricks environment for data science, which enable users to collaboratively analyze data in real-time with data science notebooks and immediately deploy them as production Spark jobs and workflows. The announcement was made today at the 2016 Amazon Web Services (AWS) re:Invent conference.

The production features announced today enable users to effortlessly setup and run Spark jobs and workflows without humans in the loop via APIs, monitor performance and troubleshoot errors with detailed logs, manage AWS EC2 costs with AWS Tags, control access to resources with AWS IAM Roles, and increase the scalability of long-running workloads with encrypted AWS Elastic Block Storage (EBS). Databricks is the first and only vendor to offer a SOC2 and HIPAA compliant Spark platform that provides turnkey deployment of both real-time analysis and production Spark workloads with a seamless transition from analysis to production.

As organizations across industries deploy Apache Spark in the public cloud, the task of minimizing costly downtimes of mission-critical workloads, such as applications that predict equipment failure, falls on data engineering teams. Yet, building sophisticated systems around Spark to ensure that such workloads are resilient, easy to troubleshoot, and secure, requires a high level of technical expertise and meticulous efforts that most organizations struggle to spare.

"As enterprises increasingly rely on Apache Spark to power more diverse production workloads supporting more people, it becomes critical to prevent business system outages that could cost millions of dollars," said Nik Rouda, Senior Analyst at Enterprise Strategy Group.

In Databricks' production environment, data engineers can bypass the difficult and tedious tasks of developing, configuring, tuning and securing infrastructure to easily achieve production requirements with features such as:

  • HIPAA and SOC2-compliant Apache Spark clusters fully managed and tuned by the Spark committers at Databricks;
  • REST APIs to orchestrate and monitor sophisticated Spark jobs and workflows programmatically, without humans in the loop;
  • End-to-end logs and performance metrics to easily debug and fine-tune Spark workloads, accessible via APIs programmatically or in the Databricks user interface;
  • Customizable AWS tags to manage the AWS EC2 usage of each Spark cluster;
  • Encrypted AWS Elastic Block Storage (EBS) to increase the reliability of long-running Spark jobs on AWS EC2 instances by automatically providing additional storage;
  • AWS IAM Roles integration to provide secure access to AWS resources to diverse user groups in the same organization;
  • Direct integration with the data science environment to let organizations instantly move exploratory work to production without re-engineering;
  • SSH Access to provide engineers direct access to the production environment to troubleshoot and inspect the Spark clusters.

"Databricks is experiencing unprecedented demand for a robust and secure Apache Spark platform in the cloud to run production workloads," says Ali Ghodsi, CEO and Co-Founder of Databricks. "We are proud to enable one of our core user groups, the data engineers, to meet the most stringent of operational requirements."

Visit databricks.com or Booth #1341 at AWS re:Invent to learn more.

Contact Databricks to get started: http://go.databricks.com/contact-databricks.

About Databricks
Databricks' vision is to empower anyone to easily build and deploy advanced analytics solutions. The company was founded by the team who created Apache® Spark™, a powerful open source data processing engine built for sophisticated analytics, ease of use, and speed. Databricks is the largest contributor to the open source Apache Spark project. The company has also trained over 20,000 users on Apache Spark, and has the largest number of customers deploying Spark to date. Databricks provides a just-in-time data platform, to simplify data integration, real-time experimentation, and robust deployment of production applications. Databricks is venture-backed by Andreessen Horowitz and NEA. For more information, contact [email protected].

© Databricks 2016. All rights reserved. Apache, Apache Spark and Spark are trademarks of the Apache Software Foundation.

More Stories By Marketwired .

Copyright © 2009 Marketwired. All rights reserved. All the news releases provided by Marketwired are copyrighted. Any forms of copying other than an individual user's personal reference without express written permission is prohibited. Further distribution of these materials is strictly forbidden, including but not limited to, posting, emailing, faxing, archiving in a public database, redistributing via a computer network or in a printed form.

Latest Stories
Amazon started as an online bookseller 20 years ago. Since then, it has evolved into a technology juggernaut that has disrupted multiple markets and industries and touches many aspects of our lives. It is a relentless technology and business model innovator driving disruption throughout numerous ecosystems. Amazon’s AWS revenues alone are approaching $16B a year making it one of the largest IT companies in the world. With dominant offerings in Cloud, IoT, eCommerce, Big Data, AI, Digital Assista...
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists looked at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deliver...
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), provided an overview of various initiatives to certify the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldwide re...
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cl...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
Wooed by the promise of faster innovation, lower TCO, and greater agility, businesses of every shape and size have embraced the cloud at every layer of the IT stack – from apps to file sharing to infrastructure. The typical organization currently uses more than a dozen sanctioned cloud apps and will shift more than half of all workloads to the cloud by 2018. Such cloud investments have delivered measurable benefits. But they’ve also resulted in some unintended side-effects: complexity and risk. ...
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, discussed some of the security challenges of the IoT infrastructure and related how these aspects impact Smart Living. The material was delivered interac...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities. In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, posited that disruption is inevitable for comp...
It is ironic, but perhaps not unexpected, that many organizations who want the benefits of using an Agile approach to deliver software use a waterfall approach to adopting Agile practices: they form plans, they set milestones, and they measure progress by how many teams they have engaged. Old habits die hard, but like most waterfall software projects, most waterfall-style Agile adoption efforts fail to produce the results desired. The problem is that to get the results they want, they have to ch...
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...
"We are a monitoring company. We work with Salesforce, BBC, and quite a few other big logos. We basically provide monitoring for them, structure for their cloud services and we fit into the DevOps world" explained David Gildeh, Co-founder and CEO of Outlyer, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
In 2014, Amazon announced a new form of compute called Lambda. We didn't know it at the time, but this represented a fundamental shift in what we expect from cloud computing. Now, all of the major cloud computing vendors want to take part in this disruptive technology. In his session at 20th Cloud Expo, Doug Vanderweide, an instructor at Linux Academy, discussed why major players like AWS, Microsoft Azure, IBM Bluemix, and Google Cloud Platform are all trying to sidestep VMs and containers wit...
While DevOps most critically and famously fosters collaboration, communication, and integration through cultural change, culture is more of an output than an input. In order to actively drive cultural evolution, organizations must make substantial organizational and process changes, and adopt new technologies, to encourage a DevOps culture. Moderated by Andi Mann, panelists discussed how to balance these three pillars of DevOps, where to focus attention (and resources), where organizations might...
"When we talk about cloud without compromise what we're talking about is that when people think about 'I need the flexibility of the cloud' - it's the ability to create applications and run them in a cloud environment that's far more flexible,” explained Matthew Finnie, CTO of Interoute, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The Internet giants are fully embracing AI. All the services they offer to their customers are aimed at drawing a map of the world with the data they get. The AIs from these companies are used to build disruptive approaches that cannot be used by established enterprises, which are threatened by these disruptions. However, most leaders underestimate the effect this will have on their businesses. In his session at 21st Cloud Expo, Rene Buest, Director Market Research & Technology Evangelism at Ara...