Welcome!

News Feed Item

Fujitsu Develops Industry's First Integrated Development Platform for Big Data

Slashes data processing development time by 80%; enables integrated development for large-scale stored data analysis and complex event processing

Kawasaki, Japan, Aug 21, 2012 - (JCN Newswire) - Fujitsu Laboratories Limited today announced the development of the industry's first integrated big data development platform for processing large volumes of diverse time-series data.

In recent years, massive amounts of diverse data - as represented by sensor data, human location data, and other kinds of time-series data - continue to grow at an explosive pace. This has prompted the development of parallel batch processing technologies such as Hadoop(1), as well as complex event processing technologies(2) for processing data in real time. However, because each processing technology has employed different types of development and execution environments, it has been difficult to quickly apply insights gained from data analysis results to real-time processing applications. Moreover, maximizing the performance of Fujitsu's event processing engine(3) has required considerable knowledge about parallel application design, such as how to estimate network traffic.

By developing an integrated development platform able to handle description languages for both stored data analysis and complex event processing, Fujitsu was able to reduce development time for both batch and event processing by roughly 80% (from 8 weeks to 1.5 weeks) in a case study involving POS analysis-based coupon issuing. The platform is also equipped with a newly developed parallelism extraction function that automatically improves the processing efficiency of complex event processing. This function automatically extracts parallelism opportunities from event processing applications and recommends how to combine each analysis step in a way that optimizes execution plans without any extra effort.

This is one of the technologies that will be put to use to support human-centric computing, which will provide precisely targeted services anywhere.

Details of the new technology will be published at the IPSJ/SIGSE Software Engineering Symposium 2012 (SES2012), to be held from August 27 - 29, 2012.

Background

In recent years, massive amounts of diverse data - sensor data, human location data, and other kinds of time-series data - continue to grow at an explosive pace. There is a strong demand for taking this kind of "big data" and efficiently extracting valuable information that can be put to immediate use in delivering services, such as various navigation services.

Challenges

To process big data, Hadoop and other parallel batch processing technologies are deployed to analyze large volumes of stored data, as well as complex event processing technologies for processing event data in real time as it arrives. At the same time, these technologies are supported by different development and execution environments that, until now, have not been integrated. As a result, these disparate environments have made it difficult for analysts to quickly apply insights gained from data analysis results to complex event processing.

Furthermore, high-speed processing employing multiple servers in the cloud has proven to be crucial for performing complex event processing of large volumes of events. While Fujitsu has implemented a distributed event processing engine, its ability to raise performance by simply provisioning new servers, so as to take advantage of a cloud computing environment, relied on a complex application design phase that aims to identify effective ways to distribute each processing step.

Newly Developed Technology

Fujitsu has developed an integrated development platform that combines big data analysis and complex event processing. Using this platform, for example, companies can analyze the most up-to date purchasing trends from accumulated POS data and then hone in on a specific customer segment to issue coupons in real time, all as part of a simple process that does not require additional programming.

This technology consists of two parts: 1) A development platform integration feature that easily performs automatic program generation, regardless of development language; and 2) a parallelism extraction function that automatically improves the processing efficiency of complex event processing.

Features of the newly developed technology are as follows.

1. Development platform integration feature

After the processing details have been defined via data-flow diagrams and properties (corresponding to processing parameters), a proper set of patterns are selected and used to automatically generate either batch or real-time processing programs. During this generation phase, the operations produced by the selected templates are automatically supplemented with data conversion steps wherever necessary. The generated programs are finally deployed and executed on either a batch or real-time execution environment.

2. Parallelism extraction function for complex event processing

The parallelism extraction function extracts parallelism from real-time processing programs that have been automatically generated by the integrated development platform. The function will automatically recommend optimal combinations of parallelization schemes in order to decrease network traffic.

In real-time processing, incoming events can be distributed among multiple servers for parallel execution. For each processing step, various distribution schemes are applicable, and performance varies greatly depending on the scheme that is chosen. Generally speaking, a distribution scheme with finer granularity makes it easier to evenly distribute loads and facilitates better performance. For event processing, however, better performance is achieved by reducing network traffic. Here, traffic is minimized by the application of a uniform distribution scheme that tries to place inter-dependent processing steps on a same server whenever possible. By doing so, it avoids intermediary transfers and optimizes the overall application performance.

At runtime, the recommended distribution scheme will be used to select an optimal server allocation strategy in response to event volume fluctuations, resulting in better overall performance.

Results

1. Development platform integration feature

Using the new integrated development platform, as demonstrated during a Fujitsu case study, it was possible to shorten development time for both batch and event processing by approximately 80% (from 8 weeks to 1.5 weeks). Moreover, because parameters for each kind of processing can be easily modified without additional programming, trial-and-error tests on the development platform can be easily performed, such as for quickly applying insights gained from data analysis results to event search criteria.

2. Parallelism extraction function for complex event processing

The new parallelism extraction function generates executable programs that are specifically adapted for dynamic load balancing. Those programs can be easily scaled-out or down without having to re-compile the original application.

In addition, after measuring the performance of sample programs with different event distribution schemes, Fujitsu Laboratories confirmed that a uniform distribution scheme, by placing inter-dependent processing steps onto a same server, is able to reduce communications traffic by 60% and achieve a 3.5x improvement in processing efficiency compared to isolated distribution schemes that distribute each processing step independently.

Future Developments

Going forward, Fujitsu plans to further expand the features of the new technology while aiming to commercialize it in the company's platforms and middleware for big data by fiscal year 2013. Fujitsu will also explore deploying the technology in a wide range of applications, such as its services and products, in order to enable the utilization of valuable information generated through the process of collecting, accumulating and analyzing large volumes of sensor data.

For details please see http://www.fujitsu.com/global/news/pr/archives/month/2012/20120820-03.html.

(1) Parallel batch processing technologies such as Hadoop:A technique in which massive data sets are converted to batches, which are processed in parallel. Developed and released by the Apache Software Foundation (ASF), Hadoop is an open-source framework for efficiently performing distributed parallel processing of massive volumes of data.
(2)Complex event processing (CEP):A method of extracting valuable information from a stream of big data in real time. By processing data in memory in accordance with pre-defined rules (described by EPL), the data can be processed in real time. Event Processing Language (EPL) is a description language for describing the content of complex event processes.
(3) Event processing engine:PRESS RELEASE "Fujitsu Develops Distributed and Parallel Complex Event Processing Technology that Rapidly Adjusts Big Data Load Fluctuations" (December 16, 2011)


About Fujitsu Laboratories

Founded in 1968 as a wholly owned subsidiary of Fujitsu Limited, Fujitsu Laboratories Limited is one of the premier research centers in the world. With a global network of laboratories in Japan, China, the United States and Europe, the organization conducts a wide range of basic and applied research in the areas of Next-generation Services, Computer Servers, Networks, Electronic Devices and Advanced Materials. For more information, please see: http://jp.fujitsu.com/labs/en.

About Fujitsu Limited

Fujitsu is the leading Japanese information and communication technology (ICT) company offering a full range of technology products, solutions and services. Over 170,000 Fujitsu people support customers in more than 100 countries. We use our experience and the power of ICT to shape the future of society with our customers. Fujitsu Limited (TSE:6702) reported consolidated revenues of 4.5 trillion yen (US$54 billion) for the fiscal year ended March 31, 2012. For more information, please see www.fujitsu.com.



Source: Fujitsu Limited

Contact:
Fujitsu Limited
Public and Investor Relations
www.fujitsu.com/global/news/contacts/
+81-3-3215-5259


Copyright 2012 JCN Newswire. All rights reserved. www.japancorp.net

More Stories By JCN Newswire

Copyright 2008 JCN Newswire. All rights reserved. Republication or redistribution of JCN Newswire content is expressly prohibited without the prior written consent of JCN Newswire. JCN Newswire shall not be liable for any errors or delays in the content, or for any actions taken in reliance thereon.

Latest Stories
SYS-CON Events announced today that Elastifile will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Elastifile Cloud File System (ECFS) is software-defined data infrastructure designed for seamless and efficient management of dynamic workloads across heterogeneous environments. Elastifile provides the architecture needed to optimize your hybrid cloud environment, by facilitating efficient...
“Why didn’t testing catch this” must become “How did this make it to testing?” Traditional quality teams are the crutch and excuse keeping organizations from making the necessary investment in people, process, and technology to accelerate test automation. Just like societies that did not build waterways because the labor to keep carrying the water was so cheap, we have created disincentives to automate. In her session at @DevOpsSummit at 20th Cloud Expo, Anne Hungate, President of Daring System...
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
Most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes a lot of work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reduction in cost ...
"Cloud computing is certainly changing how people consume storage, how they use it, and what they use it for. It's also making people rethink how they architect their environment," stated Brad Winett, Senior Technologist for DDN Storage, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We provide IoT solutions. We provide the most compatible solutions for many applications. Our solutions are industry agnostic and also protocol agnostic," explained Richard Han, Head of Sales and Marketing and Engineering at Systena America, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that GrapeUp, the leading provider of rapid product development at the speed of business, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market acr...
@DevOpsSummit at Cloud Expo taking place Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center, Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is ...
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taug...
Recently, IoT seems emerging as a solution vehicle for data analytics on real-world scenarios from setting a room temperature setting to predicting a component failure of an aircraft. Compared with developing an application or deploying a cloud service, is an IoT solution unique? If so, how? How does a typical IoT solution architecture consist? And what are the essential components and how are they relevant to each other? How does the security play out? What are the best practices in formulating...
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
Cloud adoption is often driven by a desire to increase efficiency, boost agility and save money. All too often, however, the reality involves unpredictable cost spikes and lack of oversight due to resource limitations. In his session at 20th Cloud Expo, Joe Kinsella, CTO and Founder of CloudHealth Technologies, tackled the question: “How do you build a fully optimized cloud?” He will examine: Why TCO is critical to achieving cloud success – and why attendees should be thinking holistically ab...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, will introduce two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a...
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud com...
In his session at @ThingsExpo, Arvind Radhakrishnen discussed how IoT offers new business models in banking and financial services organizations with the capability to revolutionize products, payments, channels, business processes and asset management built on strong architectural foundation. The following topics were covered: How IoT stands to impact various business parameters including customer experience, cost and risk management within BFS organizations.