Welcome!

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, Linux Containers, Containers Expo Blog, @CloudExpo

@DXWorldExpo: Blog Post

In-Memory Computing By @GridGain | @CloudExpo [#BigData]

The best way to clear the air around IMC is to start with a simple explanation of what, in fact, in-memory computing is

The Facts and Fiction of In-Memory Computing

In the last year, conversations about In-Memory Computing (IMC) have become more and more prevalent in enterprise IT circles, especially with organizations feeling the pressure to process massive quantities of data at the speed that is now being demanded by the Internet. The hype around IMC is justified: tasks that once took hours to execute are streamlined down to seconds by moving the computation and data from disk, directly to RAM. Through this simple adjustment, analytics are happening in real-time, and applications (as well as the development of applications) are working at-pace with this new standard of technology and speed.

Despite becoming both more cost-effective and accepted within enterprise computing, there are still a small handful of falsehoods that confuse even the most technical of individuals in enterprise IT.

Myth: In-memory computing is about databases, so this isn't really relevant to my business.

The best way to clear the air around IMC is to start with a simple explanation of what, in fact, in-memory computing is. While many assume that because we are talking about RAM, we are having a conversation about databases and storage, but this is not the case.

IMC, at its most basic level, is using a middleware software that allows one to store data in RAM - across a broad cluster of computers - and do any and all processing where it resides (in the memory). With traditional methods, data processing is often confined to spinning disks.

By comparison, in-memory computing speeds up this process by roughly 5,000 times. Now you can see that we're not talking about storage only - instead active, fluid data and computing.

Which brings me to another, more tangible point about computing efficiency. By incorporating in-memory, a handful of processes are streamlined in order to save time, resources, and money.

To start, in-memory requires much less hardware; the result - significantly decreased capital, operational and infrastructure overhead.

Moreover, IT departments can also significantly extend the life of existing hardware and software through the increased performance that is inherent with IMC - thus amplifying the ROI on the machines that have already been purchased.

Surprisingly, in-memory computing is not a new phenomenon. Since the inception of RAM, IMC has been viewed as reliable accelerant for high-performance computing, bringing us to the next crucial misconception about this technology.

Myth: In-memory computing is expensive, therefore not practical for my operation.

There is a reason that this is one of the most common misunderstandings about IMC, because there was a point in time where the cost of memory was once quite high. That being said, the cost of RAM has been dropping consistently, at a rate of about 30% - for the last 5 years.

Today, the price of a 1 Terabyte RAM cluster can go for anywhere between $20 and $40 thousand - including all of the CPUs, networking, etc. A few years from now that same setup will likely be available for half that price.

Regardless of the future price of RAM, which based upon current projections will likely continue to fall, the current economics have already placed this technology well within the reaches of the enterprise computing budgets that require this level of scale.

Myth: My needs are already being met by Flash.

There are three different reasons why this mentality is held by IT folks, each of which are highly misinformed. I'll start with the most common, which is the idea that your business doesn't need the Lambourgini-esque super-computing power of IMC.

The hard yet obvious reality is that if your business is in any way data-driven, you likely cannot survive without speed and agility in this department. As time goes on, the amount of data that businesses accumulate compounds with new streams and variances. This is a sink-or-swim reality.

Another myth commonly used to dispel IMC is that if businesses are able to just effectively mount RAM disk, they will get in-memory processing. Unfortunately, it's not that easy. As mentioned earlier, IMC works through middleware to effectively unlock its power.

Finally, there's the notion that one can just replace their HDDs with SSDs in order to get this super-charged performance. For SSDs - in certain situations - the performance gain that you can pull from flash storage in lieu of spinning disk is enough.

However, speed matters - and is rapidly becoming more of a requirement every day. At this point, it's like comparing apples to oranges with speed improvements of 10 to 100x over SSDs.

Myth: Memory is not durable enough to be truly sustainable.

This is another notion that for whatever reason has been both widely perpetuated - and is entirely false.

The fact is - almost all in-memory computing middleware (apart from very simplistic ones) offer one or multiple strategies for in-memory backups, durable storage backups, disk-based swap space overflow, etc.

More sophisticated vendors provide a comprehensive tiered storage approach where users can decide what portion of the overall data set is stored in RAM, local disk swap space or RDBMS/HDFS - where each tier can store progressively more data but with progressively longer latencies.

Yet another source of confusion is the difference between operational datasets and historical datasets. In-memory computing is not aimed at replacing enterprise data warehouse (EDW), backup or offline storage services - like Hadoop, for example. The goal of IMC is to improve the operational datasets that require mixed OLTP and OLAP processing and in most cases are less than 10TB in size. That is to say, in-memory computing is not "all or nothing" - and does not require that every aspect of data be housed in memory.

The in-memory computing revolution is by no means intended to obliterate disks from the enterprise. For now, the disk still serves a well-defined role for offline/backup use cases - tasks that are not the focus of IMC.

Myth: In-memory is inaccessible to my business because so few developers actually know how to use it.

Yes indeed, In-memory computing is a highly complex technology, that for now, only a few vendors have even been able to successfully develop offerings for. However, like much of high-technology, in-memory computing has entered the world of open source - bringing its capabilities and power to the fingertips of developers around the world.

Currently, with GridGain, developers have the ability to get their hands on IMC with a simple download at http://gridgain.org/.

In-memory computing is already being tapped across a broad range of functions and industries including (but not limited to) financial trading systems, online game, bioinformatics, hyper-local advertising, cognitive computing, and geospatial analysis.

By raising awareness, and bringing the capabilities of IMC to more developers and organizations - industries around the globe are poised to experience entirely new standards of speed, computing, and performance.

More Stories By Nikita Ivanov

Nikita Ivanov is founder and CEO of GridGain Systems, started in 2007 and funded by RTP Ventures and Almaz Capital. Nikita has led GridGain to develop advanced and distributed in-memory data processing technologies – the top Java in-memory computing platform starting every 10 seconds around the world today.

Nikita has over 20 years of experience in software application development, building HPC and middleware platforms, contributing to the efforts of other startups and notable companies including Adaptec, Visa and BEA Systems. Nikita was one of the pioneers in using Java technology for server side middleware development while working for one of Europe’s largest system integrators in 1996.

He is an active member of Java middleware community, contributor to the Java specification, and holds a Master’s degree in Electro Mechanics from Baltic State Technical University, Saint Petersburg, Russia.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Latest Stories
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settle...
The use of containers by developers -- and now increasingly IT operators -- has grown from infatuation to deep and abiding love. But as with any long-term affair, the honeymoon soon leads to needing to live well together ... and maybe even getting some relationship help along the way. And so it goes with container orchestration and automation solutions, which are rapidly emerging as the means to maintain the bliss between rapid container adoption and broad container use among multiple cloud host...
Blockchain is a shared, secure record of exchange that establishes trust, accountability and transparency across business networks. Supported by the Linux Foundation's open source, open-standards based Hyperledger Project, Blockchain has the potential to improve regulatory compliance, reduce cost as well as advance trade. Are you curious about how Blockchain is built for business? In her session at 21st Cloud Expo, René Bostic, Technical VP of the IBM Cloud Unit in North America, discussed the b...
Blockchain. A day doesn’t seem to go by without seeing articles and discussions about the technology. According to PwC executive Seamus Cushley, approximately $1.4B has been invested in blockchain just last year. In Gartner’s recent hype cycle for emerging technologies, blockchain is approaching the peak. It is considered by Gartner as one of the ‘Key platform-enabling technologies to track.’ While there is a lot of ‘hype vs reality’ discussions going on, there is no arguing that blockchain is b...
Imagine if you will, a retail floor so densely packed with sensors that they can pick up the movements of insects scurrying across a store aisle. Or a component of a piece of factory equipment so well-instrumented that its digital twin provides resolution down to the micrometer.
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
ChatOps is an emerging topic that has led to the wide availability of integrations between group chat and various other tools/platforms. Currently, HipChat is an extremely powerful collaboration platform due to the various ChatOps integrations that are available. However, DevOps automation can involve orchestration and complex workflows. In his session at @DevOpsSummit at 20th Cloud Expo, Himanshu Chhetri, CTO at Addteq, will cover practical examples and use cases such as self-provisioning infra...
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, discussed how data centers of the future will be managed, how the p...
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory? In her Day 2 Keynote at @DevOpsSummit at 21st Cloud Expo, Aruna Ravichandran, VP, DevOps Solutions Marketing, CA Technologies, was jo...
"Since we launched LinuxONE we learned a lot from our customers. More than anything what they responded to were some very unique security capabilities that we have," explained Mark Figley, Director of LinuxONE Offerings at IBM, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
The cloud era has reached the stage where it is no longer a question of whether a company should migrate, but when. Enterprises have embraced the outsourcing of where their various applications are stored and who manages them, saving significant investment along the way. Plus, the cloud has become a defining competitive edge. Companies that fail to successfully adapt risk failure. The media, of course, continues to extol the virtues of the cloud, including how easy it is to get there. Migrating...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...