Welcome!

Related Topics: Microservices Expo, @CloudExpo

Microservices Expo: Blog Feed Post

Importance of ILM and Data Archiving

How do you manage data over its lifetime of many years and decades?

With the noise of cloud computing rising by the day, there are basic operational issues one should not forget – cloud or no cloud. One such issue is the discipline of ILM (Information Life-cycle Management). How do you manage data over its lifetime of many years and decades? Do you keep all data current which drastically impacts the performance of applications using them? As everyone knows the appetite for data is growing by leaps and bounds. Not far from now, “personal petabyte” is quite viable given the need to store audio and video stuff. A petabyte is one thousand terabytes which is 1000 gigabytes which is 1000 megabytes. Now do the math. A petabyte is ten to the power of 15 bytes. And 1000 petabytes is one “exabyte”. Back in 2002, one petabyte would have cost  $2M, whereas in 2012 (ten years) its cost will be $2K. This is real Moore’s law in disk storage!

Most of enterprise business data is resident as structured data managed by DBMS (e.g. Oracle or DB2). There are production databases of the size of 100 plus terabytes , mostly in places such as Walmart’s data warehouse for retail transactions. Telcos also have huge databases for call records. With the growth in size, performance degradation is normal. Hence enterprises must create a multi-tiered archiving policy. For example, current data can be in active databases for 2-3 years, followed by 2-4 years of inactive data followed by several years of historical data. As we move further behind, such data can be part of cloud storage. But access is paramount even if data is stored in multiple levels. For compliance and legal reasons, historical data should be easily accessible at high speed with smart search.

Another aspect of ILM is management of copies of data. Some companies may need 8-20 copies of active data for test, development, disaster recovery, quality control, etc.  A 200 GB database may end up like 1200 GBs of data with six copies. Such issues are normally not reflected as part of planning, but IT shops get shocked when they see such numbers and the associated cost factors. Anther area at many enterprises is the “application retirement” issue. This happens with M&A or as a precursor to move into the cloud. This area is addressed in a very adhoc way resulting in unforeseen delays and cost. Any automation here should be highly welcome.

Gartner Group said this last year, “The return on the investment for implementing a structured data archiving solution is exceptionally high, especially for application retirement or when deployed for a packaged application for which vendor-supplied templates are available to ease implementation and maintenance.”

One company (I am an adviser) leading in this space is Solix that provides all the tools mentioned above. Their Enterprise Data Management System (EDMS) platform provides a comprehensive set of ILM tools for  enterprises. Solix even introduced an appliance to ease the cost and administrative burdens for clients. The rapid adoption of Solix products is a testimony to the growing importance of data archiving, application retirement, data masking, and test data management.

ILM should be a well-thought-out discipline at every IT organization.

Read the original blog entry...

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

Latest Stories
Although it has gained significant traction in the consumer space, IoT is still in the early stages of adoption in enterprises environments. However, many companies are working on initiatives like Industry 4.0 that includes IoT as one of the key disruptive technologies expected to reshape businesses of tomorrow. The key challenges will be availability, robustness and reliability of networks that connect devices in a business environment. Software Defined Wide Area Network (SD-WAN) is expected to...
Between the mockups and specs produced by analysts, and resulting applications built by developers, there exists a gulf where projects fail, costs spiral, and applications disappoint. Methodologies like Agile attempt to address this with intensified communication, with partial success but many limitations. In his session at @DevOpsSummit at 19th Cloud Expo, Charles Kendrick, CTO at Isomorphic Software, will present a revolutionary model enabled by new technologies. Learn how business and deve...
DevOps at Cloud Expo – being held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Am...
Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of (at least) three separate application components: the software embedded in the device, the back-end service, and the mobile application for the end user’s controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target –...
SYS-CON Events announced today that Hitrons Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Hitrons Solutions Inc. is distributor in the North American market for unique products and services of small and medium-size businesses, including cloud services and solutions, SEO marketing platforms, and mobile applications.
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - comp...
SYS-CON Events announced today that Numerex Corp, a leading provider of managed enterprise solutions enabling the Internet of Things (IoT), will exhibit at the 19th International Cloud Expo | @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Numerex Corp. (NASDAQ:NMRX) is a leading provider of managed enterprise solutions enabling the Internet of Things (IoT). The Company's solutions produce new revenue streams or create operating...
Enterprises have forever faced challenges surrounding the sharing of their intellectual property. Emerging cloud adoption has made it more compelling for enterprises to digitize their content, making them available over a wide variety of devices across the Internet. In his session at 19th Cloud Expo, Santosh Ahuja, Director of Architecture at Impiger Technologies, will introduce various mechanisms provided by cloud service providers today to manage and share digital content in a secure manner....
We are always online. We access our data, our finances, work, and various services on the Internet. But we live in a congested world of information in which the roads were built two decades ago. The quest for better, faster Internet routing has been around for a decade, but nobody solved this problem. We’ve seen band aid approaches like CDNs that attack a niche's slice of static content part of the Internet, but that’s it. It does not address the dynamic services-based Internet of today. It doe...
As the world moves toward more DevOps and Microservices, application deployment to the cloud ought to become a lot simpler. The Microservices architecture, which is the basis of many new age distributed systems such as OpenStack, NetFlix and so on, is at the heart of Cloud Foundry - a complete developer-oriented Platform as a Service (PaaS) that is IaaS agnostic and supports vCloud, OpenStack and AWS. Serverless computing is revolutionizing computing. In his session at 19th Cloud Expo, Raghav...
While DevOps promises a better and tighter integration among an organization’s development and operation teams and transforms an application life cycle into a continual deployment, Chef and Azure together provides a speedy, cost-effective and highly scalable vehicle for realizing the business values of this transformation. In his session at @DevOpsSummit at 19th Cloud Expo, Yung Chou, a Technology Evangelist at Microsoft, will present a unique opportunity to witness how Chef and Azure work tog...
As cloud adoption continues to transform business, today’s global enterprises are challenged with managing a growing amount of information living outside of the data center. The rapid adoption of IoT and increasingly mobile workforce are exacerbating the problem. Ensuring secure data sharing and efficient backup poses capacity and bandwidth considerations as well as policy and regulatory compliance issues.
Why do your mobile transformations need to happen today? Mobile is the strategy that enterprise transformation centers on to drive customer engagement. In his general session at @ThingsExpo, Roger Woods, Director, Mobile Product & Strategy – Adobe Marketing Cloud, covered key IoT and mobile trends that are forcing mobile transformation, key components of a solid mobile strategy and explored how brands are effectively driving mobile change throughout the enterprise.
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, will discuss the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
The Internet of Things can drive efficiency for airlines and airports. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Sudip Majumder, senior director of development at Oracle, will discuss the technical details of the connected airline baggage and related social media solutions. These IoT applications will enhance travelers' journey experience and drive efficiency for the airlines and the airports. The session will include a working demo and a technical d...