Blog Feed Post

Log Management for IBM Bluemix and Cloud Foundry

Enterprises without DevOps teams and culture typically experience long applications deployment cycles. Once developers make a new release the operations team needs to deploy it to one of the existing or new servers, and it often takes a long time (up to several weeks is not uncommon) to get a new application deployed to production. This process may include hardware purchases, configuration, and troubleshooting sessions with the development team. Especially the troubleshooting portion might take a while, with the typical e-mail ping-pong with commands for the administrators, sysadmins sending log files to the developers… Modern digital businesses don’t operate that way any more.  Instead, modern DevOps teams often utilize PaaS to boost productivity, without even thinking of server resources, ansible/chef/puppet scripts and related stuff. They push their application releases with a few commands to “the cloud” – typically a platform as a service (PaaS) like IBM Bluemix! Once deployments get simplified in such a way, the next logical step is to disrupt the e-mail ping-pong for troubleshooting by introducing centralized log management. Logs are the most important source of information for application troubleshooting. Central log management is crucial for providing real-time log access to relevant teammates.  

In this post, we’ll show how to do this by deploying a Node.js app to IBM Bluemix and using Logsene for Log Management.

Let’s start with the deployment of a Node.js app on IBM Bluemix. First, we’ll clone a git repository.

> git clone hello-world-app
> cd hello-world-app

Next, because IBM Bluemix is based on Cloud Foundry, we’ll use a few commands to get access to IBM Bluemix:

> cf api https://api.ng.bluemix.net> cf login

When prompted for username and password enter your IBM ID (email) and password. The final step is to push the app in the current directory to IBM Bluemix:

> cf push app1

This takes a while and the command prints the deployment status to the console. Once your app starts successfully, it will be available via the displayed URL (orgSpace-appName), e.g.

Showing health and status for app hello-world-seti123 in org ORG1 / space seti321 as [email protected]…
requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: hello-world-seti123.mybluemix.net
last uploaded: Tue Dec 13 16:02:11 UTC 2016
stack: cflinuxfs2
buildpack: SDK for Node.js(TM) (ibm-node.js-4.6.2, buildpack-v3.9-20161128-1327)
     state     since                    cpu    memory         disk           details
#0   running   2016-12-13 05:03:27 PM   0.0%   114.9M of 1G   117.8M of 1G

Access the app via curl or the browser and check the most recent application logs in the IBM Bluemix user interface.

image02https://i0.wp.com/sematext.com/wp-content/uploads/2017/01/image02.png?re... 300w, https://i0.wp.com/sematext.com/wp-content/uploads/2017/01/image02.png?re... 768w, https://i0.wp.com/sematext.com/wp-content/uploads/2017/01/image02.png?re... 1024w, https://i0.wp.com/sematext.com/wp-content/uploads/2017/01/image02.png?w=... 1296w" sizes="(max-width: 648px) 100vw, 648px" data-recalc-dims="1" />

While at first you may think Bluemix provides log management service and wonder why you’d need to ship your logs from Bluemix to a Log Management SaaS, you will quickly realize that IBM Bluemix shows only the most recent logs. That, of course, is a no-go for production, where we need logs for at least the last few days, several weeks, or even archived for a longer period.

The Cloud Foundry documentation shows the option to forward logs via so called log drains from any Cloud Foundry app to centralized Log Management tools like Logsene. Here you can choose the retention time, save search queries, define alerts on logs, or visualize them in Kibana! The setup for this is done with another small set of  Cloud Foundry commands:

  1. Create a Logsene Application to obtain the Logsene Application token
  2. Create the log drain to your Logsene App (note the Logsene token in the URL):
    cf cups logsene-log-drain -l https://logsene-cf-receiver.sematext.com/1a4a748f-33c3-4e73-9e25-f3f0c82ff0f2
  3. Bind the log drain to your app. You could optionally bind multiple apps to the same log drain:
    cf bind-service YOUR-APP-NAME logsene-log-drain
  4. Re-stage the app to pick up the newly bound service:
    cf restage YOUR-APP-NAME

A few seconds later your logs will start arriving to Logsene and you will be able to search and visualize them in Logsene’s built-in Kibana:image01https://i1.wp.com/sematext.com/wp-content/uploads/2017/01/image01.png?re... 300w, https://i1.wp.com/sematext.com/wp-content/uploads/2017/01/image01.png?re... 768w, https://i1.wp.com/sematext.com/wp-content/uploads/2017/01/image01.png?re... 1024w, https://i1.wp.com/sematext.com/wp-content/uploads/2017/01/image01.png?w=... 1296w" sizes="(max-width: 648px) 100vw, 648px" data-recalc-dims="1" />

IBM Bluemix logs in Kibana

image00https://i2.wp.com/sematext.com/wp-content/uploads/2017/01/image00.png?re... 300w, https://i2.wp.com/sematext.com/wp-content/uploads/2017/01/image00.png?re... 768w" sizes="(max-width: 648px) 100vw, 648px" data-recalc-dims="1" />

Alert definition for logs in Logsene

Your logs are safe for the configured retention time, and if you need to archive your logs to Amazon S3, Logsene can archive your logs to S3 for you, too. Just another little time safer for modern DevOps teams!


Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
China Unicom exhibit at the 19th International Cloud Expo, which took place at the Santa Clara Convention Center in Santa Clara, CA, in November 2016. China United Network Communications Group Co. Ltd ("China Unicom") was officially established in 2009 on the basis of the merger of former China Netcom and former China Unicom. China Unicom mainly operates a full range of telecommunications services including mobile broadband (GSM, WCDMA, LTE FDD, TD-LTE), fixed-line broadband, ICT, data communica...
Whether you like it or not, DevOps is on track for a remarkable alliance with security. The SEC didn’t approve the merger. And your boss hasn’t heard anything about it. Yet, this unruly triumvirate will soon dominate and deliver DevSecOps faster, cheaper, better, and on an unprecedented scale. In his session at DevOps Summit, Frank Bunger, VP of Customer Success at ScriptRock, discussed how this cathartic moment will propel the DevOps movement from such stuff as dreams are made on to a practic...
In their Live Hack” presentation at 17th Cloud Expo, Stephen Coty and Paul Fletcher, Chief Security Evangelists at Alert Logic, provided the audience with a chance to see a live demonstration of the common tools cyber attackers use to attack cloud and traditional IT systems. This “Live Hack” used open source attack tools that are free and available for download by anybody. Attendees learned where to find and how to operate these tools for the purpose of testing their own IT infrastructure. The...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex softw...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
SYS-CON Events announced today that Cloud Academy will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud computing technologies. Ge...
Historically, some banking activities such as trading have been relying heavily on analytics and cutting edge algorithmic tools. The coming of age of powerful data analytics solutions combined with the development of intelligent algorithms have created new opportunities for financial institutions. In his session at 20th Cloud Expo, Sebastien Meunier, Head of Digital for North America at Chappuis Halder & Co., will discuss how these tools can be leveraged to develop a lasting competitive advanta...
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, will posit that disruption is inevitable for c...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.