Welcome!

Blog Feed Post

How to Detect Malicious Traffic in Your Server Logs: Sematext + Access Watch

Do you know what portion of your traffic comes from bots? Do you know which bots are good and which ones are bad? Do you know what the bad bots are up to?

We all know Googlebot and consider it a good bot. It crawls your site and makes it possible for others to find you via Google. Great! What do you know about Mars? No, not the red planet – Mars is one of the top bad bots that’s probably going through your site right now. What about all the other bad bots? In the simplest case, they could be consuming your bandwidth, using up your server resources, and increasing your monthly bill. They could also be stealing your content, modifying your pages to include viruses, or performing other malicious acts.

But while some bad bots are easy to spot, there are a lot more bad bots pretending to be regular human-controlled browsers, making them very hard to detect. This is where Access Watch comes into play. Access Watch, a startup from Berlin, deploys the industry’s most precise robot intelligence that can be easily plugged into any existing data pipeline, such as those handling web server logs.

Web server logs typically contain the following information:

–       IP Address of the client

–       Request URL

–       User agent

–       HTTP protocol version

–       HTTP method etc.

Web server logs are often analyzed purely for getting web access statistics – most popular pages, top countries, etc. Sometimes web server logs are enriched with GeoIP information to get a bit more information about visitors. Using a threat intelligence databases we could figure out a lot more about our “visitors”, some of which are not visitors at all, but malicious bots. Some IP addresses are known to spread viruses or are abused to execute hacker attacks. Many attacks have a typical fingerprint – a combination of URL, header fields, user agent and IP address from a blacklisted server. The relevant information changes frequently and accurate classification requires real-time access to a threat intelligence database. Access Watch REVEAL is able to identify good and malicious web traffic and provides this information via HTTP API.

Enriched web server log with request reputationhttps://sematext.com/wp-content/uploads/2017/08/enriched-web-server-log-... 300w, https://sematext.com/wp-content/uploads/2017/08/enriched-web-server-log-... 768w" sizes="(max-width: 880px) 100vw, 880px" />

Enriched web server log with request reputation and threat analysis from Access Watch API call.

All we need to do to get accurate threat intelligence information is call the Access Watch API with information from our web server logs, and then store the enriched web server log to visualize and analyze the malicious traffic. To make this super simple, Logagent users can use the new Access Watch plugin to perform security and traffic analysis, store that in Sematext Cloud or any place else (e.g. their own Elasticsearch cluster) for further analysis, visualization, etc.

Sematext Visualisation of bot traffic with bad reputationhttps://sematext.com/wp-content/uploads/2017/08/Sematext-Visualisation-o... 300w, https://sematext.com/wp-content/uploads/2017/08/Sematext-Visualisation-o... 768w" sizes="(max-width: 975px) 100vw, 975px" />

Visualisation of bot traffic with bad reputation

Combining real-time security analysis of logs with alerting and ChatOps integration one is able to receive real-time alerts about malicious traffic and take countermeasures like blocking specific clients. Another obvious application of data gained via Access Watch is the exclusion of all bot traffic prior to website traffic analysis in order to get more accurate statistics.

Sematext Visualisation of enriched logs in Kibanahttps://sematext.com/wp-content/uploads/2017/08/Sematext-Visualisation-o... 300w, https://sematext.com/wp-content/uploads/2017/08/Sematext-Visualisation-o... 768w" sizes="(max-width: 975px) 100vw, 975px" />

Visualisation of enriched logs in Kibana

To make this super simple, Logagent users can use the new Access Watch plugin to perform security and traffic analysis, store that in Sematext Cloud or any place else (e.g. their own Elasticsearch cluster) for further analysis, visualization, etc.

Using Access Watch and Sematext provides you with:

  • Detection of all robotic behaviour, good and bad, profiled, and threat assessed
  • Clear and precise insights into the makeup of your traffic
  • Knowledge of what robot activity comes from search engine crawlers, feed readers, price or data scrapers as well as abusive activity from brute force bots and more

Interested in seeing what Sematext Cloud can do for you?   SIGN UP TODAY

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

Latest Stories
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
Most of the time there is a lot of work involved to move to the cloud, and most of that isn't really related to AWS or Azure or Google Cloud. Before we talk about public cloud vendors and DevOps tools, there are usually several technical and non-technical challenges that are connected to it and that every company needs to solve to move to the cloud. In his session at 21st Cloud Expo, Stefano Bellasio, CEO and founder of Cloud Academy Inc., will discuss what the tools, disciplines, and cultural...
Why Federal cloud? What is in Federal Clouds and integrations? This session will identify the process and the FedRAMP initiative. But is it sufficient? What is the remedy for keeping abreast of cutting-edge technology? In his session at 21st Cloud Expo, Rasananda Behera will examine the proposed solutions: Private or public or hybrid cloud Responsible governing bodies How can we accomplish?
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp em...
Real IoT production deployments running at scale are collecting sensor data from hundreds / thousands / millions of devices. The goal is to take business-critical actions on the real-time data and find insights from stored datasets. In his session at @ThingsExpo, John Walicki, Watson IoT Developer Advocate at IBM Cloud, will provide a fast-paced developer journey that follows the IoT sensor data from generation, to edge gateway, to edge analytics, to encryption, to the IBM Bluemix cloud, to Wa...
With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In his session at @DevOpsSummit at 21st Cloud Expo, Chris Van Tuin, Chief Technologist, NA West at Red Hat, will discuss: The top security r...
There is huge complexity in implementing a successful digital business that requires efficient on-premise and cloud back-end infrastructure, IT and Internet of Things (IoT) data, analytics, Machine Learning, Artificial Intelligence (AI) and Digital Applications. In the data center alone, there are physical and virtual infrastructures, multiple operating systems, multiple applications and new and emerging business and technological paradigms such as cloud computing and XaaS. And then there are pe...
SYS-CON Events announced today that B2Cloud will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. B2Cloud specializes in IoT devices for preventive and predictive maintenance in any kind of equipment retrieving data like Energy consumption, working time, temperature, humidity, pressure, etc.
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real r...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Your clients expect transactions to never fail, cloud access to be fast and always on, and their data to be protected - no exceptions. Hear about how Secure Service Container (SSC), an IBM-exclusive open technology, enables secure building and hosting of next-generation applications, both cloud and on-premises. SSC protects the full stack from external and insider threats, allows automatic encryption of data in-flight and at-rest, and is tamper-resistant during installation and runtime – with no...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Suzuki Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Suzuki Inc. is a semiconductor-related business, including sales of consuming parts, parts repair, and maintenance for semiconductor manufacturing machines, etc. It is also a health care business providing experimental research for...