Observations on Smoke Tests – Part 2

In my last blog post, I talked about the value of performing application smoke testing and some of the shortcomings to be aware of. In this post, I want to offer a brief comparison between the two main types of smoke test scanners: cloud-based and desktop-based.

There are a variety of scanning tools in the market today, from commercial to open source. Some are intended only for identifying a particular vulnerability or class of vulnerabilities, such as weak encryption settings for SSL/TLS. Other scanners are designed for comprehensive, deep-dive web application assessments or for ongoing application vulnerability management. Most commercial application scanners can be divided into two categories according to the environment from which they execute: cloud-based and desktop-based. Both have pros and cons. 

Advantages of Cloud-based vs. Desktop-based:

1. Multi-user management.  Most cloud-based scanners allow multiple users to share the same license. Once the scanner admin receives the license, they simply invite other users to create/register for their accounts via email. In addition, cloud-based scanners often enable admins to define user groups with different permission sets, and access may be logged. These convenience features make cloud-based scanners attractive in terms of user provisioning and management.
2. The ability to run multiple tests at the same time. Due to the inherent limitations of local hardware resources, running multiple tests simultaneously from the desktop is less feasible than from the cloud. Desktop-based scanners are still highly capable given enough CPU horsepower and memory, but default to consuming most of the machine’s available resources. Cloud-based scanners, on the other hand, tend to have vast processing and memory capabilities. The obvious benefit is that it offloads much of the work from the local machine.
3. More reliable and user friendly. This is one of my favorites. After initiating the scan, security professionals have the freedom to use their machine for other tasks, or even reboot their computer. It is still recommended to monitor the scan in case of unexpected situations. The primary benefit is that a power outage, memory corruption or system fault on the local machine won’t interrupt or affect the scan.
4. Dashboard overview. Cloud-based scanners commonly furnish a summary or overview screen that provides scan statistics and progress on one page. This is useful for monitoring the progress of one or more scans in one convenient view. The following screenshot shows a dashboard view, including a list of recent scans, vulnerabilities found, to-do issues, and charts showing severities and trends. This is a very straightforward way to visualize findings over time. 

Figure 1: Dashboard of one cloud-based scanner

Disadvantages of Cloud-based vs. Desktop-based:

1. Lack of details during the scan. While running the scan, some cloud-based scanners show only a progressing bar (number of completed requests or estimated requests to be completed). Most desktop-based scanners, on the other hand, show requests currently under analysis or sequence, URLs being crawled, number of requests sent, and real-time bandwidth usage. These details give users more accurate information on the scan health and progress rate and indicate which requests the user should take a closer look at later.
2. Slower to complete. Based on my experience, compared to desktop-based, cloud-based scanners take more time to complete a single scan on average. For example, I ran four identical tests from both environments, using the same tool vendor, and the results showed that the cloud scanner took at least 30 percent more time (depending on if the website had a lot of dynamic pages) to complete the process. 
3. Less flexibility to change a pre-scheduled test. Some cloud-based scanners do not allow the user to change the scheduled testing window once the test begins. Desktop scanners generally have no such limit. For instance, say the user schedules the testing window from 8:00 a.m. to 5:00 p.m. and then runs the scan. Then if the user needs to adjust the time window to be 9:00 a.m. to 6:00 p.m., they have to either manually start/stop at the right time, or stop and configure another scan. Unlike scan policies, I believe testing windows should be changeable, even after a scan begins.

In this blog post I gave a brief overview of the pros and cons of performing web application scans from desktop vs. cloud environments, in addition to some of the factors to consider when choosing between these platforms. There are plenty of tools and features to select from, but in the end your choice should align with your overall desired results.