Everything you need to know about the vulnerability scanning process

With high-risk vulnerabilities popping up every other week, realising there is no such dream ‘patch everything’ and configuration changes slowly add up to weakening your infrastructure security. Vulnerability scanning and management are core components of a solid cyber security strategy, ensuring a sound risk management process. Vulnerability management helps organisations keep an eye on their assets, both from asset management and operational security.

Whether your organisation is starting or higher up on the cyber security maturity ladder, this guide is a good source for understanding and improving existing security controls. It covers:

• Why do we need scans?
• What is vulnerability scanning?
• Advantages of vulnerability scanning
• Vulnerability scanning tools
• Types of vulnerability scanning
• Vulnerability scanning process
• Vulnerability scanners
• How often should you scan your assets?

Feel free to watch this video containing a condensed version of the article.

This article is not related to deep dive in technical security assessments such as red teaming, application or mobile security testing.

Lets’ get started.

Why do we need vulnerability scanning?

You need regular scanning to identify the gaps left during the development and deployment process. And to do this before attackers catch you in your blind spots.

Here’s a more detailed response to the above mentioned two sides.

Essentially, any business with access to IT systems to communicate, transact or process information with other businesses or people. It includes external systems (facing the internet) and internal systems (inside a controlled private network such as a wireless network in an office or wired networks within a multi-national).

Keeping the above info in mind, businesses regularly use digital technology such as procurement portals, CRM, eCommerce applications, portals, and numerous other business uses.

• How do you avoid security concerns that may introduce during the development and deployment stage? Even if it’s a well-known product in use, what may introduce security vulnerabilities during configuration and implementation stages in your environment.
• How about third-party components used in your systems?
• How is your remote connectivity environment from the outside?
• Systems where sensitive data is stored – how is data at rest and data in transit secured?
• Are you ensuring endpoint security for the systems and devices connecting to your production and corporate environments?

Attackers regularly use automated tools to scan the internet for exploitable opportunities, particularly websites and devices. Specifically, the main flaws these software look for are:

• Default or weak credentials
• Finding vulnerabilities where popular and reliable exploit code is available
• Broken access controls
• Information leakages
• Presence of default files/content

What is vulnerability scanning?

Vulnerability scanning is the technique of identifying security vulnerabilities affecting an organisation. It includes security software tools such as vulnerability scanners to identify issues after conducting hundreds of checks.

Scanning tools include thousands of checks and signatures used to gather information, probe systems, and identify security vulnerabilities that could be used by threat actors that could negatively impact an organisation. It could be a simple information leakage issue to more serious issues, such as a critical patch on an internet-facing server or broken access controls on an application.

Implications of such issues could mean a threat actor gaining remote access leading to complete control of a network for data theft, data breach or disrupting business operations using Denial of Service (DoS), Distributed Denial of Services (DDoS) attacks.

This process of identifying vulnerabilities coupled with quantification and risk remediation together is known as vulnerability management. Vulnerability management is more complex due to skill-set and resources, the ongoing nature of the work, and changes affecting systems’ functioning (pre and post risk remediation).

The vulnerability management process is often planned and provides valuable input to the cyber security risk assessment. Risk remediation involved with this process is mostly quick and planned, with some exceptions being strategic in nature (where significant impacts are involved).

Advantages of vulnerability scanning

Automation

An automated vulnerability scan can be scheduled, executed on-demand or included in the asset lifecycle process based on anticipated changes such as a new project release, server deployment, etc. It helps identify blind spots to enable up to date views of the threat landscape around your assets.

Compliance

Vulnerability scanning services include bespoke checks aimed to comply with compliance requirements. This could easily help organisations fulfil various compliance checks. These include but not limited to CIS benchmarks, PCI DSS, ISO, others. Wider technical compliance audits include Amazon Web Services (AWS) compliance, Citrix, Cisco, Data configuration audits, F5, HP, Juniper, FireEye audit compliance, Fortinet FortiOS Audit, Azure Audit compliance, MongoDB, VMware, SonicWALL, Windows configuration audit and other devices/systems audit compliance. Some configuration audits revolve around WordPress security audits, Joomla, Magento, and other eCommerce platforms that help security teams close vulnerabilities left during the development and deployment process.

Speed

It provides a significantly faster pace than manual testing techniques such as pen-testing. Scenarios where hundreds of websites and systems require thousands of checks in a limited time, are the ideal fit that is otherwise impossible with costly penetration testing exercises.

Costs

The benefits of utilising scanners, speed and automation elements make scanning a cost-effective and economic model for organisations.

Scalability

With the current trends of hybrid or cloud-based models, the number of resources can increase or decrease in no time.

What is the main difference between vulnerability scanning and penetration testing?

Vulnerability scanning vs penetration testing is a popular question, and sometimes both terms are used interchangeably. Technically they are two different service offerings with differences in price, scope, depth and output focus.

Vulnerability scan is aimed at identifying, quantifying and categorising security vulnerabilities in a network. This provides an insight into cyber security risks affecting your infrastructure.

Don’t be fooled into thinking ‘a scan a day keeps attackers away’. It works with penetration testing as part of the security strategy and ensures coverage across the depth and breadth of your estate.

Penetration testing is an in-depth exercise to identify and safely exploit weaknesses in an organisation’s networks, applications, or systems. It includes information to help businesses remediate the identified risks and help prepare an effective risk remediation plan.

It is thorough and involves scope focussed on specific assets (e.g., a server), assets (a set of applications) or asset category (networks).

‘Light touch’ Penetration testing

Both vulnerability scanning, as well as pentesting, have their functional requirements for businesses. They cannot replace each other and nor do they deliver the same objective.

Please be clear around situations where ‘light touch’ penetration testing is sold in disguise of scans or vice versa. Security testing has its own benefits, and there is no such concept as ‘high-level pen testing’ or covering only certain parts through pick and choose.

Take an example; let’s say the scope is to pentest a retail web application. First, it is learnt how a web application operates and if third-party components deliver certain modules such as SSO, payments, APIs, etc. Almost all major components in the web application are often inter-connected in handling user information, or underlying data and assessment of certain specific modules could deliver the inefficient assessment. This will not add to your validation program due to missing out on comprehensive checks and assurance. Security by obscurity is simply not a valid reason, and shortcuts in cyber security cost dearly.

If you fancy reading this topic in detail, we have a dedicated article around pen testing vs vulnerability scanning.

Vulnerability scanning tools

Cyber security is a problem for businesses of all sizes. Whether you are an individual, a small business, or a big business, it’s ridiculous to consider small businesses have no valuable data on them. Only big businesses are concerned with data breaches. Or data breaches affect big organisations due to their reputation.

Given businesses are transacting in an app-driven world, it is sporadic to see many organisations doing business without technological dependencies. Whether your organisation allows personal devices at work, a small website, laptops, delivering advertising or marketing services or internet exposed services, almost all assets have risk exposure with various types of threats. This constitutes the attack surface of an organisation. Measuring the attack surface is very important before you act on reducing the risks affecting these assets.

Cybercriminals do not target businesses based on big or small; it is based on the effort required to compromise, the sensitivity of the information you hold and its value in the underground markets. ID frauds, sale of personally identifiable information (PII), health care records, payment information value in the underground markets is what drives criminals.

Successful attempts to attack systems lead to compromises for short (days) to long term periods (months) and based on the threat actor’s objectives. Attackers objectives could range from data modification, data theft, data leakage online to ransomware attacks. Such outcomes situations could impact negatively on your organisations, including but not limited to reputational, financial and/or legal implications. Privacy regulations such as GDPR make great progress in ensuring sensitive data is held in line with good security practices and processed in a fair, lawful manner.

Threats vary from organisation to organisation and the assets involved. A cyber security strategy that stands up to the threats of an organisation has to be contextual. The vulnerability scanning process helps businesses ensure they are on top of asset management, weak areas in security posture, and need to invest resources.

Vulnerability scanning tools add input to the wider risk management programme by highlighting where most serious risks lie and helping the risk remediation plans.

Vulnerability scanning process

Vulnerability scanning is most effective when utilised as a part of the organisation-wide Vulnerability Management Program (VMP). This program includes assets and classification, identification, analysis and remediation of vulnerabilities followed by remediation and verification of fixes. This process is defined below in the five stages:

1.   Discover – Asset discovery and classification

This is the foremost step for a vulnerability management program (VMP) that includes the asset discovery and classification phase.

Asset discovery

Modern environments, big or small, have various IT systems, devices, smart devices in use. An asset could be a physical or virtual entity that vulnerabilities are linked. It could be:

•       A network device such as a router or firewall
•       A virtual or physical piece of equipment such as switch, server
•       A web application
•       Cloud-based endpoint or a host

An asset finding exercise includes scanning for assets and recording the identified output in an asset register. A common challenge faced during these exercises is the remote workforce or where users work from their personal mobile devices. It is important to consider common remotely available services and avoid common vulnerabilities (utilising automatic software updates).

Classification

Once the assets have been identified, logical grouping should be performed. It could be based on asset category to define separate, simpler scopes for vulnerability scanning. These scopes can be based on several criteria such as:

External exposure-based assets such as internet-facing customer portals, remote connectivity servers, VPN endpoints. Since the pandemic started, several critical vulnerabilities have been identified in the remote access setups and other popular products from top vendors. This number has been the highest ever compared to the previous years, stressing the need for constant checks on the exposed assets. Some of these exploited vulnerabilities made headlines due to a large number of compromised assets around the globe. These bugs include Citrix VPN Appliance vulnerabilities (CVE-2019-19781), F5 Big-IP vulnerability (CVE-2020-5902), Pulse secure VPN (CVE-2019-11510), Exchange server (CVE-2020-0688) and loads more.

Internal endpoints that are acting as entry point for external traffic. These include laptops, desktops/workstations and mobile devices. Phishing attacks such as whaling, spear-phishing often target employee laptops as the first point of entry with malicious email attachments, malicious content from third-party websites that exploit lack of patching or misconfigurations endpoints. This stage is known as exploitation in the cyber kill chain.

It is important to include coverage of endpoints to ensure they are closely aligned with secure hardening baselines established by the IT security team.

The third type includes hosts, applications, and systems storing sensitive data and hosted on behalf of the organisation. Often, it includes managed services provided to host CRM, dashboards, corporate websites, marketing and other advertising portals. Losing such data may attract reputational or regulatory penalties (GDPR in personnel data from EU and UK). It is important to be aware of the GDPR understanding, including reporting of data breaches.

2.   Assess – Vulnerability detection

This phase involves a vulnerability assessment aimed at identifying vulnerabilities in the target environment. This phase consists of the following steps:

•   Initial sweeps to identify live systems and ensuring they are network accessible for the upcoming scanning phases. It involves ping requests or sending them TCP/UDP packets.
•   Perform port scanning to find out open ports and carry out fingerprinting services running on the identified open ports.
•   Gather detailed information by performing version fingerprinting of the system (Operating system), running services, presence of default files, and many checks based on identified services.
•   Correlation of data with known vulnerabilities to find out vulnerable services and risk scoring.

A good scanning exercise would find out a variety of systems on a network. It includes network equipment such as routers and switches, file and print servers, databases, virtual and physical servers, laptops, desktops, workstations, mobile devices, firewalls, security devices, etc.

A well-configured vulnerability scanner is a vital component of the entire process. It may include excluding scans that are dangerous for the current architecture that could have adverse consequences such as crashing fragile services, disrupt networks or system during scanning. Similarly, another factor considered is the off-peak hours when scans can efficiently utilise available bandwidth without affecting the production and corporate networks. Third, a well-configured scan could add to the accuracy of results and fewer false positives.

A thorough vulnerability assessment must be planned strategically and not left with a point and click scan functionality to reflect the accuracy and consistent input to the next phase. Therefore, who must take a balance of security needs and business needs into account before big decisions.

3.   Analyse – Vulnerability triage

The vulnerability identification phase outputs a lot of information around vulnerabilities affecting the environment. Vulnerability scanners provide different risk ratings and scoring without taking into context the target environment. Therefore, using a scoring criterion such as CVSS and calculating risks in line with risk management strategy are significant factors in deciding which risks to mitigate first.

The following factors are taken into account when performing vulnerability triage:

•   Checks to identify if a vulnerability is a false or true positive
•   Is it exploitable from the internet?
•   Is there any published exploit code for a vulnerability?
•   What is the likelihood of attack (difficulty) and its impact should a vulnerability be exploited?
•   What are security controls helpful in reducing the attack likelihood/impact?

Like any other software, vulnerability scanners are also prone to reporting data based on their instructions (underlying code) and may report false positives. Therefore, your team needs to add a human edge in this phase to confirm the identified vulnerabilities.

Vulnerability validation exercise exists in both vulnerability scanning and pentesting exercises (also known as VAPT).

Just like tactical patch management, analysis of the vulnerabilities and risk focussed prioritization is the key here.

4.   Fix – Vulnerability remediation

Risk remediation is the next phase after vulnerability validation to decide how best to treat the identified risks. What can manage a risk in one of the following ways:

•   Avoid/resolve it by eliminating the risk, for example, applying a patch or fixing a vulnerability.
•   Mitigate the risk by reducing the likelihood or impact of a vulnerability being exploited
•   Transfer the risk to a different entity such as cyber insurance
•   Accept the risk by acknowledging it and selecting not to resolve, transfer or mitigate

Not all vulnerabilities need fixing. Some issues may not require fixing if compensatory controls have taken over. For example, vulnerable browser plugin versions may not require any fixes if the organisation disables the browser plugin.

You cannot depend on scanners’ remediation measures alone as every organisation has their own risk appetite. The correct approach must be agreed upon by the security team and the relevant system owners. Remediation timelines should be quick, planned as part of ongoing support/maintenance programs or strategic changes to the system state; investments might be needed.

5.   Verify – Validation of fixes

The validation phase involves the verification of fixes to ensure the new attack surface is minimal. This is often an ongoing exercise to ensure all input is fed back into the vulnerability management program (via internal dashboards/metrics).

Types of vulnerability scans

There are three types of vulnerability scans. These are internal scans, external scans and host scans.

1. Internal scans: Internal vulnerability scans are aimed at identifying vulnerabilities within an internal network of an organisation. It could be a cloud network, network segment, a wireless network, a corporate network, or the entire organisation consisting of multiple networks (production, staging, corporate).
2. External scans: External vulnerability scans include the scope of internet-facing components that may consist of email, web, firewalls, applications/portals and websites.
3. Host scans: Host scans include vulnerability assessment aimed explicitly at a single or multiple hosts that serves as a database, web server, server, workstation or another function.

Vulnerability scanners

Vulnerability scanners are broadly divided into two main categories:

• Network vulnerability scanners
• Web application vulnerability scanners

Vulnerability scanner selection must be suited to your environment requirements. For instance, attack vectors vary based on services and asset types, and different vulnerabilities are identified in different ways based on the positioning of an asset. A network-based scanner accurately determines an SMB patching issue such as WannaCry or active directory related weaknesses positioned internally when configured in authenticated fashion. It is wrong to point an external scanner (internet-based) on this server’s external interface mentioned in the example. Similarly, web application scanners suited for applications have sub-categories to target native software applications, databases, mobile applications.

Network vulnerability scanners

As the name suggests, network scanners focus on identifying vulnerabilities in the services available to an internal network (inside a perimeter) or over the internet (external).

Network vulnerability scanners are a significant tool to stay on top of external footprint (over the internet) and regular internal (corporate and production environments) checks around vulnerabilities and misconfigurations.

One benefit of network vulnerability scanners is the quick installation and setup process. You do not require significant resources or an environment to start using vulnerability scanners. Cloud-based scanning over the internet is easily conducted without deployments (where scanners are available in the cloud). This point and click solution save time and cost overheads, making it easier to run.

The vulnerability scanning methodology differs based on a network vulnerability scanner targeting assets inside or outside the network. Internal scanners have additional capabilities, such as credentials, to perform in-depth checks around different policy settings and configuration checks. External scanning includes full port scans, identifying services and associated vulnerabilities over the internet.

External network vulnerability scanning

Network scanning software works by scanning systems over the internet, sending probes for open ports, using ‘fingerprinting’ or ‘banner grabbing’ techniques to identify the services, versioning information and further checks for configuration weaknesses or known vulnerabilities.

This external scanning exercise simulates an internet-based (remote) attacker without prior information about the target network. This type of activity is used to assess the exposure of internet-based assets. It helps security teams to validate their secure hardening practices at play.

Internal network vulnerability scanning

Internal network vulnerability scanning targets systems inside a network that are mostly running services not exposed to the Internet. This extends the scope of the scanning exercise to include a large number of areas in a system. For instance, to scan a windows server, an internal vulnerability scanner is expected to perform an authenticated scan that identifies outdated versions or vulnerable versions of a browser otherwise not identified by an external network scanner.

Internal scanners can sweep the entire estate, particular network segments or systems based on the configuration. Based on the objectives, scanners can be configured to perform checks on specific vulnerabilities or services. For instance, an internal vulnerability scanner can be configured to conduct checks against all SSL/TLS services for encryption misconfigurations and vulnerabilities.

Authenticated scans are often utilised to carry out detailed checks on configurations such as domain policies. Such scans are configured with credentials used to authenticate against a system/server, information around patch management, domain security policy (password policy, accounts, Kerberos, user rights, etc.), services, and OS configuration various other areas are captured.

What are the common vulnerabilities identified during internal vulnerability scanning?

The following list provides examples of various issues identified during vulnerability scans:

• Insecure patch management detailing missing Operating System and third-party application patches
• Unsupported software or OS in use
• Use of weak credentials (default or weak passwords)
• Exposure of sensitive information
• Insecure cryptography usage
• Use of clear-text services
• Security misconfiguration such as insecure service permissions

Web application vulnerability scanners

Web application scanners are designed to identify web applications/websites, web services and native applications.

These scanners work by browsing through the different web pages (crawling or spidering) like a search engine bot and collecting information based on the header responses and web page response body. Web scanners automate this mapping work using spidering to guesstimate the user’s journey in an application. High-performance scanners give you customization to educate the scanner on how best to understand the application with custom setups (web page types/formats, input sequences, in-scope elements and vulnerabilities).

This information is then analysed, and relevant probes are conducted against form fields and web pages identified further from links within these pages.

Application vulnerability scanners should work in two session states, unauthenticated and authenticated. Unauthenticated scanning covers the areas outside the login page from a threat actor’s perspective without any credentials. Authenticated scanning includes assessing web application beyond the login stage from the insider attacker perspective with access to a user account.

Web application vulnerability scanning would never match the application pen testing results that have a focussed approach. It is because application scanners cannot identify issues related to business logic and complexity. However, this doesn’t mean that application scanners should not be used against specific applications. These scanners help identify common and less complex weaknesses in nature (where multi-step input, workflows or processes are not included).

What are the common vulnerabilities identified during automated web application scanning?

The OWASP Top 10 is the go-to benchmark for web application vulnerabilities. A similar version exists for API top 10 security risks. As mentioned above, it is impossible to identify all types of vulnerabilities with scanners that can detect certain OWASP risks reliably. OWASP top 10 web application vulnerabilities are:

1. Injection attacks (SQL injection, LDAP, OS command injection)
2. Broken authentication
3. Sensitive data exposure
4. XML External Entities (XXE)
5. Broken access controls
6. Security misconfigurations
7. Cross-site scripting (XSS)
8. Insecure deserialization
9. Using components with known vulnerabilities
10. Insufficient logging and monitoring

Other issues identified during vulnerability scanning include asynchronous SQL Injection, blind SSRF (Server-side request forgery), CSRF (Cross-site request forgery) and many more modern application security vulnerabilities.

When used in combination with network vulnerability scanners, web application security scanners provide an excellent tool-set to continually present the risk findings to the internal risk register.

Custom applications, thick clients, source code reviews are performed by specific capabilities baked into native software scanners. These scanners are developed to identify common flaws that are introduced during the design and development phases. Such scanners require access to source code and identify issues that web application scanners would not identify. Detection of vulnerabilities during the software development process is related to other exercises such as code reviews, threat modelling that is not directly relevant to the scanning scope.

How often should you run a vulnerability scan?

It is important to understand the drivers behind this objective to make an informed choice. Frequency of vulnerability scanning is often done based on either of the following factors:

• Changes (either code or infrastructure): Modern setups that include DevOps or continual changes to code or infrastructure are pushed regularly. Vulnerability scanning needs to catch up to ensure unknown vulnerabilities are identified. New vulnerabilities are often introduced when new changes are introduced to code or system configuration. Although these may be unintentional by the teams working on these assets, the security team’s core responsibility is to ensure no time is wasted detecting such issues so that timely triage and remediation can occur. In this case, scanning is performed on a bimonthly or quarterly basis.
• Proactive approach: Many organisations that have mature security processes like to perform such activities proactively than reactively. Due to the sheer number of vulnerabilities being discovered by security researchers or appearing in the news due to data breaches and attacks, it is important to know any of your assets are vulnerable. In these cases, vulnerability scanning is performed monthly.
• Compliance scanning: Compliance requirements often include vulnerability scanning or pen test exercises describing the frequency/schedule and submission of reports. PCI DSS, ISO 27001, commission audits, etc., often require regular vulnerability scans with frequency varying between quarterly scans and annual. This may not be a good approach because certain vulnerabilities are exploited well within a 90 days window. An organisation may be stuck with compliance requirements while missing out on the wildly exploited flaws before the next scanning. Therefore, it is always recommended to opt for monthly basis vulnerability scans. Additionally, incremental scans should be conducted when changes take place in the infrastructure or code.

We have covered vulnerability scanning frequency best practices in this article:

How often should you perform vulnerability scanning? Best practices shared

Conclusion

Vulnerability scanning is an essential component of your risk management programme. It feeds directly into your cyber security risk assessment and helps to identify and classify threats affecting the target environment. This task’s ongoing nature ensures continual evaluation of assets, allowing the security team to focus on other priorities.

With regular reports and scanning details, you can present the reports to internal security teams for fixing the identified risks and customers/partners who may require such assurance.

Get in touch directly with our security experts to discuss your concerns.