Zyxel Firewall H Series Security Policy - How to Detect and Prevent TCP Port Scanning with DoS Prevention

This article explains how to configure a USG FLEX H DoS Prevention profile to detect and block TCP port scanning attempts. The DoS Prevention feature analyzes network traffic for protocol anomalies, RFC violations (Requests for Comments), and suspicious traffic patterns, helping to protect the network from reconnaissance activities before an attack occurs.

Enhanced DoS Prevention

Starting from firmware version 1.38, the USG FLEX H Series introduces an enhanced DoS Prevention engine designed to improve the detection and mitigation of port scanning and flooding attacks.

The enhanced mechanism analyzes suspicious traffic at an earlier stage of packet processing, allowing the firewall to detect and block malicious traffic before it consumes unnecessary system resources. This approach improves overall performance by reducing CPU utilization while maintaining effective protection against reconnaissance activities.

The DoS Prevention engine supports detection of multiple scan and flood attack types, including:

• TCP Port Scan
• UDP Port Scan
• IP Protocol Scan
• TCP Sweep
• UDP Sweep
• ICMP Sweep
• IP Protocol Sweep
• TCP Flood
• UDP Flood
• ICMP Flood
• IP Flood

Each detection rule can be individually configured to log events, block offending hosts, and automatically blacklist source IP addresses for a configurable period.

Set Up the DoS Prevention

In the USG Flex H, go to Security Policy > Dos Prevention > Add a profile. Configure a Name for you to identify the profile such as “DoS_Prevention”. Configure the Scan Detection and Flood Detection to block when the Dos prevention events were detected.

DoS Prevention Policy – defines on which interfaces the protection is applied.

DoS Prevention Profile – defines which anomalies are detected and how the firewall responds.

Step 1. Enable DoS Prevention

Open the DoS Prevention Policy tab.

1. Enable DoS Prevention.
2. Click Add to create a new policy.
3. Configure the following settings:

Click Apply to save the policy.

Note: Applying the policy to the WAN interface ensures that Internet-originated reconnaissance traffic is inspected before reaching the firewall.

Step 2. Create a DoS Prevention Profile

Open the Profile tab and click Add.

Configure a descriptive profile name, for example:

DOS_PREVENTION_PROFILE

This profile contains multiple detection engines, including Scan Detection, Flood Detection, and Protocol Anomaly Detection.

Step 3. Configure Scan Detection

Under Scan Detection, configure the following settings:

Enable the following detection rules:

• TCP Port Scan
• UDP Port Scan
• TCP Sweep
• UDP Sweep
• ICMP Sweep
• IP Protocol Scan
• IP Protocol Sweep

For each enabled rule:

• Log: Enable
• Action: Block

These settings allow the firewall to automatically detect reconnaissance activity and temporarily blacklist the scanning host.

Step 4. Configure Flood Detection

The Flood Detection engine protects the firewall from excessive volumes of traffic that may indicate a Denial-of-Service (DoS) attack. Each rule monitors a specific protocol and can automatically block hosts that exceed the configured threshold.

For this example, enable the following protection rules:

Configure the following settings:

Note: Detection thresholds may need to be adjusted depending on your network environment. Enabling flood protection with overly aggressive settings can cause legitimate high-volume traffic to be temporarily blocked.

Step 5. Configure Protocol Anomaly Detection

The Protocol Anomaly Detection engine identifies malformed packets and protocol violations that may be used to bypass security controls or exploit vulnerabilities.

For this configuration, enable the following protection:

Leave the following protections disabled unless they are specifically required by your environment:

Important: Security features that inspect protocol anomalies may occasionally identify legitimate traffic as malicious. Before enabling additional anomaly detection rules in a production environment, validate them in a test environment or monitor the logs to ensure that critical services are not affected.

Test the Result

Using the port scan tool Nmap or hping3 to scan the wan interface. For example, using Nmap security scanner for testing the result: Open the Nmap GUI, set the Target to be the WAN IP of USG Flex H (10.214.48.19 in this example) and set Profile to be Intense Scan and click Scan.

Navigate to Log & Report > Log / Events, you will see log of blocked messages.