Protocol attack
A protocol attack is a DDoS attack that exploits weaknesses in network protocols themselves, exhausting server or network equipment resources rather than raw bandwidth.
What makes this different from a volumetric flood
A protocol attack does not necessarily need enormous bandwidth. It exploits how a specific protocol, such as TCP's connection setup process, is designed to behave, exhausting connection tables or processing capacity on the target or the network equipment in front of it.
A classic example
A SYN flood sends a large number of connection initiation requests without ever completing the handshake, leaving the target holding open half-finished connections until it runs out of capacity to accept new, legitimate ones.
See our DDoS protection guide for small businesses for the mitigations that fit a small business.
Recognising a SYN flood on your server
A SYN flood fills the queue of half-open connections, as RFC 4987 describes, so the evidence lives in the network stack rather than in your web server logs: the handshake never completes, so nginx never sees a request. Three places to look:
ss -Htan state syn-recv | wc -l
journalctl -k | grep -i 'syn flooding'
nstat -az TcpExtTCPReqQFullDoCookies TcpExtTCPReqQFullDropThe first command counts connections stuck in the SYN-RECV state. A handful is normal, a number that keeps growing is not, so compare against your own baseline. The second searches the kernel log for the "Possible SYN flooding" warning, which Linux prints when a listening socket's SYN queue overflows. The third prints two counters: replies sent with a SYN cookie, and SYNs dropped because cookies were off. The warning alone does not prove an attack, since a legitimately busy listener can trigger it too, so check whether the source addresses look real before you act. If you are not sure how to read connection state, the Linux firewall guide covers the tooling.
Kernel settings and their trade-offs
Linux already carries the main defense. Read the current values first:
sysctl net.ipv4.tcp_syncookies net.ipv4.tcp_synack_retries net.core.somaxconnThe kernel documentation gives tcp_syncookies a default of 1, meaning cookies are sent when the SYN backlog overflows; setting 2 sends them unconditionally. tcp_synack_retries defaults to 5, about 31 seconds before the last retransmission, and somaxconn defaults to 4096 (it was 128 before Linux 5.4). Your distribution may differ, so trust the output, not the table.
None of these is free. RFC 4987 notes that common cookie schemes drop some TCP options such as window scaling, and that a lost final ACK is a problem for protocols where the server speaks first, such as SMTP. Shortening the SYN-RECEIVED timer frees entries faster but can also kill slow legitimate handshakes. Beyond tuning, the RFC describes putting a firewall or proxy in front that completes the handshake on the server's behalf, which is what a CDN does for you. Add per-address limits on new connections with rate limiting and keep the layered setup from the DDoS protection guide.
Related terms: Volumetric attack, Application-layer attack, Firewall, SYN flood, Botnet.
Frequently asked questions
Is a protocol attack harder to detect than a volumetric one?
In some ways yes, since the traffic volume itself might look modest while still exhausting a specific resource like connection table capacity, rather than saturating bandwidth outright.
Can a well-configured firewall reduce exposure to protocol attacks?
Yes, to a meaningful degree. Rate limiting new connections and enabling SYN cookies at the operating system level are common, effective mitigations for this category specifically.
Does this type of attack target the application or the infrastructure?
Mainly the infrastructure, meaning the network stack and connection-handling equipment, rather than the application logic running on top of it.
Are SYN cookies enabled by default on Linux?
The kernel documentation lists net.ipv4.tcp_syncookies with a default of 1, which sends cookies when the SYN backlog overflows. Distributions can change defaults, so confirm the live value with sysctl net.ipv4.tcp_syncookies.
Will a SYN flood show up in my nginx access log?
No. The attacker never completes the handshake, so nginx never receives a request to log. Look at the kernel log, the count of connections in the SYN-RECV state from ss, and the nstat counters instead.
Want to see where your own server stands?
Run the free, read-only server check, or open the Security Lab and watch the detect, contain, recover, verify loop in action.
Get your free server checkOpen the Security Lab