Port Forwarding with nftables + flowtable: How to Set It Up and Why It Beats iptables

Scenario

A relay server with a single NIC (eth0) forwards a few ports to the target host 1.2.3.4:

Every forward rewrites two addresses:

• DNAT: change the destination from the relay to the target.
• SNAT: change the source from the client to the relay. Without SNAT the target sends its replies straight to the client, but the client connected to the relay, so the replies don't match and the connection never comes up.

conntrack (the connection tracking table) remembers the rewrites and reverses them for replies, so the return path needs no rules of its own. Whether you use nftables or iptables, enable kernel forwarding first:

echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-forward.conf
sysctl --system

All commands below run as root. 1.2.3.4 is an example address; check the NIC name with ip -br link. On cloud servers ens3 and enp1s0 are common too.

The complete nftables + flowtable config

/etc/nftables.conf:

#!/usr/sbin/nft -f

# 只替换自己这张表，不碰其他程序的规则。
# 先声明再删除：第一次加载时表还不存在，也不会报错。
table ip relay
delete table ip relay

table ip relay {
    # 快速路径：已登记的连接在 eth0 入口直接转发
    flowtable ft {
        hook ingress priority 0
        devices = { eth0 }
    }

    # DNAT：每个转发一行
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        meta l4proto { tcp, udp } th dport 54000 counter dnat to 1.2.3.4         # 端口不变
        meta l4proto { tcp, udp } th dport 54001 counter dnat to 1.2.3.4:6667    # 换端口
        tcp dport 54002 counter dnat to 1.2.3.4                                   # 只转 TCP
    }

    # 把转发的 TCP/UDP 连接登记进 flowtable
    chain forward {
        type filter hook forward priority filter; policy accept;
        meta l4proto { tcp, udp } flow add @ft
    }

    # SNAT：凡是做过 DNAT 的连接，源地址都换成中转机自己的
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        ct status dnat masquerade
    }
}

Load it and enable it at boot:

nft -c -f /etc/nftables.conf       # 只检查，不生效
nft -f /etc/nftables.conf          # 加载
systemctl enable nftables          # 开机自动加载

On Debian and Ubuntu the nftables service reads /etc/nftables.conf; on the RHEL family it reads /etc/sysconfig/nftables.conf, so adjust the path. The sections below walk through the config piece by piece.

flowtable: the entry to the fast path

hook ingress attaches the flowtable at the earliest point of packet reception, before the packet reaches the IP layer. devices lists the NICs that forwarded traffic enters and leaves through: on a single-NIC relay that is just eth0, while a router with two NICs lists both. Only packets received on these NICs are looked up in the flowtable.

A new flowtable is empty; the flow add rule in the forward chain fills it with connections.

prerouting: DNAT

dnat to 1.2.3.4 changes only the address and keeps the port; dnat to 1.2.3.4:6667 changes the port as well. meta l4proto { tcp, udp } matches both TCP and UDP, and th dport reads the destination port from the transport header, which works for either protocol.

Chains of type nat only see the first packet of each connection; later packets are rewritten from the mapping stored in conntrack. So the counter here counts new connections, not packets or traffic. In my test, a connection that carried 50 MiB left its rule at packets 1 bytes 60, and those 60 bytes are the SYN.

forward: registering connections

flow add @ft registers the connection in the flowtable (older versions spell it flow offload @ft, with the same effect). The first packet of each connection always takes the regular path; the connection is registered when later packets pass this rule. Once registered, its packets no longer go through the forward chain.

postrouting: one SNAT rule for everything

ct status dnat matches connections that were DNATed, and masquerade uses the outgoing NIC's address as the source. The rule names no port or target, so it never needs to change when you add or remove forwards. If the relay has a fixed IP, you can also write ct status dnat snat to <中转机 IP>.

The ip family handles IPv4 only; to forward IPv6 as well, use the inet family.

iptables -I DOCKER-USER -m conntrack --ctstate DNAT -j ACCEPT

Day-to-day: adding, removing, persisting

The flowtable itself never needs changing: every forwarded TCP/UDP connection is offloaded automatically, and the SNAT rule is generic. So adding a forward means adding one DNAT line.

Recommended: edit the config file and reload

Edit the config file and reload it, so the change takes effect immediately and is also loaded at boot:

nano /etc/nftables.conf            # 在 chain prerouting 里加一行，例如：
#   meta l4proto { tcp, udp } th dport 54000 counter dnat to 1.2.3.4         # 端口不变
#   meta l4proto { tcp, udp } th dport 54001 counter dnat to 1.2.3.4:6667    # 换端口
#   tcp dport 54002 counter dnat to 1.2.3.4                                   # 只转 TCP
nft -c -f /etc/nftables.conf       # 先检查语法
nft -f /etc/nftables.conf          # 原子替换，现有连接不断；计数器会清零

Removing a forward works the same way: delete the line from the file and run nft -f again.

The whole file is committed in one transaction: either all of it takes effect or none of it does, so there is never a gap where the old rules are gone and the new ones are not yet in place. The address mappings of established connections live in conntrack, not in the rules, so a reload does not interrupt them: offloaded connections fall back to the regular path and are registered again when their next packet passes the forward chain. In my test, reloading in the middle of a TCP transfer did not break the connection. The counters reset because the rules are deleted and recreated.

Temporary rules (gone after a reboot)

nft 'add rule ip relay prerouting meta l4proto { tcp, udp } th dport 54000 counter dnat to 1.2.3.4'
nft -a list chain ip relay prerouting                  # 查看规则的 handle 编号
nft delete rule ip relay prerouting handle <编号>

Wrap the whole rule in single quotes so the shell leaves characters such as { } and ; alone. -a prints a number such as # handle 36 after each rule; use it to delete the rule. Temporary rules are not in the config file, so the next nft -f removes them as well.

Checking that forwarding and offloading work

nft list chain ip relay prerouting     # 每条 DNAT 的 counter：命中过多少个新连接
nft list flowtables                    # flowtable 是否存在、挂在哪些网卡上
conntrack -L | grep OFFLOAD            # 正在走快速路径的连接（需要安装 conntrack 包）

Connections on the fast path carry an [OFFLOAD] tag in conntrack -L, roughly like this (the addresses are illustrative):

tcp      6 src=203.0.113.7 dst=198.51.100.1 sport=51234 dport=54000 src=1.2.3.4 dst=198.51.100.1 sport=54000 dport=51234 [OFFLOAD] mark=0 use=2

After offloading, the counters in the forward chain stop growing, and the traffic that conntrack -L shows freezes at the moment of offload. To keep conntrack counting traffic, add a counter line to the flowtable (supported since Linux 5.7).

When forwards pile up: use a map

Once you have many forwards, you can put "port → target address and port" into a map and use a single DNAT rule that does one hash lookup:

table ip relay
delete table ip relay

table ip relay {
    flowtable ft {
        hook ingress priority 0
        devices = { eth0 }
    }

    map portmap {
        type inet_service : ipv4_addr . inet_service
        elements = {
            54000 : 1.2.3.4 . 54000,
            54001 : 1.2.3.4 . 6667,
        }
    }

    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        meta l4proto { tcp, udp } dnat ip addr . port to th dport map @portmap
    }

    # forward 和 postrouting 两条链与上面相同，这里省略
}

Entries can be added and removed at runtime without touching the rules:

nft add element ip relay portmap '{ 54003 : 5.6.7.8 . 443 }'
nft delete element ip relay portmap '{ 54003 }'

A few things to watch:

• This rule treats TCP and UDP alike, so a TCP-only port still needs its own tcp dport … dnat to … line.
• Don't name the map fwd: it is an nft keyword, and nft fails with syntax error, unexpected fwd.
• Entries added with add element are not in the file and disappear on the next nft -f; put the entries you want to keep in the file. In my test, replacing the delete table at the top of the config with flush table clears only the rules: the map's elements all stay, and entries you delete from the file do not go away.

A map mostly makes the config easier to maintain. As for performance, nat chains only see the first packet of each connection anyway, so matching a few dozen DNAT rules one by one costs little.

Why it is faster than iptables

The regular path of a forwarded packet

Without a flowtable, every forwarded packet walks the whole path, with iptables and nftables alike:

1. The NIC receives the packet, and it enters the IP layer.
2. PREROUTING hook: conntrack finds the connection the packet belongs to. nat rules are matched only for new connections; after that, the stored mapping is applied directly.
3. A route lookup decides which NIC the packet leaves through.
4. FORWARD hook: filter rules are matched one by one.
5. POSTROUTING hook: the SNAT rewrite happens, and a new connection is committed to conntrack here.
6. The neighbour subsystem (which resolves the next hop's MAC address) and the NIC queue send the packet out.

So a few extra DNAT rules are not the problem, because nat rules only apply to new connections. The cost every single packet pays is the conntrack lookup, the route lookup, and walking the rules of every table on three hooks.

The flowtable fast path

The flowtable adds a shortcut at the very start of this path. When it registers a connection, it stores the addresses and ports of both directions, how NAT rewrites them, the output NIC and the next hop. From then on, a packet received on one of the devices costs one hash lookup at the ingress hook:

• Hit: rewrite the addresses and ports as stored, decrement the TTL, and hand the packet straight to the output NIC. IP-layer receive processing, the three netfilter hooks, the conntrack lookup, the route lookup and all rules are skipped.
• Miss: take the regular path described above.

flowchart TD
    rx["NIC receives"] --> hit{"Connection in the flowtable?"}
    hit -- "yes" --> nat["Rewrite addresses and ports, decrement TTL"]
    nat --> tx["NIC sends"]
    hit -- "no" --> pre["PREROUTING: conntrack, DNAT"]
    pre --> rt["Route lookup"]
    rt --> fwd["FORWARD: flow add registers it"]
    fwd --> post["POSTROUTING: SNAT"]
    post --> tx

For a packet of an established connection, the work compares as follows:

What nftables itself improves

Beyond the flowtable, nftables brings a few real improvements:

• Rule updates: with the legacy backend, every iptables -A reads the whole table out of the kernel, modifies it and writes the whole table back. The more rules there are, the slower it gets, and programs that change rules at the same time have to queue on a lock with -w. nftables adds and deletes single rules in netlink transactions, and nft -f commits a whole file as one atomic transaction.
• Sets and maps: one hash lookup replaces a chain of rules matched one by one, like the portmap above.
• One tool: the inet family handles IPv4 and IPv6 together, instead of separate iptables and ip6tables rule sets.

But if you only translate iptables rules into nft rules one by one, the forwarding path does not change: both use the same netfilter hooks and conntrack. What makes the difference is the flowtable.

When you will notice

The fast path saves CPU, so it shows most when the CPU is the bottleneck: lots of small packets and high packet rates (game or voice traffic over UDP, for example), or a small VPS or software router with weak single-core performance. If the relay's bandwidth is modest and the CPU is idle anyway, speed and latency may not change noticeably, but softirq load (si in top) will be lower.

Packets that skip the fast path

The kernel documentation lists these cases:

• Only TCP and UDP over IPv4/IPv6 are handled; ICMP and other protocols always take the regular path.
• The first packet of each connection takes the regular path, and TCP FIN and RST packets go back to the regular path so that the connection can close cleanly.
• Fragments and packets larger than the output MTU go back to the regular path (the latter so that the sender gets an ICMP "packet too big").

Also, offloaded packets no longer pass any netfilter hook after ingress, so rate limits, logging and counters in the forward chain never see them. To rate-limit forwarded traffic, use tc on the egress queue. flags offload can push forwarding further down into the NIC hardware, but it needs support from the NIC or the router chip; the virtual NICs of cloud servers generally cannot use it, and the software flowtable is enough.

How iptables did forwarding before

The same three forwards look like this with iptables:

# 54000 → 1.2.3.4:54000，TCP + UDP
iptables -t nat -A PREROUTING -p tcp --dport 54000 -j DNAT --to-destination 1.2.3.4:54000
iptables -t nat -A PREROUTING -p udp --dport 54000 -j DNAT --to-destination 1.2.3.4:54000
iptables -t nat -A POSTROUTING -d 1.2.3.4 -p tcp --dport 54000 -j MASQUERADE
iptables -t nat -A POSTROUTING -d 1.2.3.4 -p udp --dport 54000 -j MASQUERADE

# 54001 → 1.2.3.4:6667，换端口
iptables -t nat -A PREROUTING -p tcp --dport 54001 -j DNAT --to-destination 1.2.3.4:6667
iptables -t nat -A PREROUTING -p udp --dport 54001 -j DNAT --to-destination 1.2.3.4:6667
iptables -t nat -A POSTROUTING -d 1.2.3.4 -p tcp --dport 6667 -j MASQUERADE
iptables -t nat -A POSTROUTING -d 1.2.3.4 -p udp --dport 6667 -j MASQUERADE

# 54002 → 1.2.3.4:54002，只转 TCP
iptables -t nat -A PREROUTING -p tcp --dport 54002 -j DNAT --to-destination 1.2.3.4:54002
iptables -t nat -A POSTROUTING -d 1.2.3.4 -p tcp --dport 54002 -j MASQUERADE

Key points:

• An iptables rule can name only one protocol, so TCP and UDP each get their own rule.
• POSTROUTING runs after DNAT and sees the rewritten destination address and port, so the SNAT rules for the port-changing forward match --dport 6667, not 54001.
• MASQUERADE can also be replaced with -j SNAT --to-source <中转机 IP>.
• Every forward adds two to four rules, and removing it means deleting all of them.

iptables can in fact do a generic SNAT too: match DNATed connections with the conntrack module. The effect is the same as ct status dnat masquerade in nftables, and each forward is left with only its DNAT rules:

iptables -t nat -A POSTROUTING -m conntrack --ctstate DNAT -j MASQUERADE

Listing, deleting and persisting:

iptables -t nat -L PREROUTING -n --line-numbers    # 带编号列出
iptables -t nat -D PREROUTING 3                    # 按编号删除第 3 条
apt install iptables-persistent                    # Debian / Ubuntu
netfilter-persistent save                          # 保存到 /etc/iptables/rules.v4，开机自动恢复

On the RHEL family, use the iptables-services package; service iptables save saves the rules to /etc/sysconfig/iptables. If the FORWARD policy is DROP, you also have to accept the forwarded connections (in the DOCKER-USER chain when Docker is installed):

iptables -I FORWARD -m conntrack --ctstate DNAT -j ACCEPT

The iptables you type may already be nftables

First check the output of iptables -V: (nf_tables) means the nf_tables backend, and (legacy) the legacy one. Since Debian 10, Ubuntu 21.10 and RHEL 8, the default is the nf_tables backend: the iptables commands you type are translated into nftables rules, and nft list ruleset shows tables such as table ip nat.

So iptables is not slow because of the commands themselves, but because its syntax has no flowtable. OpenWrt once patched iptables with a FLOWOFFLOAD target (the "software flow offloading" switch of the fw3 era); since 22.03, fw4 uses the native nftables flowtable instead.

Migrating from iptables to nftables

1. Export the existing nat rules and let the tool translate them as a reference:

   bashCopy

   iptables-save -t nat > /root/nat.rules
   iptables-restore-translate -f /root/nat.rules

   The output is a list of nft commands, for example:

   textCopy

   add rule ip nat PREROUTING tcp dport 54000 counter dnat to 1.2.3.4:54000
   add rule ip nat PREROUTING udp dport 54000 counter dnat to 1.2.3.4:54000
   add rule ip nat POSTROUTING ip daddr 1.2.3.4 tcp dport 54000 counter masquerade

   The translation is one to one: it does not merge TCP and UDP, does not switch to a generic SNAT and does not add a flowtable. Rewriting it into /etc/nftables.conf from the template above gives a cleaner result.

2. Install and load the new config, and enable it at boot:

   bashCopy

   apt install nftables
   nft -c -f /etc/nftables.conf
   nft -f /etc/nftables.conf
   systemctl enable nftables

   Old and new rules can coexist for a while: a connection is rewritten only by the first nat rule that matches it, and since both sides point to the same target, the result is the same and forwarding is not interrupted.

3. Delete the old iptables forwarding rules. Delete only the ones you added (iptables -t nat -D …); on a machine with Docker, do not run iptables -t nat -F, which wipes Docker's rules as well.

4. Keep the old rules from coming back after a reboot: once they are deleted, run netfilter-persistent save again; if you no longer save rules with iptables at all, run systemctl disable netfilter-persistent.

5. Connect to a forwarded port from a client, then confirm with conntrack -L | grep OFFLOAD that the connection took the fast path.

Summary

• Forwarding with nftables + flowtable has three parts: one DNAT line per forward in prerouting, one generic ct status dnat masquerade in postrouting, and one flow add in forward that hands connections to the flowtable.
• Day-to-day changes touch only the DNAT line: check the edited file with nft -c, then load it atomically with nft -f.
• The speed comes not from nftables syntax but from the flowtable: once a connection is established, a packet costs one hash lookup at the NIC's ingress and goes straight out, skipping the IP layer, the netfilter hooks, conntrack and the route lookup.
• iptables needs separate TCP and UDP DNAT and SNAT rules for every forward and has no flowtable; when migrating, use iptables-restore-translate as a reference and rewrite the rules from the template.

References: Netfilter's flowtable infrastructure (kernel documentation), Flowtables (nftables wiki).