Cannot remove `xt_recent` module

Even after removing iptable rules that reference Recent module, I cannot remove the kernel module xt_recent and get the error that it is still in use.

user@host:~$ sudo iptables -S-P INPUT DROP-P FORWARD ACCEPT-P OUTPUT ACCEPT-A INPUT -i lo -j ACCEPT-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT-A OUTPUT -o lo -j ACCEPTuser@host:~$ lsmod | grep xt_recentxt_recent              24576  2x_tables               40960  7 ip6table_filter,xt_conntrack,iptable_filter,xt_tcpudp,xt_recent,ip6_tables,ip_tablesuser@host:~$ sudo modprobe -r xt_recentmodprobe: FATAL: Module xt_recent is in use.``