Unload snd_usb_audio modules

I am trying to unload the snd_usb_audio kernel module in order to try some other module parameters (with modprobe).

This fails:

# rmmod snd_usb_audio

With the following error:

rmmod: ERROR: Module snd_usb_audio is in use

How can I quickly force the USB audio module to unload (without logging out/restarting)?