Which operating mode should I use?

In general we recommend using the gateway version to block viruses at the gateway and implement advanced content security rules, and to use the VS API version mainly to block internal virus outbreaks.

If you don't have Exchange 2000/2003, you must use the SMTP gateway version.
If you have Exchange 2000/2003, you can choose. If you have a large network or many users on Exchange, its better to install GFI MailSecurity in gateway mode at the perimeter of your network and use the VS API mode only to block internal virus outbreaks. If you have a small number of users, you can just install the VS API version.
If you want to block entire mails, rather then message parts, you need to use the SMTP gateway version.
SMTP gateway mode is the correct mode if you run Exchange 5.5, Lotus Notes or another SMTP/POP3 server!