Here are a few things we commonly tell customers:
Make sure you're moving cases out of Undecided, whether they're spam or not. The spam filter's decision process is trying to figure out not just whether something is spam or not, but where to put it other than Undecided. If you are closing cases in Undecided, then you're not training the filter that those cases are not spam. Autosort will only leave training mode and start sorting once you have moved 15 non-spam messages out of Undecided. We have a feature request in to warn you when you're closing cases in Undecided. The case number is 1675911.
Close Autosorted cases in the project and area they're sorted into. If you move the case out of the project, FogBugz Autosorter loses track of it. This is not ideal operation, but is due to the need to support multiple Autosort projects. We have a case for this, too, though I can't find it right now. Ping me if you want the number to keep track of it.
Limit the number of Autosort areas to five or six per project. This kind of sorting takes a lot of computational resources. Deciding between seven areas is about three times as intense as deciding between four areas. Fewer choices also give you better sorting.
Have an "Undeliverable" area. Bounces from email servers and out-of-office auto-replies will create new cases. Because they're so predictably structured, though, it's easy to get the sorter trained to sort these out into their own area. We use "Undeliverable" because it sorts nicely to the bottom of the Inbox filter. I just resolve and close them periodically. We should have some sort of special operation here, to close these out easily. I've created case 1695675.
Add areas slowly and organically. (This is taken from Spooky's answer.) Your first order of business should be to train spam vs. nonspam. Once you have mail reliably being sorted into Not Spam, you can add areas one by one to segment out certain types of mail. The more areas you have, the less accurate it will be. Also, it almost never works to send emails that you think will look like the emails you intend to receive. The signals that the sorter listens to go far beyond the actual text content of the email, so it's really not worth your while to send 100 messages with your product name into the Inbox and sort them to "pre-train" the filter. Rather, just open it up and train it naturally. It will respond very quickly to things like web form submissions and auto-generated emails.
