1

I was setting up a new filter for handling certain cases and I eventually discovered that just using search parameters was about 4x faster than doing the equivalent filter with one search field.

Namely, a saved filter searching for "project:Beta openedby:beta title:SpecialUpload" runs much faster than a filter on project Beta opened by Beta, searching the title for "SpecialUpload."

Edit for more information:

It's actually even more wierd than that. A non-saved search is about 3x than the same search saved as a filter, and a filter exactly equivalent to the saved search is about 2x slower than the saved search.

Here's a little background information: we have a special FogBugz virtual user that gets assigned emails for our beta@ account. We added a large-file-uploader to our web-page that sends notifications to the same account. I wanted to create a filter for those uploads. Here are some sizes for search results: project:beta returns 165, project:Beta openedby:beta returns 101, project:Beta openedby:beta title:SpecialUpload returns 6.

If I do a (non-saved) search for: project:"Beta" openedby:"beta" title:"SendThisFile", a ctrl-F5 finishes consistently in 2-3 seconds. I've saved that exact search as a filter, and a ctrl-F5 on that consistently finishes in 8-10 seconds. If I build a filter for project:beta, openedByUser:beta (selecting the exact match for both), with a search for title:"SendThisFile", a ctrl-F5 finishes consistently in 22-24 seconds.

Update (FIXED):

I have verified that after upgrading from 8.6.5 to 8.7.58, the raw search is much faster than 2-3 seconds, and the saved search seems to run at the same speed as the raw search.

flag

3 Answers

2

Short answer:

Full-Text search + sorting has degenerate cases, and they become more likely when you're combining search criterias with filters (/saving searches) instead of using the search box directly.

It sounds like you're generating the title field programmatically (hence the "title:SpecialUpload" criteria) -- You could programmatically add a tag to the cases you wish to filter for (tag:SpecialUpload) and then filter based on the tag (a non-full-text operation) rather than the title (a full-text operation) in order to get much faster results.

Long answer:

The behavior you're seeing is most likely caused by combinations of the following factors:

  • Full-text searches (such as title:) are much slower than other search/filter operations
  • Full-text searches combined with a sort order can lead to a degenerate case, but even when they don't will be strictly slower than non-ordered full-text search operations
  • Filters tend to have sort orders, whereas searches you type directly into the search bar are only sorted if you specify a sort order (...otherwise they sort by a relevance metric)

There are two types of operations which FogBugz will perform when you run a filter or a search:

  • Result filtering based on values in the database
  • Full-Text search using Lucene

Filters, filters with a search component, and searches entered into the search field all make their way to the same "query parser". This parser extracts the full-text components from the query, runs them against Lucene, and then combines the Lucene results with the database column filtering requested by the query.

For example, searching for Jude status:active will retrieve full-text matches of my name, from Lucene, and then join that result set with the items from the database which match the criteria of status:active.

I'm leaving out a few steps here (like permissions checks), but that's the gist of it.

In your case,

project:Beta openedby:beta title:SpecialUpload

The Project and OpenedBy axes are filtering using the database, while the title axis is filtering against lucene. Because we incur the cost of interacting with the full-text search engine, this query is strictly slower than

project:Beta openedby:beta

which doesn't interact with Lucene at all.

Searches embedded in filters are slightly more advanced: the query parser combines the filter logic with the search logic in order to put together it's best effort approximation of the desired query. Since the search component can be arbitrary (and even contradict the filter component), it's possible to create some interesting results here. Regardless, it tends to be the case that queries created by combining filters with searches (rather than performing the 'equivalent' search using the search box) actualy have some additional implicit filtering criteria (such as sort order.)

e.g., a filter with a specific sort order (almost all filters have this) plus a full-text search component is going to be strictly slower than that same filter without the sort, which in turn is going to be strictly slower than that same filter without the full-text search.

I would guess that the multi-tiered performance profiles you're seeing are:

  1. Non-full-text filters/search (regardless of sort order)
  2. Full-text search (no sort)
  3. Filter with full-text search (same as #2, but with additional filtering overhead)
  4. Filter with sort ordering and full-text search
link|flag
"otherwise they sort by a relevance metric": sure? Compare fogbugz.stackexchange.com/questions/7274/… – Michel de Ruiter Dec 14 at 22:29
Unfortunately, what we're dealing with here is sorting of incoming emails, so tagging isn't an option. I guess you could consider this another +1 for rule-based email filtering. – Bryce Schober Dec 16 at 16:23
1 
@Michel Interesting... alas not 100% accurate. In the absence of any other filtering (which is actually fairly uncommon) search results are ordered based on Lucene.NET's relevance metric. The algorithm we use is "Oh Lucene, please do your best, but have the following biases: Weight title text at 2x the influence of other text that you index, and if you're indexing a case then add +0.05% relevance per case event and, if the case is marked as 'spam', divide the final relevance score by 4." Unfortunately this algorithm isn't very good, so it's largely more accurate to say "what relevance?" – Jude Allred Dec 21 at 18:17
1

We're working all-out on performance, including search performance, for upcoming releases. I'll create a case to inspect this scenario and roll it into our work.

link|flag
0

Your use of the search axis is going to the database and doing something like:

AND sTitle LIKE ('%SpecialUpload%')

Because this is a database operation, it's faster than your other search, which goes to the search index and so hits a completely separate code path.

If you add a search for title:SpecialUpload rather than adding a search for SpecialUpload alone.

link|flag
You mean searching for title:something in the filter Search for is different from title:something in the search box? – Michel de Ruiter Dec 13 at 9:52
The question was edited after my answer. – Rich Armstrong Dec 14 at 16:10

Your Answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.