New Sphinx-based search engine: known issues

#1 User is offline   Brian Livingston

  • 4-Star Lounger (over 400 posts)
  • Group: Member
  • Posts: 461
  • Joined: 2008-12-19
  • Location:Seattle, Washington, USA

Posted 2010-02-18 02:42

We recently replaced the default search engine that comes with Invision Power Board 3.0 — the software platform that runs the Lounge — with the more powerful Sphinx search engine, an open-source project.

The benefits of using Sphinx are apparent. The old, default search engine did not consider a query like microsoft excel to mean microsoft AND excel, as Google and most other search engines do today. Instead, the old software considered the query to mean microsoft OR excel. This matched many irrelevant documents. Sphinx treats queries like this the same way Google does, with an implied AND between terms. Sphinx also supports many other options, as explained in the software's syntax documentation.

However, several known issues with Sphinx remain. Sphinx is the only third-party search engine that has official support from Invision. However, we found that it took many weeks to get Sphinx to successfully index the Lounge's 700,000+ posts, because certain aspects of IP.Board 3.0 do not integrate with Sphinx as expected. While we attempt to program around these limitations or obtain patches from Invision, our new, improved search engine will still exhibit at least the following known issues:

1. Default order of phrase-search results. Sphinx supports phrase searching like Google, where a search on "microsoft excel" (with quotation marks) matches only documents that contain that phrase. Also, parentheses can be used to create phrase searches, such as (microsoft excel). However, the default order in which the results are sorted does not place phrase matches first. Instead, documents that contain both words but not as a phrase can appear at the top of the results. We hope to make the order default to the most relevant matches first.

2. Member names in the Members tab. IP.Board 3.0 is not allowing Sphinx to show links in the Members tab of the search results page to the members who match a query. Invision technical support says this feature should be working in IPB 3, but they agree that it is not. While we wait for a patch, you can click any member's name in the profile area of any post so see the member's profile (if you are signed in). Posts by a particular member still appear in the Posts tab of the search results page, just as with the old search engine. EDIT: This is fixed, see next post.

3. Help info in the Help Topics tab. Sphinx should also match documents in the Help file, but Invision currently does not display any hits in the Help Topics tab of the search results page. Fortunately, the table of contents of the help material, which is accessible from the main Lounge menu, is usually easy to use to locate help topics.

If you see other bugs in the new search engine that our developers might be able to correct, please reply to this thread and let us know. We are still in the middle of the beta process of rewriting every aspect of the Lounge software. Improving the search engine, however, is one of our top priorities. Thanks for your help.

This post has been edited by Brian Livingston: 2010-03-03 22:07
Reason for edit: to add reference to member-name fix


#2 User is offline   Brian Livingston

  • 4-Star Lounger (over 400 posts)
  • Group: Member
  • Posts: 461
  • Joined: 2008-12-19
  • Location:Seattle, Washington, USA

Posted 2010-03-03 22:06

Member lookup in Advanced Search is fixed. As of Feb. 26, 2010, when you query a Lounge member's name in the Advanced Search form, the Members tab of the search-results page will show the member and a link to his or her profile page. Searching on something other than a member name, or in addition to a member name, however, does not show any results on the Members tab.

We're still working on the issue that Invision Power Board is not allowing our new Sphnix search engine to search help topics. More on that as soon as we have more information.

#3 User is offline   jscher2000

  • LoungeFan
  • Group: Moderator
  • Posts: 21336
  • Joined: 2001-02-08
  • Location:Silicon Valley, California, USA

Posted 2010-03-13 19:22

Sphinx does not seem to be searching thread titles. For example, there was a recent thread on the Word Processing forum named Custom Toolbars - Word 2007, last updated on March 2, 2010. But it doesn't come up in a search for custom toolbars.

== Edit ==

An alternate explanation is that the index has not been updated since mid-February (e.g., on the Word Processing board, searches do not find any posts after 2/21/2010).

== Edit ==

It may be both. If I use Advanced search and enter StarPost and tick the "Search titles only" box, I get zero results.

This post has been edited by jscher2000: 2010-03-13 19:37


#4 User is offline   Keely Dolan

  • Star Lounger (over 50 posts)
  • Group: Administrator
  • Posts: 98
  • Joined: 2010-02-13
  • Location:Seattle, Washington, USA

Posted 2010-03-15 18:08

We're working quickly to resolve this, Loungers. Thank you for your patience!

#5 User is online   Tony Johnston

  • 2-Star Lounger (over 100 posts)
  • Group: Administrator
  • Posts: 139
  • Joined: 2008-12-19
  • Location:Seattle, Washington, USA

Posted 2010-03-16 14:24

This issue has been fixed. The problem was, indeed, that the process that should have been updating the indexes was not running.

Thank you for bringing this to my attention.

#6 User is offline   jscher2000

  • LoungeFan
  • Group: Moderator
  • Posts: 21336
  • Joined: 2001-02-08
  • Location:Silicon Valley, California, USA

Posted 2010-03-16 23:10

View postTony Johnston, on 2010-03-16 12:24, said:

This issue has been fixed. The problem was, indeed, that the process that should have been updating the indexes was not running.

Thanks, Tony. There still seems to be a glitch with thread titles (anything I search with the checkbox checked finds nothing).

== Edit ==

Meant to click Preview... that's the "Search titles only" checkbox.

This post has been edited by jscher2000: 2010-03-16 23:11


#7 User is online   Tony Johnston

  • 2-Star Lounger (over 100 posts)
  • Group: Administrator
  • Posts: 139
  • Joined: 2008-12-19
  • Location:Seattle, Washington, USA

Posted 2010-03-18 12:15

View postjscher2000, on 2010-03-17 05:10, said:

Thanks, Tony. There still seems to be a glitch with thread titles (anything I search with the checkbox checked finds nothing).

== Edit ==

Meant to click Preview... that's the "Search titles only" checkbox.


Ok, thanks for all your work on this. I'll be looking into it.


1 reading this thread
0 members, 1 guests, 0 anonymous