Dead Ends and Searching
Dead ends occur whenever you search only to see that dreaded page that says "0 results." It can be a poor user experience to be limited to refining your search with a single textbox, not knowing if your search will return any results. Web search engines like Google can get away with this because they have billions of results to be found and dead ends are few and far between. That, and the wild wild web is so unpredictable and inconsistent that they can't easily implement more advanced features with any real benefit.
Attribute-Driven Search Engines
I've made up the term "Attribute-Driven Search Engine" because I don't know what else to call it. The idea is to display the attributes for the current search results and allow the user to click an attribute to further filter the results. As you filter the results, the new attribute list is also filtered. As a result, you'll never reach a dead end. This characteristic is what makes this type of search engine so awesome. NewEgg, Amazon, StackOverflow -- these are just some of the sites that have adopted this type of search engine.
An attribute-driven search engine can be implemented in any database-driven site where you have some level of control over your content. If you have a site like NewEgg with a wide variety of products then you'll probably need a combination of categories and attributes. For example, when searching for processors you'll need categories for the brand (AMD, Intel), number of cores (1, 2, 3, 4), socket type (AM2, AM3), etc.
But if you're dealing with content more like StackOverflow you can get away with simply using tags. You can assign a variable number of tags to a given article, product, or whatever it is, and the search engine will treat the tags as the attributes. This greatly simplifies things from a management point of view as you don't have to continually modify your categories as your site grows. You can see this in action at stackoverflow.com/questions if you start clicking the tags on the right hand side under "Related Tags." If you'd like you can keep clicking tags to add to the filter until you're down to just 1 result.
The SilverlightXAP.com Search Engine
So I've been giving all of this a lot of thought. I think having a good search engine can be very important for a website. Maybe it's not such a big deal when it's small. But once you have accumulated 500+ items to search through, then it quickly becomes more difficult to navigate the site.
So what I'm implementing for SilverlightXAP.com takes aspects of sites I've done before and combines them with features of StackOverflow. Every pixel shader will have tags assigned to it, and combined with full-text searching this should allow you to find what you're looking for.
Oh, and there's one other thing. Since I still allow you to search using a textbox, there's still the possibility of reaching dead ends. So I've implemented a simple feature to prevent this. I've added some AJAX which searches as you type and tells you how many results there will be before you actually click "Search."
This is all working with my sample data. And I think this combination of features should allow this search engine to scale well. But only time will tell. I'm just about to move from shared hosting to a virtual dedicated server. After that's done I'll try to get a beta site out there where you guys can check this out. I'd love to hear some feedback.