CODEWITHSUNDEEP
searchindexingweb-crawleropensearch
Tim Henderson
Tim Henderson

Reputation: 21

How can I crawl but not index web pages in OpenSearchServer?

I'm using OpenSearchServer to provide search functionality on a web site. I want to crawl all pages on the site for links to follow but I want to exclude some pages from the index. I can't work out how to do this.

Specifically the website includes a shop that has its own product search and I am keeping this search for products and categories. The product pages have URLs like http://www.thesite/p/123 so I don't want to include any page like this in the search results. However some product pages reference background info pages and I want these to be included in the search index.

The problem I have is that the filter has no effect on the results - it doesn't filter out the /p/ and /c/ results. If I change the filter by unticking the negative box I get no results so it seems to be either the contents of the field or the filter criteria that is causing the problem.

I've tried adding a negative filter to the default query called search in the Query > Filter tab on the index with url:"http://www.thesite/p/*" but it seems that wildcards are not supported for query filters although they are supported for Crawler > Exclusion list filters.

I've tried adding a new field called urlField in Schema > Fields and populating it using an analyzer configured using the Whitespace Tokenizer and a regular expression (http://www.thesite/(c|p)/). When I use the Test button it seems to generate two tokens for my test URL http://www.thesite/p/123:

I'd hoped to be able to use the first one in a Query > Filter to exclude all the shop results and optionally be able to use the p (for product) or c (for category) if I need to search the product pages sometime in the future.

The urlShop field in the schema is set up as follows:

Indexed: yes
Stored: no (because I don't need the field back, just want to be able to filter on it)
TermVector: No
Analyzer: urlShop
Copy of: url

I've added urlFilter:"http://www.thesite/p/" to Query > Filters with the negative box ticked.

This seems to have no effect on the results when I use the default renderer.

To see whether it affects the returned results I unticked the negative box in the query filter I get no results in the default renderer. This leads me to believe that the urlShop field is not being populated but I'm not sure how to check this directly.

I would like to know whether there is an easier way to do this but if my approach makes sense in the context of OpenSearchServer please can you help me identify what's wrong?

The website is running under IIS and OpenSearchServer will be configured on the same server running in Tomcat.

Upvotes: 2

Views: 727

Answers (1)

Rob
Rob

Reputation: 1364

Finally figured this out...

Go to query and hit edit for your configured query. Then go to the filters tab. Add a query filter like this:

urlExact:"http://myurltoexclude*"

Check the "negative" box. Click add.

Now make sure to click "save in the tiny little button on the right hand side. This is the part I missed. The URLS are still in the DB and crawl, but at least they aren't returned in results.

Upvotes: 2

Related Questions