Drupal and Google Search Appliance (Google Mini)

Question

I have a Drupal site with pages indexed by a Google Mini search appliance.

Earlier in the week I noticed that a bunch of links were marked as indexed, but excluded because there was a 'print this page' link back to the same page and had a rel="nofollow". I took the nofollow out and let the GSA reindex the site 2 days ago.

Now, the pages in question are marked as indexed inside the GSA, but they are not showing up in the search results of the site.

I can search at /search/google_appliance/TERM and they do not show up. When I search for other terms, they do show up. In other words, I know that GSA is working.

When I search at /search/node/TERM [Drupal default search], I get the Drupal results which are different[pages with the term shows up]. This makes me pretty sure I'm hitting GSA.

Any ideas on why the newly indexed pages aren't showing up in GSA search?

EDIT/Solved: There were a couple of issues. Previously the search used an xslt to handle how it displayed the page, and where it sent the queries on the page when you hit submit (on the appliance, not the submit button on the site). The queries string was passed in the old format to the site, which then gave a 404 (same thing as if you do a search of bookstore.site.com, and origin.site.com). More of a ‘can’t get there from here’ sort of problem than anything having to do with searching. I’ve removed the xslt, so it just used the default google look and feel, and lets us do nice, generic searches against the appliance’s database.

However, there were still some weird search results coming back that the Drupal module could not parse and The logs were getting hit with simplexml_load_string() [function.simplexml-load-string]: ^ in \sites\all\modules\google_appliance\GoogleMini.php on line 318.

I experimented with some querystring variables and commenting out the line that sets the Output encoding and all seems to work. The line in question is in google_appliance.module on line 322:

$gm->setOutputEncoding('utf8');

Answer 1

I recently struggled with something similar.

One suggestion here -- pick a page which you know has the search term. Open the HTML of the page in your browser and make sure you see that term. Absolutely sure.

Next, take that URL and put it in as one of the starting pages in your crawl.

After the crawl, go into the Search Diagnostics and drill down to that page. Do you see it crawled? Okay, great, now go look at the cache of the page. Right below "Link to this page" ought to be a hyperlink called "Cached version." Look at that. You may be in for a surprise! I certainly was.

Answer 2

I am not 100% sure I got your question right. I am assuming here that:

1. What GSA is missing to index are the pages from which there is the link "print this page" (rather than assuming those pages are indexed and the problem is in indexing the printable version of such pages)
2. The following bit means that you can find other pages which contain other terms, and not that you can find the missing pages if you search them with another term.

I can search at /search/google_appliance/TERM and they do not show up. When I search for other terms, they do show up. In other words, I know that GSA is working

Please correct me if I misunderstood your question. Should I have got it wrong, please provide some more details about the terms you are using.

This is however what I I would do for identifying the source of the problem (although I would probably not do these in this precise order):

1. I would try to understand what are the distinctive elements of the "bad pages" (if any) that trigger the odd behaviour. It seems that you have already done some of this digging and consider the culprit to be the print link. Have you verified this by removing the link altogether and see if the pages get correctly indexed in this case?
2. I would check if there is any rule in robots.txt that might interfere with the indexing. GSA honors that file, so for example if your pages' URL is beginning with /admin/, those pages will be skipped.
3. I would check if my pages have some kind of access control restricting their view. Should this be the case, I would check that GSA has been configured for that. (The same applies for unpublished pages of course, where you have to be admin to see or index them with an external application).
4. I am not sure if GSA uses sitemap.xml to perform the indexing. However I would inspect the drupal generated sitemap.xml file (if any) to check for blatant errors like a priority set to 0, for example. If you haven't such file, and know that GSA uses it, I would try to generate one with the appropriate module and see if this solves the problem.
5. I would inspect the sitemap generated by GSA to see if it shows any blatant anomaly too. This would clearly not be the problem, but any kind of self-explanatory anomaly could put you on the right track.
6. I the problem is not specific to the page structure (see point #1 of this list) I would begin to systematically search what is the non-structural element that generates the error. Does a different theme solves the problem. Does deactivating a given module solves the problem? (Maybe the problem is with meta-tags? Maybe with the "print this page" module? Maybe a module sets the language of those pages to a different language than the rest of the site?). All of these are rather unlikely possibilities, but before smashing down the GSA with an sledgehammer I would try that too.
7. I would go through (probably for the Nth time) all the settings of my GSA.

All of the above - if I had the chance to - I would do it with a peer. He or she could help ruling out the "human factor" as source of the problem (i.e. that little checkbox in the configuration panel that to him/her is so paramount but that you never noticed before...).

If you manage to find out any more hints on what is going on, report them back here. If it is a problem on the drupal side I'm pretty sure me or somebody else of the excellent "drupalists" hanging around on SO will be able to help.

HTH!