Sitecore custom index configure languages

Question

I created a custom Lucene index in a Sitecore 8.1 environment like this:

<?xml version="1.0"?>
<configuration xmlns:x="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
        <indexes hint="list:AddIndex">
          <index id="Products_index" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">
            <param desc="name">$(id)</param>
            <param desc="folder">$(id)</param>
            <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
            <configuration ref="contentSearch/indexConfigurations/ProductIndexConfiguration" />
            <strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
            </strategies>
            <commitPolicyExecutor type="Sitecore.ContentSearch.CommitPolicyExecutor, Sitecore.ContentSearch">
              <policies hint="list:AddCommitPolicy">
                <policy type="Sitecore.ContentSearch.TimeIntervalCommitPolicy, Sitecore.ContentSearch" />
              </policies>
            </commitPolicyExecutor>
            <locations hint="list:AddCrawler">
              <crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <Database>master</Database>
                <Root>/sitecore/content/General/Product Repository</Root>
              </crawler>
            </locations>
            <enableItemLanguageFallback>false</enableItemLanguageFallback>
            <enableFieldLanguageFallback>false</enableFieldLanguageFallback>
          </index>
        </indexes>
      </configuration>

      <indexConfigurations>
        <ProductIndexConfiguration type="Sitecore.ContentSearch.LuceneProvider.LuceneIndexConfiguration, Sitecore.ContentSearch.LuceneProvider">
          <initializeOnAdd>true</initializeOnAdd>
          <analyzer ref="contentSearch/indexConfigurations/defaultLuceneIndexConfiguration/analyzer" />
          <documentBuilderType>Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilder, Sitecore.ContentSearch.LuceneProvider</documentBuilderType>
          <fieldMap type="Sitecore.ContentSearch.FieldMap, Sitecore.ContentSearch">
            <fieldNames hint="raw:AddFieldByFieldName">
              <field fieldName="_uniqueid" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">
                <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
              </field>
              <field fieldName="key" storageType="YES" indexType="UNTOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider"/>
            </fieldNames>
          </fieldMap>
          <documentOptions type="Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilderOptions, Sitecore.ContentSearch.LuceneProvider">
            <indexAllFields>true</indexAllFields>
            <include hint="list:AddIncludedTemplate">
              <Product>{843B9598-318D-4AFA-B8C8-07E3DF5C6738}</Product>
            </include>
          </documentOptions>
          <fieldReaders ref="contentSearch/indexConfigurations/defaultLuceneIndexConfiguration/fieldReaders"/>
          <indexFieldStorageValueFormatter ref="contentSearch/indexConfigurations/defaultLuceneIndexConfiguration/indexFieldStorageValueFormatter"/>
          <indexDocumentPropertyMapper ref="contentSearch/indexConfigurations/defaultLuceneIndexConfiguration/indexDocumentPropertyMapper"/>
        </ProductIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

Actually a quite straightforward index that points to a root, includes a template and stores a field. It all works. But I need something extra:

• Limit the entries in the index to the latest version
• Limit the entries in the index to one language (English)

I know this can be easily triggered in the query as well, but I want to get my indexes smaller so they will (re)build faster.

The first one could be solved by indexing web instead of master, but I actually also want the unpublished ones in this case. I'm quite sure I will need a custom crawler for this.

The second one (limit the languages) is actually more important as I will also need this in other indexes. The easy answer is probably also to use a custom crawler but I was hoping there would be a way to configure this without custom code. Is it possible to configure an custom index to only include a defined set of languages (one or more) instead of all?

Answer 1

After some research I came to the conclusion that to match all the requirements a custom crawler was the only solution. Blogged the code: http://ggullentops.blogspot.be/2016/10/custom-sitecore-index-crawler.html

The main reason was that we really had to be able to set this per index - which is not possible with the filter approach. Also, when we added a new version the previous version had to be deleted from the index - also not possible without the crawler...

There is quite some code so I won't copy it all here but we had to override the DoAdd and DoUpdate methods of the SitecoreItemCrawler, and for the languages we also had to make a small override of the Update method.

Answer 2

You can easily create a custom processor that will let you filter what you want:

 namespace Sitecore.Custom
 {
    public class CustomInboundIndexFilter : InboundIndexFilterProcessor
    {
        public override void Process(InboundIndexFilterArgs args)
        {
            var item = args.IndexableToIndex as SitecoreIndexableItem;
            var language = Sitecore.Data.Managers.LanguageManager.GetLanguage("en");

            if (item != null && (!item.Item.Versions.IsLatestVersion() || item.Item.Language == language))
            {
                args.IsExcluded = true;
            }
        }
    }
 }

And add this processor into the config:

<pipelines>
   <indexing.filterIndex.inbound>
     <processor type="Sitecore.Custom.CustomInboundIndexFilter, Sitecore.Custom"></processor>
   </indexing.filterIndex.inbound>
</pipelines>