CODEWITHSUNDEEP
lucene.nettokenlucene
ET.
ET.

Reputation: 1969

Extending TokenStream

I am trying to index into a document a field with one term that has a payload. Since the only constructor of Field that can work for me takes a TokenStream, I decided to inherit from this class and give the most basic implementation for what I need:

public class MyTokenStream : TokenStream
{
    TermAttribute termAtt;
    PayloadAttribute payloadAtt;
    bool moreTokens = true;
    public MyTokenStream()
    {
        termAtt = (TermAttribute)GetAttribute(typeof(TermAttribute));
        payloadAtt = (PayloadAttribute)GetAttribute(typeof(PayloadAttribute));
    }
    public override bool IncrementToken()
    {
        if (moreTokens)
        {
            termAtt.SetTermBuffer("my_val");
            payloadAtt.SetPayload(new Payload(/*bye[] data*/));
            moreTokens = false;
        }

        return false;
    }     
}

The code which was used while indexing:

        IndexWriter writer = //init tndex writer...

        Document d = new Document();
        d.Add(new Field("field_name", new MyTokenStream()));
        writer.AddDocument(d);
        writer.Commit();

And the code that was used during the search:

        IndexSearcher searcher = //init index searcher
        Query query = new TermQuery(new Term("field_name", "my_val"));
        TopDocs result = searcher.Search(query, null, 10);

I used the debugger to verify that call to IncrementToken() actually sets the TermBuffer. My problem is that the returned TopDocs instance returns no documents, and I cant understand why... Actually I started from TermPositions (which gives me approach to the Payload...), but it also gave me no results. Can someone explain to me what am I doing wrong? I am currently using Lucene .NET 2.9.2

Upvotes: 0

Views: 138

Answers (1)

Jf Beaulac
Jf Beaulac

Reputation: 5246

After you set the TermBuffer you need to return true from IncrementToken, you return false when you have nothing to feed the TermBuffer with anymore

Upvotes: 2

Related Questions