CODEWITHSUNDEEP
solr
TMS
TMS

Reputation: 23

How to get the total word count per document in SOLR?

I would like to retrieve some summary statistics from the text documents I have indexed in Solr. In particular, the word count per document.

For example, I have the following three documents indexed:

{
 "id":"1",
 "text":["This is the text in document 1"]},
{
 "id":"2",
 "text":["some text in document 2"]},
{
 "id":"3",
 "text":["and document 3"]}

I would like to get the total number of words per each individual document:

"1",7,
"2",5,
"3",3,

What query can I use to get such a result?

I am new to Solr and I am aware that I can use facets to get the count of the individual words over all documents using something like:
http://localhost:8983/solr/corename/select?q=*&facet=true&facet.field=text&facet.mincount=1
But how to get the total word count per document is not clear to me.

I appreciate your help!

Upvotes: 2

Views: 727

Answers (1)

uylmz
uylmz

Reputation: 1552

If you do a faceted search over id and an inner facet over text, the inner facet count will give the number of words in that document with that id. But text field type must be text_general or something equivalent (tokenized).

If you only want to count "distinct" words per document id, it is actually much easier:

{
"query": "*:*",
"facet": {
    "document": {
        "type": "terms",
        "field": "id",
        "facet": {
            "wordCount": "unique(message)"
        }
    }
}
}

Gives distinct word count per document. Following gives all words and all counts per document but it's up to you to sum them to get total amount (also it's an expensive call)

{
"query": "*:*",
"facet": {
    "document": {
        "type": "terms",
        "field": "id",
        "facet": {
            "wordCount": {
                "type": "terms",
                "field": "message",
                "limit": -1
            }
        }
    }
}
}

@MatsLindth's comment is something to consider too. Solr and you might not agree on what's a "word". Tokenizer is configurable to a point but depending on your needs it might not be very easy.

Upvotes: 3

Related Questions