Lucene/Solr - Indexing publications/texts

Question

I want to be able to search publications with facets. These documents will be annotated so I will upload the annotation to the solr instance. The annotation will have fields which are the terms in the document. Here is an example:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<add>
    <doc>
        <field name="Title">High Glucose Increases the Expression of Inflammatory Cytokine Genes in
Macrophages Through H3K9 Methyltransferase Mechanism.</field>
        <field name="Cytokine">INTERFERON </field>
        <field name="Cytokine">CYTOKINE </field>
        <field name="Cytokine">CYTOKINE</field>
        <field name="Cytokine">MEC</field>
        <field name="Cytokine">EPA</field>
        <field name="Cytokine">DIA</field>
        <field name="Cytokine">FIC</field>
        <field name="Cytokine">CYTOKINES</field>
        <field name="Cytokine">INTERLEUKIN-6 </field>
        <field name="Cytokine">INTERLEUKIN</field>
        <field name="Cytokine">IL-12P40</field>
        <field name="Cytokine">IL-12</field>
        <field name="Cytokine">IL-1</field>
        <field name="Cytokine">P40</field>
        <field name="Cytokine">MACROPHAGE INFLAMMATORY PROTEIN-1</field>
        <field name="Cytokine">MACROPHAGE INFLAMMATORY PROTEIN</field>
    </doc>
</add>

These terms are all from a Cytokine ontology.

I want be able to set the facet as Cytokine, then select the term and find all of the documents that contain the selected term.

Here is the catch:

• I want to be able to store the location of said term found in the document (it can show up in multiple locations. So I can highlight later). All of these locations are stored in the annotation.
• I want to be able to select one of the terms from the facet and also bring up documents that contains that terms synonyms but not upload it as a term in the facet (or it being distinguished as a synonym some how (like subcategory)). e.g. automobile and car
• I want to be able to do a cross search e.g. find documents that contain MEC and EPA.
• I have a list of terms I do want to index and want to search the documents by. These terms have synonyms which I have entered into the synonyms.txt file.

Also When a term shows up multiple times in the document the annotation has multiple instances of this term with different locations, how should I handle this? Will solr automatically deal with duplication and not give me the documents twice?

One more thing: What about uploading the entire publication to solr, and indexing it on the predefined list of terms?

Answer 1

I understand that, you have synonyms and a search term should be verified directly and also with synonyms and return the results. Let me know if I got it.

If you have all the synonyms while indexing, then you can index them as multi valued field and search on that field.

Faceting is for searching, where the results are grouped.