Symfony 2 FOSElasticaBundle : elastic search documents are not automatically synchronized with entities : need to run ' fos:elastica:populate'

Question

I am working with FOSElasticaBundle on a basic entity nammed 'Foobar'. I have defined 1 index nammed 'foobar' with 2 types 'foobar_published' and 'foobar_draft'. The problem is that my Foobar entities are not automatically synchronized with elastic search, I need to run after each modification on a Foobar entity, the following command : "php app/console fos:elastica:populate". After that, my documents in elasticsearch are as they should be, this means that elastic documents are like the related entities in my database.

How should I proceed to automatically get my documents in elasticsearch 'up to date' whithout running all the time 'php app/console fos:elastica:populate' ?

Is there something wrong in my config ?

The version information from my composer.json : "friendsofsymfony/elastica-bundle": "~3.0.0"

The elasticsearch mapping is describe here :

fos_elastica:
    clients:
        default: { host: %elasticsearch_host%, port: %elasticsearch_port% }
    indexes:
        foobar:
             client:     default
             types:
                foobar_published:
                    mappings:
                        state: ~
                        field1: ~
                    persistence:
                        driver:   orm
                        model:    App\MyBundle\Entity\Foobar
                        provider:
                            query_builder_method: createIsIndexablePublishedQueryBuilder
                        listener: {immediate: ~}
                        finder:   ~

                foobar_draft:
                    mappings:
                        state: ~
                        field1: ~
                    persistence:
                        driver:   orm
                        model:    App\MyBundle\Entity\Foobar
                        provider:
                            query_builder_method: createIsIndexableDraftQueryBuilder
                        listener: {immediate: ~}
                        finder:   ~

The Foobar entity is defined as follow :

/**
 * @ORM\Entity(repositoryClass="App\MyBundle\Repository\FoobarRepository")
 */
class Foobar extends BaseEntity
{
    const STATE_TO_BE_INDEXED_IN_ELA = 'published';
    const STATE_DRAFT_TO_BE_INDEXED_IN_ELA = 'draft';
    const STATE_NOT_TO_BE_INDEXED_IN_ELA = 'unpublished';

    /**
     * @ORM\Column(type="string")
     */
    private $state;
    /**
     * @ORM\Column(type="string")
     */
    private $field1;

    /**
     * @return mixed
     */
    public function getState()
    {
        return $this->state;
    }

    /**
     * @param mixed $state
     */
    public function setState($state)
    {
        $this->state = $state;
    }

    /**
     * @return mixed
     */
    public function getField1()
    {
        return $this->field1;
    }

    /**
     * @param mixed $field1
     */
    public function setField1($field1)
    {
        $this->field1 = $field1;
    }


}

The FoobarRepository looks like this :

class FoobarRepository extends EntityRepository
{
    public function createIsIndexablePublishedQueryBuilder()
    {
        $qb = $this->createQueryBuilder('foobar');
        $qb
            ->where('foobar.state = :state')
            ->setParameter('state', Foobar::STATE_TO_BE_INDEXED_IN_ELA);
        return $qb;
    }


    public function createIsIndexableDraftQueryBuilder()
    {
        $qb = $this->createQueryBuilder('foobar');
        $qb
            ->where('foobar.state = :state')
            ->setParameter('state', Foobar::STATE_DRAFT_TO_BE_INDEXED_IN_ELA);
        return $qb;
    }
}

The test method in a controller :

 public function indexAction()
    {
        $em = $this->get("doctrine.orm.default_entity_manager");

        $foobar = new Foobar();
        $foobar->setField1('should be indexed in ela');
        $foobar->setState(Foobar::STATE_TO_BE_INDEXED_IN_ELA);

        $foobar2 = new Foobar();
        $foobar2->setField1('should NOT be indexed in ela');
        $foobar2->setState(Foobar::STATE_NOT_TO_BE_INDEXED_IN_ELA);


        $foobar3 = new Foobar();
        $foobar3->setField1('should  be indexed in ela');
        $foobar3->setState(Foobar::STATE_DRAFT_TO_BE_INDEXED_IN_ELA);

        $em->persist($foobar);
        $em->persist($foobar2);
        $em->persist($foobar3);
        $em->flush();


        $existingFoobar = $em->getRepository('AppMyBundle:Foobar')->findAll();
        foreach ($existingFoobar as $foo) {
            echo $foo->getId() . '  : ' . $foo->getField1() . '  ' . $foo->getState();
            echo '<br>';
        }


        return new Response('done');


    }

And finally, I monitor elasticsearch with this query :

curl -XGET "http://localhost:9200/foobar/_search" -d'
{
   "query": {
      "match_all": {}
   }
}'

Answer 1

After a few hours, the solution came from this ressource that allows you to define a callback that returns if an entity must be indexed in elastic search.

So the config now looks like this :

fos_elastica:
    clients:
        default: { host: %elasticsearch_host%, port: %elasticsearch_port% }
    indexes:
        foobar:
             client:     default
             types:
                foobar_published:
                    indexable_callback: 'isPublished'
                    mappings:
                        state: ~
                        field1: ~
                    persistence:
                        listener: ~
                        driver:   orm
                        model:    App\MyBundle\Entity\Foobar
                        provider:
                            query_builder_method: createIsIndexablePublishedQueryBuilder
                        finder:   ~

                foobar_draft:
                    indexable_callback: 'isDraft'
                    mappings:
                        state: ~
                        field1: ~
                    persistence:
                        listener: ~
                        driver:   orm
                        model:    App\MyBundle\Entity\Foobar
                        provider:
                            query_builder_method: createIsIndexableDraftQueryBuilder
                        finder:   ~

And I also implement the callbacks method in the Foobar entity :

class Foobar extends BaseEntity {
...   


    public function isPublished()
    {

        if ($this->getState() === self::STATE_TO_BE_INDEXED_IN_ELA) {
            return true;
        }
        return false;
    }


    public function isDraft()
    {

        if ($this->getState() === self::STATE_DRAFT_TO_BE_INDEXED_IN_ELA) {
            return true;
        }
        return false;
    }
}

And now everything is running fine, the documents are synchronised with documents in the right index ant types. I use those queries to checks results :

curl -XPOST "http://localhost:9200/foobar/foobar_published/_search?size=300"
curl -XPOST "http://localhost:9200/foobar/foobar_draft/_search?size=300"

Note : I also change the listener property in my config file to 'listener: ~' instead of 'listener: {immediate: ~}'.

I test on insert,delete and update statement and everything now works fine !