Rebuilding ElasticSearch index using django-haystack rebuild_index command

Question

I am trying to get ElasticSearch / Haystack set up on my local dev environment (vagrant VM running Ubuntu 12.04), and I can't work out the re-indexing process.

ES is running, and I have created a new index (I am using elasticsearch-head to view index status in the browser). I can create a new index, and query it, so I know that ES is working.

My problem is with the Haystack rebuild_index command:

(.venv)vagrant@precise32:/app$ foreman run ./manage.py rebuild_index

WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y

Removing all documents from your index because you said so.
DEBUG Making a request equivalent to this: curl -XDELETE 'http://127.0.0.1:9200/test_app' -d '""'
INFO Starting new HTTP connection (1): 127.0.0.1
DEBUG "DELETE /test_app HTTP/1.1" 200 31
DEBUG response status: 200
DEBUG got response {u'acknowledged': True, u'ok': True}
DEBUG Making a request equivalent to this: curl -XPOST 'http://127.0.0.1:9200/test_app/_refresh' -d '""'
DEBUG "POST /test_app/_refresh HTTP/1.1" 404 66
DEBUG response status: 404
Failed to clear Elasticsearch index: (404, u'IndexMissingException[[test_app] missing]')
ERROR Failed to clear Elasticsearch index: (404, u'IndexMissingException[[test_app] missing]')
All documents removed.

Looking at this loggging - it seems as if Haystack is attempting to refresh an index that it has just deleted - which would always fail.

What am I doing wrong?

[UPDATE 1]

If I split the POST requests I can get this to run:

(.venv)vagrant@precise32:/app$ curl -XPOST 'http://127.0.0.1:9200/test_app/'
{"ok":true,"acknowledged":true}

(.venv)vagrant@precise32:/app$ curl -XPOST 'http://127.0.0.1:9200/test_app/_refresh' -d    '""'
{"ok":true,"_shards":{"total":10,"successful":5,"failed":0}}

[UPDATE 2]

Digging in to the code, the ES backend method that is called when running clear_index is:

    def clear(self, models=[], commit=True):
        [...]
        if not models:
            self.conn.delete_index(self.index_name)
        else:
            [...]
        if commit:
            self.conn.refresh(index=self.index_name)

Which looks wrong as it will call conn.refresh on the index that it has just deleted?

[UPDATE 3]

I think the above errors may be a red herring, as the management commands will ignore the errors and continue, giving this error, which I think is more serious:

(.venv)vagrant@precise32:/app$ foreman run ./manage.py update_index --verbosity=3
Skipping '<class 'django.contrib.auth.models.Permission'>' - no index.
Skipping '<class 'django.contrib.auth.models.Group'>' - no index.
Skipping '<class 'django.contrib.auth.models.User'>' - no index.
Skipping '<class 'django.contrib.contenttypes.models.ContentType'>' - no index.
Skipping '<class 'django.contrib.sessions.models.Session'>' - no index.
Skipping '<class 'django.contrib.sites.models.Site'>' - no index.
Skipping '<class 'django.contrib.admin.models.LogEntry'>' - no index.
Skipping '<class 'django.contrib.flatpages.models.FlatPage'>' - no index.
ERROR Error updating test_app using default
Traceback (most recent call last):
  File "/home/vagrant/.venv/src/django-haystack/haystack/management/commands/update_index.py", line 210, in handle_label
    self.update_backend(label, using)
  File "/home/vagrant/.venv/src/django-haystack/haystack/management/commands/update_index.py", line 239, in update_backend
    end_date=self.end_date)
  File "/home/vagrant/.venv/src/django-haystack/haystack/indexes.py", line 157, in build_queryset
    index_qs = self.index_queryset(using=using)
TypeError: index_queryset() got an unexpected keyword argument 'using'
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vagrant/.venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/.venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/vagrant/.venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/vagrant/.venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/home/vagrant/.venv/src/django-haystack/haystack/management/commands/update_index.py", line 184, in handle
    return super(Command, self).handle(*items, **options)
  File "/home/vagrant/.venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 341, in handle
    label_output = self.handle_label(label, **options)
  File "/home/vagrant/.venv/src/django-haystack/haystack/management/commands/update_index.py", line 210, in handle_label
    self.update_backend(label, using)
  File "/home/vagrant/.venv/src/django-haystack/haystack/management/commands/update_index.py", line 239, in update_backend
    end_date=self.end_date)
  File "/home/vagrant/.venv/src/django-haystack/haystack/indexes.py", line 157, in build_queryset
    index_qs = self.index_queryset(using=using)
TypeError: index_queryset() got an unexpected keyword argument 'using'

[UPDATE 4]

OK - so it's my fault, I was using an old search_indexes.py file, and my index_queryset() method was incorrect. I won't close this as it may be useful for others.

Answer 1

Answering this one myself - albeit it's just an admission of my own stupidity in this one.

I carried a search_indexes.py file from the 1.x version of Haystack into a new branch of our project that was using the 2.x version of Haystack, which is configured slightly differently. In the new version, the index_queryset() method now requires a new using parameter (defaults to None). The older version didn't require this.

The new signature should therefore be:

def index_queryset(self, using=None):
    pass