CODEWITHSUNDEEP
pythonelasticsearch
Tomás Boccardo
Tomás Boccardo

Reputation: 33

TransportError(503, u'') when trying to use a recently created elasticsearch index

I'm creating an Elasticsearch index using Python API like this: 

from elasticsearch import Elasticsearch

es = Elasticsearch()

index_body = {"mappings": {".percolator": {"properties": {"message": {"type": "string", "analyzer": "english"}}}}}
# Creates the index if it doesn't exist
if not es.indices.exists('test'):
    es.indices.create(index='test', body=index_body)

print es.exists(index='test', id='1')

The index is created successfully, but when I check for the existence of a document inside the index it fails with this error:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
    print es.exists(index='test', id='1')
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 282, in exists
    self.transport.perform_request('HEAD', _make_path(index, doc_type, id), params=params)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 86, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 102, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.TransportError: TransportError(503, u'')

If I run this script a second time, with the index already created it works just fine. Does anyone have an idea of what could be going wrong?

Upvotes: 3

Views: 10166

Answers (2)

wbartussek
wbartussek

Reputation: 2028

You can add the parameter wait_for_active_shards (defaults to 1) like so:

es.indices.create(index='test', body=index_body, wait_for_active_shards=all)

See also: the elasticsearch-py manual for create(**kwargs)

Upvotes: 0

mark
mark

Reputation: 6559

When creating a new index, you need to wait until all shards are allocated.

The best way I know of how to do is this:

  1. fetch <your_index>/_status
  2. iterate over all indices.<your_index>.shards and verify that routing.state = STARTED everywhere
  3. Go to 1) unless all shards are started

Here's a (PHP) project which did this for unit tests:

protected function _waitForAllocation(Index $index)
{
    do {
        $settings = $index->getStatus()->get();
        $allocated = true;
        foreach ($settings['shards'] as $shard) {
            if ($shard[0]['routing']['state'] != 'STARTED') {
                $allocated = false;
            }
        }
    } while (!$allocated);
}

Wrong answer:

You've to give ES some space before your continue. ES is near realtime, so delays are to be expected. Especially when you're running your code sequentially with next to no delay.

I think you simply need to call the _refresh endpoint and you're covered.

In practice I've to do the same in my unit tests. They execute very fast and creating/pumping data/destroying indices takes time so in my setUp() I've a call to _refresh before handing over to the respective test*() method. And in some tests, where I index data, I also have to place _refresh calls.

Usually, during normal operation you don't need to call it shouldn't. Keep in mind that the default refresh_interval is 1s. If you're regularly update your index and expect sub-second updates to be reflected (I'm talking about e.g. _search), that's where you need to start.

Upvotes: 3

Related Questions