fiware NGSI-LD orion-ld context broker didn't send all entities for a type as expected

Question

I filled a fiware orion-ld with ca 90.000 addresses (ID's like 'urn:ngsi-ld:address:Testtown:12345:Haus-Knipp-Str:15'). If I read all entities for type "Address" from orion-ld broker via url (with offset settings, step by step with limit 1000) like "http://{{orion}}/ngsi-ld/v1/entities?type=Address&offset=0&limit=1000" I receive at any time the expected answer from orion-ld until I try to request "http://{{orion}}/ngsi-ld/v1/entities?type=Address&offset=25000&limit=1000". Then the unexpected answer is "[]". It should be filled with 1000 entities.

If I delete all entities for type "Address" stepwise by 1000 entities (each time I request the first 1000 entities and delete them afterward), I am able to delete >98000 entities. So all 98000 entities were in the orion-ld. But I wasn't able to read all entities from orion-ld.

I try to find out the last readable entity, it is "http://{{orion}}/ngsi-ld/v1/entities?type=Address&offset=25248&limit=1" the next request failed: "http://{{orion}}/ngsi-ld/v1/entities?type=Address&offset=25249&limit=1" If I try to get entities above the 25248 it failed at all.

If I check, if all 98000 ID's for type "Address" are in orion-ld, I request them by url "http://{{orion}}/ngsi-ld/v1/entities/{{id}}". I receive for each ID the expected answer. So all ID's are in the orion-ld. Only the read request failed, starting with offset>25248.

Is there an error in orion-ld, or an unknown threshold, which needs to be handled in another way?

---

Today I create 100,000 entities of type "TestTestTest" like:

{
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld",
    "id": "TestTestTest:20000",
    "type": "TestTestTest",
    "name": {
        "type": "Property",
        "value": 20000
    }
},

If I try to read all entities of type "TestTestTest" again I found another threshold: 87637

read all entities for type TestTestTest with offset=86000 and limit=1000 --> successfull: read 1000 values
read all entities for type TestTestTest with offset=87000 and limit=1000 --> error: read 0 values
read all entities for type TestTestTest with offset=87000 and limit=500 --> successfull: read 500 values
read all entities for type TestTestTest with offset=87500 and limit=500 --> error: read 0 values
read all entities for type TestTestTest with offset=87500 and limit=250 --> error: read 0 values
read all entities for type TestTestTest with offset=87500 and limit=125 --> successfull: read 125 values
read all entities for type TestTestTest with offset=87625 and limit=125 --> error: read 0 values
read all entities for type TestTestTest with offset=87625 and limit=62 --> error: read 0 values
read all entities for type TestTestTest with offset=87625 and limit=31 --> error: read 0 values
read all entities for type TestTestTest with offset=87625 and limit=15 --> error: read 0 values
read all entities for type TestTestTest with offset=87625 and limit=7 --> successfull: read 7 values
read all entities for type TestTestTest with offset=87632 and limit=7 --> error: read 0 values
read all entities for type TestTestTest with offset=87632 and limit=3 --> successfull: read 3 values
read all entities for type TestTestTest with offset=87635 and limit=3 --> successfull: read 3 values
read all entities for type TestTestTest with offset=87638 and limit=3 --> error: read 0 values
read all entities for type TestTestTest with offset=87638 and limit=1 --> error: read 0 values
try to read the last readable entity with type TestTestTest with offset=87637 and limit=1 --> id=TestTestTest:87637
try to read an entity after the last readable entity with type TestTestTest with offset=87638 and limit=1 --> []
ERROR, the requested amount of entities (iOffset=87637) seems not the expected count.

I use the following version of orion-ld on a docker environment:

{
    "orionld version": "post-v0.8.1",
    "orion version": "1.15.0-next",
    "uptime": "7 d, 12 h, 2 m, 55 s",
    "git_hash": "nogitversion",
    "compile_time": "Wed Dec 15 15:12:50 UTC 2021",
    "compiled_by": "root",
    "compiled_in": "",
    "release_date": "Wed Dec 15 15:12:50 UTC 2021",
    "doc": "https://fiware-orion.readthedocs.org/en/master/"
}

It would be nice if you can help in this case. Thanks, Knigge

Answer 1

So, I filled my DB with 27000 entities and tested:

ldcurl GET "/entities?type=T&offset=26000&count=true&limit=1000"

(ldcurl is a tiny script of mine that adds host, port, accept-header etc)

I had no problem, got this output for the sample test with offset 26000

HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 138003
Content-Type: application/json
Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"
NGSILD-Results-Count: 27000
Date: Thu, 27 Jan 2022 19:31:14 GMT


[
  {
    "id": "urn:ngsi-ld:entities:E26001",
    "type": "T",
    "A1": {
      "type": "Property",
      "value": "E26002:A1"
    }
  },
  {
    "id": "urn:ngsi-ld:entities:E26002",
    "type": "T",
    "A1": {
      "type": "Property",
      "value": "E26003:A1"
    }
  },
  ...

I tested with what's currently in the develop branch and also with what's in release/1.0.0.

Look me up on skype, same user name as here in SOF, perhaps we can look at your problem together, as I don't seem to be able to reproduce it myself.

Meanwhile, I'll make my entities more similar to yours, see if I have more luck that way.

Answer 2

Yes, this really seems to be a bug in Orion-LD. As a bug, we need it as an issue on the github repo. Could you file an issue .... here ? (that way you'll be informed on updates)