CODEWITHSUNDEEP
cloudantdashdb
Anupam Gupta
Anupam Gupta

Reputation: 1

cloudant dashdb sync issue

We have created a warehouse with source database in cloudant, We had ran schema discovery process on near about 40,000 records initially.Our cloudant database consist of around 2 millions records. Now the issue we are facing is that we have got many records in _OVERFLOW Table in DashDB (means that they have got rejected ) with error like "[column does not exist in the discovered schema. Document has not been imported.]"

Seems to me the issue is that cloudant database which is actually result of dbcopy ,contains partials in the docs and as those partials are created internally by cloudant with value which we can judge only after the partials gets created like "40000000-5fffffff" in the dd doesn't get discovered by schema discovery process and now all docs which have undiscovered partials are being rejected by cloudant-dashdb sync.

Does anyone has any idea how to resolve it..

Upvotes: 0

Views: 92

Answers (2)

Max
Max

Reputation: 1

We had ran schema discovery process on near about 40,000 records initially. Our database consist of around 2 millions records

Do all these 2 millions share the same schema? I believe not.

"[column does not exist in the discovered schema. Document has not been imported.]"

It means that during your initial 40'000 records scan application didn't find any document with that field.

Let's say sequence of documents in your Cloudant db is:

  • 500'000 docs that match schema A
  • 800'000 docs that match schema B
  • 700'000 docs that match schema C

And your discovery process checked just first 40'000. It never got to schema B and C.

I would recommend to re-run discovery process and process all 2 millions records. It will take time, but will guarantee that all fields are discovered.

Upvotes: 0

Holger Kache
Holger Kache

Reputation: 186

The best option for you to resolve this is with a simple trick: feed the schema discovery algorithm exactly one document with the structure you want to create in your dashDB target.

If you can build such a "template" document ahead of time, have the algorithm discover that one and load it into dashDB. With the continuous replication from Cloudant to dashDB you can then have dbcopy load your actual documents into the database that serves as source for your cloudant-dashdb sync.

Upvotes: 0

Related Questions