CODEWITHSUNDEEP
google-app-enginecsvgoogle-cloud-datastoreexport-to-csv
Mohamed Khamis
Mohamed Khamis

Reputation: 8029

Uploading CSV into GAE datastore

I am using GAE's high replication datastore. I previously downloaded a table in csv using this:

appcfg.py download_data --application=<your_app_id> --kind=<kind> --url=http://your_app_id.appspot.com/[remote_api_path] --filename=<data-filename>

Now I made some modifications into that CSV file, and I want to upload it to replace the online data. So I used this:

appcfg.py upload_data --config_file=bulkloader.yaml --filename=<data-filename> --kind=<kind> --url=http://your_app_id.appspot.com/[remote_api_path]

This successfully uploaded the data, but the problem is that now all of the newly uploaded keys (including references) look like this: name=99764 instead of id=99764

GAE didn't recognize that both are the same thing, so it didn't replace the ones out there, it added new ones. But now the references are now broken.

In the bulkloader.yaml, I have this:

- kind: Kind
  connector: csv
  connector_options:
    # TODO: Add connector options here--these are specific to each connector.
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

Is this part: transform.key_id_or_name_as_string causing the problem? if so then how do I fix it?

Upvotes: 1

Views: 750

Answers (1)

Gianni Di Noia
Gianni Di Noia

Reputation: 1578

I'm not sure but maybe this can help you. I have used this only for link a ReferenceProperty, not for replace data

- property: __key__
      external_name: key
      import_transform: transform.create_foreign_key('key', key_is_id=True)
      export_transform: transform.key_id_or_name_as_string

edit:

- property: category
  external_name: category
  import_transform: transform.create_foreign_key('Category', key_is_id=True)

'Category' in uppercase is the Model referenced from the property 'category'. In csv file I have a string like '123456' and in datastore the ID of Category Kind is '123456'. In this way GAE has associated the referenceproperty to the relative Kind otherwise it would be a simple string. Now in your case is different because you work with the Key for replace the data and I don't have experience in this sense.

Upvotes: 2

Related Questions