CODEWITHSUNDEEP
mongodbcsvimport
GriffiN
GriffiN

Reputation: 84

mongoimport CSV load error E11000 duplicate key error index

i have been trying to import the file from http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=293&DB_Short_Name=Air%20Carriers using all the fields.

for this i did the following command from terminal:

mongoimport -d local -c flightdata --type csv --file 1073242969_T_T100_SEGMENT_ALL_CARRIER_modified.csv --headerline

The problem is when I run the command i get the following output:

connected to: 127.0.0.1
Tue Dec 17 21:58:34.207 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.207 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.208 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.208 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.209 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.210 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.210 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.211 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:34.211 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:37.035         Progress: 8487360/56679929  14%
Tue Dec 17 21:58:37.035             30900   10300/second
Tue Dec 17 21:58:40.004         Progress: 16952746/56679929 29%
Tue Dec 17 21:58:40.004             61600   10266/second
Tue Dec 17 21:58:43.040         Progress: 24734143/56679929 43%
Tue Dec 17 21:58:43.040             89900   9988/second
Tue Dec 17 21:58:46.042         Progress: 32521258/56679929 57%
Tue Dec 17 21:58:46.042             118100  9841/second
Tue Dec 17 21:58:49.007         Progress: 40172878/56679929 70%
Tue Dec 17 21:58:49.007             145800  9720/second
Tue Dec 17 21:58:52.021         Progress: 48124064/56679929 84%
Tue Dec 17 21:58:52.021             174600  9700/second
Tue Dec 17 21:58:54.935 check 9 205716
Tue Dec 17 21:58:55.032 E11000 duplicate key error index: local.flightdata.$_id_  dup key: { : null }
Tue Dec 17 21:58:55.032 imported 205715 objects

But when i go check the database:

>use local
>db.flightdata.count()
1

so this means only the first record was loaded. Also tried with unquoted fields and quoted.

Any ideas as to what i'm doing wrong?

edit: headers are the following

"DEPARTURES_SCHEDULED","DEPARTURES_PERFORMED","PAYLOAD","SEATS","PASSENGERS","FREIGHT","MAIL","DISTANCE","RAMP_TO_RAMP","AIR_TIME","UNIQUE_CARRIER","AIRLINE_ID","UNIQUE_CARRIER_NAME","UNIQUE_CARRIER_ENTITY","REGION","CARRIER","CARRIER_NAME","CARRIER_GROUP","CARRIER_GROUP_NEW","ORIGIN_AIRPORT_ID","ORIGIN_AIRPORT_SEQ_ID","ORIGIN_CITY_MARKET_ID","ORIGIN","ORIGIN_CITY_NAME","ORIGIN_STATE_ABR","ORIGIN_STATE_FIPS","ORIGIN_STATE_NM","ORIGIN_COUNTRY","ORIGIN_COUNTRY_NAME","ORIGIN_WAC","DEST_AIRPORT_ID","DEST_AIRPORT_SEQ_ID","DEST_CITY_MARKET_ID","DEST","DEST_CITY_NAME","DEST_STATE_ABR","DEST_STATE_FIPS","DEST_STATE_NM","DEST_COUNTRY","DEST_COUNTRY_NAME","DEST_WAC","AIRCRAFT_GROUP","AIRCRAFT_TYPE","AIRCRAFT_CONFIG","YEAR","QUARTER","MONTH","DISTANCE_GROUP","CLASS","DATA_SOURCE",

could not find the ID field by itself

Upvotes: 1

Views: 2132

Answers (1)

Asya Kamsky
Asya Kamsky

Reputation: 42352

It looks like all of your records have the _id field which is supposed to be unique - but they all have the same value, 'null'. 'mongoimport' has the unfortunate property of only checking successful inserts for first ten and last document. So you are getting 10 errors for those 11 checks.

EDIT turns out that you are trying to import this into "local" database which exists for system use (replication) and you should not be putting your own collections there.

If you choose to name your database anything other than "local" your import will succeed just fine.

Upvotes: 1

Related Questions