CODEWITHSUNDEEP
exceptionmilvusimportdata
XuKai
XuKai

Reputation: 11

pymilvus.exceptions.ParamError: <ParamError: (code=1, message=('Field data size misaligned

I want to inport some data into milvus database。

sample data:

{'CN': '计费办法名称', 'EN': 'Fee Method Name', 'ID': '0867', 'DESC': '计费办法的名称。'}

but I got this error:

Failed to insert batch starting at entity: 0/808
Traceback (most recent call last):
File "/home/itzuser/pythonProject/load_to_milvus.py", line 97, in \<module\>
MilvusWrapper.from_texts(texts,embed_model=embed_model,
File "/home/itzuser/pythonProject/milvus_wrapper.py", line 811, in from_texts
vector_db.add_texts(texts=texts, metadatas=metadatas)
File "/home/itzuser/pythonProject/milvus_wrapper.py", line 456, in add_texts
raise e
File "/home/itzuser/pythonProject/milvus_wrapper.py", line 450, in add_texts
res = self.col.insert(insert_list, timeout=timeout, \*\*kwargs)
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 497,   in insert
res = conn.batch_insert(
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/decorators.py", line 127, in  handler
raise e from e
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/decorators.py", line 123, in handler
return func(\*args, \*\*kwargs)
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/decorators.py", line 162, in handler
return func(self, \*args, \*\*kwargs)
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/decorators.py", line 102, in handler
raise e from e
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/decorators.py", line 68, in handler
return func(\*args, \*\*kwargs)
File "/usr/local/python/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 572, in batch_insert

raise err from err File "/usr/local/python/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 552, in batch_insert request = self._prepare_batch_insert_request( File "/usr/local/python/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 536, in _prepare_batch_insert_request else Prepare.batch_insert_param(collection_name, entities, partition_name, fields_info) File "/usr/local/python/lib/python3.10/site-packages/pymilvus/client/prepare.py", line 520, in batch_insert_param return cls._parse_batch_request(request, entities, fields_info, location) File "/usr/local/python/lib/python3.10/site-packages/pymilvus/client/prepare.py", line 488, in _parse_batch_request

raise ParamError(
pymilvus.exceptions.ParamError: \<ParamError: (code=1, message=('Field data size misaligned for field \[ID\] ', 'got size=\[807\] ', 'alignment size=\[808\]'))\>

What is this reason for this error.

Upvotes: 1

Views: 3675

Answers (1)

Ben Cox
Ben Cox

Reputation: 1483

The error message here is particularly unhelpful, but this error relates to the fact that the different rows of your insert have different fields in them.

As an example of how this might happen, you might have used a splitter on a corpus of data and that splitter adds metadata to the resultant splits, and then you're aiming to put the splits, their metadata, and their embedded vectors into the database, but this splitter doesn't always add all the same metadata fields to every single split.

pymilvus takes the first row of data, uses that as the canonical length (808 in your case) and then validates that every other row is that length. It comes across a row of length 807 in your case, and that makes it error.

Why does pymilvus do this? Ultimately, it comes down to this: Milvus doesn't support nullable fields at the moment. Every field must be present in every row of the data you're adding.

Upvotes: 1

Related Questions