CODEWITHSUNDEEP
pythonvector-databasemilvus
Tim Spann
Tim Spann

Reputation: 503

How to Use Memory Mapping and Other Techniques to Reduce Memory Usage

I have a sample dataset of 2M vectors with 512 dimensions, each vector being a float32. I have an id field that is int64.

So, the size of raw vectors is 2M * 512 * 4 = 4GB.

The collection is memory-mapped. I have three collections with FLAT, IVF_FLAT, and HNSW indexes on the single vector field in each collection

Simply loading the collection consumes 4GB of memory, despite being memory-mapped. Querying post-loading consumes up to 5-6GB of memory, despite being memory-mapped. Releasing the collection does clear the memory - so clearly the data resides on the disk.

I have set the queryNode.mmap.mmapEnabled to true in values.yaml.

index_type, params = 'FLAT', {}
# index_type, params = 'IVF_FLAT', {'nlist': 2000}
# index_type, params = 'HNSW', {'M': 32, 'efConstruction': 200}

# Define a collection schema
eid_field = FieldSchema(name="eid", dtype=DataType.INT64, is_primary=True, description="embedding id")
embedding_field = FieldSchema(name="content_embedding", dtype=DataType.FLOAT_VECTOR, dim=512, description="content embedding")

# Set enable_dynamic_field to True if you need to use dynamic fields. 
schema = CollectionSchema(fields=[eid_field, embedding_field], auto_id=False, enable_dynamic_field=True, description=f"{index_type} collection")

# Define index parameters
index_params = client.prepare_index_params()

index_params.add_index(
    field_name="eid",
    index_type="STL_SORT"
)

index_params.add_index(
    field_name="content_embedding", 
    index_type=index_type,
    metric_type="L2",
    params=params
)

collection_name = f"benchmarking_{index_type}"
params_str = '_'.join([f"{k}_{v}" for k,v in sorted(params.items())])
if len(params_str)>0:
    collection_name += f"_{params_str}"
if not client.has_collection(collection_name=collection_name):
    client.create_collection(
        collection_name=collection_name,
        dimension=512,  # The vectors we will use in this demo has 512 dimensions
        schema=schema,  # The collection schema we defined above
        index_params=index_params,
        properties={'mmap.enabled': True}
    )

See: https://milvus.io/docs/mmap.md

Upvotes: 0

Views: 48

Answers (1)

Tim Spann
Tim Spann

Reputation: 503

To make this work change memory mapping.

I added the following section in values.yaml:

extraConfigFiles:
  user.yaml: |+
    queryNode:
      enableDisk: true
      cache:
        memoryLimit: 536870912
      mmap:
        mmapEnabled: true
        vectorField: true
        vectorIndex: true
        scalarField: true
        scalarIndex: true
        growingMmapEnabled: true

When using the collection


collection = Collection(name=collection_name)
collection.set_properties({'mmap.enabled': True})
collection.alter_index(
    index_name="embedding",
    extra_params={"mmap.enabled": True}
)
client.load_collection(collection_name=collection_name)

By setting querynode resources.limits.memory to 4Gi, I was able to query a collection of 8M float32 vectors seamlessly with both IVF_FLAT and HNSW.

Upvotes: 0

Related Questions