Muhammad Taimur Zahid
Muhammad Taimur Zahid

Reputation: 41

Where are HNSW and IVFFLAT indexes for pgvector extension of postgres created and stored?

I am new to postgres and its pgvector extension and might not have clear understanding of the concept. I am running a benchmark against some cloud databases for postgres which support pgvector extension and I am wondering where is the index created; either in RAM or disk, where is index stored and what happens if the memory available is less than index size? Although I have read that pgvector supports creation of index that is greater than memory available but there is no concrete evidence in the documentation. The pgvector extension version is 0.6.0. Two cases come in my mind when the available memory is less than index size:-

  1. Index is created on maintenance_work_mem and imported in shared memory (shared_buffers) while the surplus index is stored on the disk
  2. Index is created on maintenance_work_mem and stored on disk alongwith the database files and then imported into the shared memory (shared_buffers). The index is partially loaded into the memory since the available memory is less than index size and relies on disk for the remaining portion of index.

My hunch is that the second case makes more sense since there's a possibility to lose index if database restarts. I have run the benchmark with maintenance_work_mem and shared_buffers less than index size and it works. Any help in clearing this idea with appropriate references would be greatly appreciated.

Upvotes: 4

Views: 225

Answers (0)

Related Questions