CODEWITHSUNDEEP
google-cloud-firestoregcloud
Diego Barreiro Clemente
Diego Barreiro Clemente

Reputation: 11

Firestore indexes composite create is Not working

I am trying to create a Vector index to find the top nearest neighbors given an input embedding Vector. Following Firestore's tutorial on this they provide the command to create the Vector index.

"To create a vector index, use gcloud alpha firestore indexes composite create:"

gcloud alpha firestore indexes composite create \
--collection-group=collection-group \
--query-scope=COLLECTION \
--field-config field-path=vector-field,vector-config='vector-configuration' \
--database=database-id

If I accomodate this command to my case following the examples they provide:

gcloud alpha firestore indexes composite create \
--collection-group=cars \
--query-scope=COLLECTION \
--field-config field-path=embedding,vector-config='{"dimension": "768", "flat": "{}"}' \

And every time I get an error saying:

argument --field-config: invalid <googlecloudsdk.command_lib.util.apis.yaml_command_schema_util._MessageFieldType object at 0x000001761A469250> value: "field-path=embedding,vector-config='{dimension:"

and if I remove the whitespace in the dictionary in vector-config I get:

argument --field-config: Expecting {dimension:768,flat:{}} to be json or arg_dict format

So I am unsure if this is a bug on my end or if the documentation is not accurate.

Upvotes: 1

Views: 232

Answers (1)

Chris du Plessis
Chris du Plessis

Reputation: 1370

It took me some time, but I gave up on having inline parameters because I assume there are multiple levels of escaping required. I had a look at the unofficial repo of the GCloud SDK but could not find what I was looking for. I did find code for the parameters for the vector-config but could not find out where to go from there.

Solution:

Make a flags YAML file (I just called it x.yaml) with the following content:

--collection-group: product_information
--query-scope: COLLECTION
--field-config: field-path=Description_embedding,vector-config={"dimension":"256","flat":"{}"}
--database: (default)

Replace Description_embedding with your field to the vector and change the dimension if need be. Change product_information to the collection of your choice. This will make use of your default Firestore database called (default).

Then in the directory of x.yaml, execute the following:

gcloud beta firestore indexes composite create --flags-file=x.yaml

I had six documents in my collection when I ran this command. Note that this worked on a Windows machine and I believe it should work on Linux and Mac systems too, but, again, I'm not going to play around with escaping everything.

After two minutes, I found this in the console: Screenshot of the index in the console

To view it in the terminal, hit gcloud beta firestore indexes composite list: The result of indexes composite list

You can delete the index in the console. I tried with the gcloud SDK command, but I killed it after it was hanging for 10 minutes.

You then need to wait a few more minutes before trying to create it again.

To create a composite vector index for a field called product_name for example, use gcloud beta firestore indexes composite create --field-config=order=ASCENDING,field-path="product_name" --flags-file=x.yaml

Upvotes: 0

Related Questions