What is the best way to implement simple text search

Question

Currently, I have a firestore database set up with a users collection and a funkoPops collection. The funkoPops collections has about 750 documents which are genres/series of funko pops. So an example of one document would be, document.id = "2014 Funko Pop Marvel Thor Series 2 Vinyl Figures" and that document has fields of funkoData, and genre. funkoData is an array of objects that are the funko pops that are within that 2014 series as you can see here. What is the best way to query for say name, where i would want to search the whole database docs for each funko pop in the funkoData array where field = name. Currently I am using somewhat of a custom function as from reading up on firestore, there is no way to search your database and just match part of a string, like you can see in the picture where if I searched Loki, firebase would not give me the funko pop that has the name = Loki - with helmet because it is not an exact match. Instead i have wrote this function here

const getFunkoPopQuery = async (req, res, next) => {
  try {
    console.log(req.params);
    const query = req.params.query.trim().toLowerCase();
    const funkoPops = await firestore.collection("funkoPops");
    const data = await funkoPops.get();
    const funkoArr = [];
    if (data.empty) {
      res.status(404).send("No Funko Pop records exsist");
    } else {
      data.forEach((doc) => {
        const funkoObj = new FunkoPop(doc.data().genre, doc.data().funkoData);
        funkoArr.push(funkoObj);
      });

      // genre matching
      let genreMatches = funkoArr.filter((funko) =>
        funko.genre.toLowerCase().includes(query)
      );
      if (genreMatches.length === 0) {
        genreMatches = `No funko pop genres with search: ${query}`;
      }
      // name & number matching
      let nameMatches = [];
      let numbMatches = [];
      funkoArr.forEach((funko) => {
        const genre = funko.genre;
        const funkoData = funko.funkoData;
        const name = funkoData.filter((data) =>
          data.name.toLowerCase().includes(query)
        );
        const number = funkoData.filter((data) =>
          data.number.toLowerCase().includes(query)
        );

        if (Object.keys(name).length > 0) {
          nameMatches.push({
            genre,
            name,
          });
        }
        if (Object.keys(number).length > 0) {
          numbMatches.push({
            genre,
            number,
          });
        }
      });

      if (nameMatches.length === 0) {
        nameMatches = `No funko pops found with search name: ${query}`;
      }
      if (numbMatches.length === 0) {
        numbMatches = `No funko pop numbers found with search: ${query}`;
      }

      const searchFinds = {
        genre: genreMatches,
        name: nameMatches,
        number: numbMatches,
      };

      res.send(searchFinds);
    }
  } catch (error) {
    res.status(400).send(error.message);
  }
};

Now this function works for full and substring of the data in the database and also handles multiple searches, meaning if i want to search for a series/genre, the name and or number of the funko pop, but it blows through the 50k read quota since its getting all the documents each time you search for something. Is there a better way of doing this or searching through firestore that would be better than my solution? I would need to obviously still search the database to match the series/genre, the name, and the number of the funko pop. This way if a person wants to search marvel, it will give all funko pops with the genre marvel in it, iron man would retrieve all iron man pops and genres for iron man, and searching a number would obviously yield all funko pops that have that number. Any help would be awesome!

Answer 1

After Algolia raised their prices 10x times and presented it as simplifying their pricing model it's time to think more before implementing.

• If you don't want to use third party APIs.

• If you feel like search for word presence is enough for you.

• If you don't need any phonetic algorithm be a part of your search.

You can use just Firebase.

Say you want your funkoData[].name to be searchable.

1. Concatenate all funkoData[].name together and lowercase it.

funkoData.map(v=>v.name).join(" ").toLowerCase()

2. Split your string by all types of whitespaces and special chars.

funkoData.map(v=>v.name).join(" ").toLowerCase().split(/[\s-\.,!?]/)

3. Filter for short words and empty strings.

funkoData.map(v=>v.name).join(" ").toLowerCase().split(/[\s-\.,!?]/).filter(v=>v.length>4)

4. Compute this using cloud functions.

Full example:

let searchable = documentData.funkoData
                    .map(v=>v.name)
                    .join(" ")
                    .toLowerCase()
                    .split(/[\s-\.,!?]/)
                    .filter(v=>v.length>4);
searchable = Array.from(new Set(searchable)); //remove duplicates
await documentRef.update({_searchArray: searchable});

Now. Search is possible with array-contains or array-contains-any:

const query = "Loki";
const normalizedQuery = query.trim().toLowerCase();
firestoreRef.collection("funkoPops").where('_searchArray', 'array-contains', normalizedQuery).get()

PS: If your searchable array has a potential to become huge, you can move it to a separate collection with the same ids and trigger search using https callable cloud function returning just ids of documents that fit search criteria. Why? Because using Admin SDK you have an option to skip specified fields to be returned in a query response.

Answer 2

I would suggest you to use Algolia for searching, it has got a lot of features, because in your case you are having 750 reads for one search and if the number of documents and/or searches increases then there will huge number of reads required and it would cost you a lot.

eg: if there are 750 documents and 100 searches a day, then you have 75,000 reads a day and in that case you are out of free quota which gives 50,000 reads per day

if you don't prefer algolia then you can use elastic search also.