Mahesh
Mahesh

Reputation: 61

How to set "orc.bloom.filter.fpp" ratio

How to set "orc.bloom.filter.fpp" based on my data I have enabled bloom filter for two columns and given fpp ratio is 0.02

These are the stats of bloomfilter enable columnn . is it good ?

Bloom filters for column 1:

  Entry 0: numHashFunctions: 7 bitCount: 95872 popCount: 10196 loadFactor: 0.1064 expectedFpp: 1.5387434E-7
  Entry 1: numHashFunctions: 7 bitCount: 95872 popCount: 10530 loadFactor: 0.1098 expectedFpp: 1.9282182E-7
  Entry 2: numHashFunctions: 7 bitCount: 95872 popCount: 10410 loadFactor: 0.1086 expectedFpp: 1.7795594E-7
  Entry 3: numHashFunctions: 7 bitCount: 95872 popCount: 10505 loadFactor: 0.1096 expectedFpp: 1.8963993E-7
  Entry 4: numHashFunctions: 7 bitCount: 95872 popCount: 10046 loadFactor: 0.1048 expectedFpp: 1.387106E-7
  Entry 5: numHashFunctions: 7 bitCount: 95872 popCount: 10056 loadFactor: 0.1049 expectedFpp: 1.3968005E-7
  Entry 6: numHashFunctions: 7 bitCount: 95872 popCount: 10617 loadFactor: 0.1107 expectedFpp: 2.0425385E-7
  Entry 7: numHashFunctions: 7 bitCount: 95872 popCount: 11361 loadFactor: 0.1185 expectedFpp: 3.2815075E-7
  Entry 8: numHashFunctions: 7 bitCount: 95872 popCount: 11267 loadFactor: 0.1175 expectedFpp: 3.096104E-7
  Entry 9: numHashFunctions: 7 bitCount: 95872 popCount: 10410 loadFactor: 0.1086 expectedFpp: 1.7795594E-7
  Entry 10: numHashFunctions: 7 bitCount: 95872 popCount: 9644 loadFactor: 0.1006 expectedFpp: 1.0422164E-7
  Entry 11: numHashFunctions: 7 bitCount: 95872 popCount: 10179 loadFactor: 0.1062 expectedFpp: 1.5208744E-7
  Entry 12: numHashFunctions: 7 bitCount: 95872 popCount: 10997 loadFactor: 0.1147 expectedFpp: 2.6126253E-7
  Entry 13: numHashFunctions: 7 bitCount: 95872 popCount: 11382 loadFactor: 0.1187 expectedFpp: 3.3242029E-7
  Entry 14: numHashFunctions: 7 bitCount: 95872 popCount: 11809 loadFactor: 0.1232 expectedFpp: 4.301792E-7
  Entry 15: numHashFunctions: 7 bitCount: 95872 popCount: 11280 loadFactor: 0.1177 expectedFpp: 3.1211977E-7
  Entry 16: numHashFunctions: 7 bitCount: 95872 popCount: 10678 loadFactor: 0.1114 expectedFpp: 2.126116E-7
  Entry 17: numHashFunctions: 7 bitCount: 95872 popCount: 9910 loadFactor: 0.1034 expectedFpp: 1.2608777E-7
  Entry 18: numHashFunctions: 7 bitCount: 95872 popCount: 10044 loadFactor: 0.1048 expectedFpp: 1.3851742E-7
  Entry 19: numHashFunctions: 7 bitCount: 95872 popCount: 9476 loadFactor: 0.0988 expectedFpp: 9.215794E-8
  Stripe level merge: numHashFunctions: 7 bitCount: 95872 popCount: 86652 loadFactor: 0.9038 expectedFpp: 0.49272844

Bloom filters for column 13:

  Entry 0: numHashFunctions: 6 bitCount: 81472 popCount: 41455 loadFactor: 0.5088 expectedFpp: 0.017354468
  Entry 1: numHashFunctions: 6 bitCount: 81472 popCount: 41380 loadFactor: 0.5079 expectedFpp: 0.017166926
  Entry 2: numHashFunctions: 6 bitCount: 81472 popCount: 41586 loadFactor: 0.5104 expectedFpp: 0.01768612
  Entry 3: numHashFunctions: 6 bitCount: 81472 popCount: 41232 loadFactor: 0.5061 expectedFpp: 0.016801808
  Entry 4: numHashFunctions: 6 bitCount: 81472 popCount: 41298 loadFactor: 0.5069 expectedFpp: 0.016963834
  Entry 5: numHashFunctions: 6 bitCount: 81472 popCount: 41556 loadFactor: 0.5101 expectedFpp: 0.017609702
  Entry 6: numHashFunctions: 6 bitCount: 81472 popCount: 41365 loadFactor: 0.5077 expectedFpp: 0.017129632
  Entry 7: numHashFunctions: 6 bitCount: 81472 popCount: 41525 loadFactor: 0.5097 expectedFpp: 0.01753104
  Entry 8: numHashFunctions: 6 bitCount: 81472 popCount: 41357 loadFactor: 0.5076 expectedFpp: 0.017109757
  Entry 9: numHashFunctions: 6 bitCount: 81472 popCount: 41434 loadFactor: 0.5086 expectedFpp: 0.01730178
  Entry 10: numHashFunctions: 6 bitCount: 81472 popCount: 41473 loadFactor: 0.509 expectedFpp: 0.017399732
  Entry 11: numHashFunctions: 6 bitCount: 81472 popCount: 41536 loadFactor: 0.5098 expectedFpp: 0.01755892
  Entry 12: numHashFunctions: 6 bitCount: 81472 popCount: 41578 loadFactor: 0.5103 expectedFpp: 0.017665721
  Entry 13: numHashFunctions: 6 bitCount: 81472 popCount: 41295 loadFactor: 0.5069 expectedFpp: 0.01695644
  Entry 14: numHashFunctions: 6 bitCount: 81472 popCount: 41591 loadFactor: 0.5105 expectedFpp: 0.017698886
  Entry 15: numHashFunctions: 6 bitCount: 81472 popCount: 41539 loadFactor: 0.5099 expectedFpp: 0.017566532
  Entry 16: numHashFunctions: 6 bitCount: 81472 popCount: 41335 loadFactor: 0.5074 expectedFpp: 0.017055225
  Entry 17: numHashFunctions: 6 bitCount: 81472 popCount: 41619 loadFactor: 0.5108 expectedFpp: 0.017770499
  Entry 18: numHashFunctions: 6 bitCount: 81472 popCount: 38566 loadFactor: 0.4734 expectedFpp: 0.01125064
  Stripe level merge: numHashFunctions: 6 bitCount: 81472 popCount: 81471 loadFactor: 1 expectedFpp: 0.9999263

what are the factors needs to consider while giving fpp ratio

Upvotes: 0

Views: 90

Answers (0)

Related Questions