Reputation: 61
How to set "orc.bloom.filter.fpp" based on my data I have enabled bloom filter for two columns and given fpp ratio is 0.02
These are the stats of bloomfilter enable columnn . is it good ?
Entry 0: numHashFunctions: 7 bitCount: 95872 popCount: 10196 loadFactor: 0.1064 expectedFpp: 1.5387434E-7
Entry 1: numHashFunctions: 7 bitCount: 95872 popCount: 10530 loadFactor: 0.1098 expectedFpp: 1.9282182E-7
Entry 2: numHashFunctions: 7 bitCount: 95872 popCount: 10410 loadFactor: 0.1086 expectedFpp: 1.7795594E-7
Entry 3: numHashFunctions: 7 bitCount: 95872 popCount: 10505 loadFactor: 0.1096 expectedFpp: 1.8963993E-7
Entry 4: numHashFunctions: 7 bitCount: 95872 popCount: 10046 loadFactor: 0.1048 expectedFpp: 1.387106E-7
Entry 5: numHashFunctions: 7 bitCount: 95872 popCount: 10056 loadFactor: 0.1049 expectedFpp: 1.3968005E-7
Entry 6: numHashFunctions: 7 bitCount: 95872 popCount: 10617 loadFactor: 0.1107 expectedFpp: 2.0425385E-7
Entry 7: numHashFunctions: 7 bitCount: 95872 popCount: 11361 loadFactor: 0.1185 expectedFpp: 3.2815075E-7
Entry 8: numHashFunctions: 7 bitCount: 95872 popCount: 11267 loadFactor: 0.1175 expectedFpp: 3.096104E-7
Entry 9: numHashFunctions: 7 bitCount: 95872 popCount: 10410 loadFactor: 0.1086 expectedFpp: 1.7795594E-7
Entry 10: numHashFunctions: 7 bitCount: 95872 popCount: 9644 loadFactor: 0.1006 expectedFpp: 1.0422164E-7
Entry 11: numHashFunctions: 7 bitCount: 95872 popCount: 10179 loadFactor: 0.1062 expectedFpp: 1.5208744E-7
Entry 12: numHashFunctions: 7 bitCount: 95872 popCount: 10997 loadFactor: 0.1147 expectedFpp: 2.6126253E-7
Entry 13: numHashFunctions: 7 bitCount: 95872 popCount: 11382 loadFactor: 0.1187 expectedFpp: 3.3242029E-7
Entry 14: numHashFunctions: 7 bitCount: 95872 popCount: 11809 loadFactor: 0.1232 expectedFpp: 4.301792E-7
Entry 15: numHashFunctions: 7 bitCount: 95872 popCount: 11280 loadFactor: 0.1177 expectedFpp: 3.1211977E-7
Entry 16: numHashFunctions: 7 bitCount: 95872 popCount: 10678 loadFactor: 0.1114 expectedFpp: 2.126116E-7
Entry 17: numHashFunctions: 7 bitCount: 95872 popCount: 9910 loadFactor: 0.1034 expectedFpp: 1.2608777E-7
Entry 18: numHashFunctions: 7 bitCount: 95872 popCount: 10044 loadFactor: 0.1048 expectedFpp: 1.3851742E-7
Entry 19: numHashFunctions: 7 bitCount: 95872 popCount: 9476 loadFactor: 0.0988 expectedFpp: 9.215794E-8
Stripe level merge: numHashFunctions: 7 bitCount: 95872 popCount: 86652 loadFactor: 0.9038 expectedFpp: 0.49272844
Entry 0: numHashFunctions: 6 bitCount: 81472 popCount: 41455 loadFactor: 0.5088 expectedFpp: 0.017354468
Entry 1: numHashFunctions: 6 bitCount: 81472 popCount: 41380 loadFactor: 0.5079 expectedFpp: 0.017166926
Entry 2: numHashFunctions: 6 bitCount: 81472 popCount: 41586 loadFactor: 0.5104 expectedFpp: 0.01768612
Entry 3: numHashFunctions: 6 bitCount: 81472 popCount: 41232 loadFactor: 0.5061 expectedFpp: 0.016801808
Entry 4: numHashFunctions: 6 bitCount: 81472 popCount: 41298 loadFactor: 0.5069 expectedFpp: 0.016963834
Entry 5: numHashFunctions: 6 bitCount: 81472 popCount: 41556 loadFactor: 0.5101 expectedFpp: 0.017609702
Entry 6: numHashFunctions: 6 bitCount: 81472 popCount: 41365 loadFactor: 0.5077 expectedFpp: 0.017129632
Entry 7: numHashFunctions: 6 bitCount: 81472 popCount: 41525 loadFactor: 0.5097 expectedFpp: 0.01753104
Entry 8: numHashFunctions: 6 bitCount: 81472 popCount: 41357 loadFactor: 0.5076 expectedFpp: 0.017109757
Entry 9: numHashFunctions: 6 bitCount: 81472 popCount: 41434 loadFactor: 0.5086 expectedFpp: 0.01730178
Entry 10: numHashFunctions: 6 bitCount: 81472 popCount: 41473 loadFactor: 0.509 expectedFpp: 0.017399732
Entry 11: numHashFunctions: 6 bitCount: 81472 popCount: 41536 loadFactor: 0.5098 expectedFpp: 0.01755892
Entry 12: numHashFunctions: 6 bitCount: 81472 popCount: 41578 loadFactor: 0.5103 expectedFpp: 0.017665721
Entry 13: numHashFunctions: 6 bitCount: 81472 popCount: 41295 loadFactor: 0.5069 expectedFpp: 0.01695644
Entry 14: numHashFunctions: 6 bitCount: 81472 popCount: 41591 loadFactor: 0.5105 expectedFpp: 0.017698886
Entry 15: numHashFunctions: 6 bitCount: 81472 popCount: 41539 loadFactor: 0.5099 expectedFpp: 0.017566532
Entry 16: numHashFunctions: 6 bitCount: 81472 popCount: 41335 loadFactor: 0.5074 expectedFpp: 0.017055225
Entry 17: numHashFunctions: 6 bitCount: 81472 popCount: 41619 loadFactor: 0.5108 expectedFpp: 0.017770499
Entry 18: numHashFunctions: 6 bitCount: 81472 popCount: 38566 loadFactor: 0.4734 expectedFpp: 0.01125064
Stripe level merge: numHashFunctions: 6 bitCount: 81472 popCount: 81471 loadFactor: 1 expectedFpp: 0.9999263
what are the factors needs to consider while giving fpp ratio
Upvotes: 0
Views: 90