CODEWITHSUNDEEP
databasekdb+
works
works

Reputation: 117

KDB table sharding based on column values

I'm currently working on scaling my kdb+/q setup for multiple risk engines, each identified by a unique risk_engine_id. The architecture involves having a common table named risk_engine_table that is subscribed to a tickerplant. The sample table schema is as follows:

risk_engine_table:([]time:`timestamp$(); sym:`long$(); risk_engine_id:`int$())

In this setup, risk_engine_id ranges from 0 to 10, and each risk engine needs to subscribe to a subset of rows in risk_engine_table based on its unique ID.

My current approach is to deploy individual RDBs for each risk engine. Each RDB is expected to subscribe to the entire risk_engine_table from one single tickerplant but filter rows based on the risk_engine_id column.So for example RDB1 will have all the rows inside the risk_engine_table but the column risk_engine_id will have value 1 for all rows inside that RDB1.

Also note that each RDB will be having their own HDB so issue coming from end of day write down from multiple RDB's for same table is solved. We are already succesfully running this.

Is this a valid approach ? So my question is can I modify .u.sub inside my risk engine RDB to do this ?

Upvotes: 0

Views: 86

Answers (1)

Alexander Unterrainer-DefconQ
Alexander Unterrainer-DefconQ

Reputation: 703

This sounds like a reasonable approach, however, .u.sub is defined in the TP, not in the RDB. What you want to do is to change the upd function (normally defined as a simple insert) in each individual RDB to filter for the specific risk engine id you are filtering for. However, this will not be the optimal solution as the TP still has to publish the full table and it will add latency given the network traffic. A better approach would be to filter on the TP for only the records with the risk_engine_id interested. Something similar is already implemented in the plain vanilla TP. .u.sel can be used to filter for specific symbols. So every subscriber can either subscribe to the full universe of symbols or a specific subset. You can change the TP to filter for risk_engine_id. I would generalise this behaviour and change the functionality of the TP, storing the handle and it's corresponding filter functions as well as other configs in .u.w. Hope this answers your question

Upvotes: 2

Related Questions