CODEWITHSUNDEEP
architectureclient-serverkdb+
user236215
user236215

Reputation: 7556

What is the use of having a tickerplant in kdb plant?

I have a data source in form of a pub-sub cache that contains a large set of data with incremental updates. The data needs to be fed to a kdb databases (multiple instances due to size of data) that will normalize and parse the data into tables for consumption by front end processes. All the individual kdb instances also publish data to an aggregator kdb process for containing aggregated data. The client needs data at both the depth and aggregate level.I am working on a design for kdb plant to cover these requirements that will work efficiently, send quick updates to GUI, and for ease of maintenance.

Most of the designs I have seen always have a tickerplant subscribing to all updates? I am thinking of having each kdb instance subscribe to its data partition from cache and process it. Then all the instances normalize the data and send to the aggregator kdb process. The GUI gets updates from all the instances. What is really the utility of having a tickerplant? Do you see any issues with this design?

Any suggestions and comments welcome

Upvotes: 3

Views: 1670

Answers (1)

Manish Patel
Manish Patel

Reputation: 4491

It's a tried and tested approach. In general, aside from being super lean, tickerplants are useful because a realtime database (being entirely in-memory) can lock or die. But since the tickerplant also creates a log file, you can happily restart a realtime and it will read up to the point of last insertion and carry on listening to new inserts.

Your idea is sound, and is indeed what many people do to balance load. Exact architecture really depends on lots of things - how many machines you have, how much redundancy you want, etc as well as the client requirements. If I were you i'd have separate tickerplants that break the data up into table/sym partitions of datasets to make it all a bit more manageable, and then have specialised tickerplants that keep the depth/order book state on every tick.

Careful of slow consumers too, which can back a tickerplant up. A GUI will generally run on a slower PC somewhere else on the network and there could be 100s of GUIs subscribed to the same tickerplant... in which case it's probably a good idea to create lots of tickerplants subscribed in chains that effectively reduce the tcp load on the "main" tickerplant(s).

Chaining tickerplants like this is very useful when you're dealing with huge datasets with lots of (possibly slow) clients.

Upvotes: 2

Related Questions