CODEWITHSUNDEEP
kdb+
Roman
Roman

Reputation: 617

Large number of connections to kdb

I have a grid with over 10,000 workers, and I'm using qpython to append data to kdb. Currently with 1000 workers, I'm getting ~40 workers that fail to connect and send data on the first try, top shows q is at 100% cpu when that happens. As I scale to 10k workers, the problem will escalate. The volume of data is only 100MBs. I've tried running extra slaves, but kdb tells me I can't use it with -P option, which I'm guessing I need to use qpython. Any ideas how to scale to support 10k workers. My current idea is to write a server in between that will buffer write requests and pass them to kdb, is there a better solution?

Upvotes: 0

Views: 245

Answers (1)

geocar
geocar

Reputation: 9305

It amazes me that you're willing to dedicate 10,000 cpus to Python but only a single one to Kdb.

Simply run more Kdb cores (on other ports) and then, enable another process to receive the updates from the ingestion cores. The tickerplant (u.q) is a good model for this.

Upvotes: 2

Related Questions