Reputation: 66
I'm developing a Forex trading application using Node.js, where I need to handle real-time updates for around 600 currency pairs. The data comes from a FIX protocol engine and needs to be updated in both Redis (for real-time updates to clients) and a MySql database (for persistence).
Problem: The server experiences high CPU and memory usage, leading to crashes. The main issue seems to be the frequent and numerous database updates.
const updateSymbolData = async (message) => {
const entries = message.Body.NoMDEntries;
const symbol = message.Body.Symbol.replace(/\.ecn|\.O|\.N/g, "");
let bestBid = { price: 0, size: 0 };
let bestAsk = { price: Infinity, size: 0 };
entries.forEach((entry) => {
if (entry.MDEntryType === "Bid" && entry.MDEntryPx > bestBid.price) {
bestBid = {
price: entry.MDEntryPx,
size: entry.MDEntrySize,
};
} else if (entry.MDEntryType === "Offer" && entry.MDEntryPx < bestAsk.price) {
bestAsk = {
price: entry.MDEntryPx,
size: entry.MDEntrySize,
};
}
});
if (bestBid.price !== 0 && bestAsk.price !== Infinity) {
await redisClient.set(`marketData:${symbol}`, JSON.stringify({ bestBid, bestAsk }));
// Database update
const query = `
UPDATE forex_symbols
SET best_bid = $1, best_ask = $2
WHERE symbol = $3
`;
await queryAsync(query, [bestBid.price, bestAsk.price, symbol]);
}
};
Questions: How can I optimize database updates to handle high-frequency updates efficiently? Is there a better way to manage real-time updates to minimize CPU and memory usage? What are best practices for synchronizing real-time data with a persistent database in such high-frequency scenarios?
Environment:
Node.js Redis MySql FIX Protocol for real-time data Any suggestions or best practices would be greatly appreciated!
Current Setup:
Upvotes: 0
Views: 42