CODEWITHSUNDEEP
backupkdb+
Jason Livesay
Jason Livesay

Reputation: 6377

How to recover HDB with tplogs?

Our system is currently backing up tplogs to S3. From what I have read, simply making sure these files are in the place that kdb expects them will allow for recovery if there is an issue with RDB during the day.

However, I did not see an explanation of how to use the tplogs to recover HDB. I tempted to create another backup system to sync the hdb folders to S3 also. That will be more work to set up and use at least double the storage, as well as being redundant. So if its not necessary then I would like to avoid that extra step.

Is there a way to recover the HDB from the tplogs in the event that we lose access to our HDB folders, or do I need to add another backup system for the HDB folders? Thanks.

Upvotes: 2

Views: 599

Answers (2)

shi
shi

Reputation: 511

To replay log file to HDB. 

.Q.hdpf[`::;get `:tpLOgFile;.z.d;`sym]

As per my experience if you are building a HDB from TP logfile load tp log file using get function and save it using dpft that is efficient.

If you want to use -11! function then you have to provide a upd function(-11! read each row from tp log file and call upd function then insert data to in memory table) to load data in memory and then save data on disk.

In both case you have to load data in memory but by using get function you can skip upd function call

-11! function is efficient for building the RDB because it will not load the full log file.

For more details read Below link http://www.firstderivatives.com/downloads/q_for_Gods_July_2014.pdf

Upvotes: 2

Jason Livesay
Jason Livesay

Reputation: 6377

OK, actually found a forum answer to a similar question, with a script for replaying log files.

https://groups.google.com/forum/#!topic/personal-kdbplus/E9OkvJKGrLI

Jonny Press says:

The usual way of doing it is to use -11! to replay the log file. A basic script would be something like 

// load schema
\l schema.q

// define upd
upd:insert

// replay log file 
-11!`:schema2015.07.09

// save
.Q.hdpf[`::;`:hdb;2015.07.09;`sym]

This will read the full log file into memory. So you will need to have RAM available.

TorQ has a TP log replay script:

https://github.com/AquaQAnalytics/TorQ/blob/master/code/processes/tickerlogreplay.q

Upvotes: 0

Related Questions