CODEWITHSUNDEEP
kdb+
Utsav
Utsav

Reputation: 5918

How to replay the TP log file when getting wsfull error even for -11!(-2;`path)

I'm getting wsfull error even when trying to get the count of messages from TPlog file using:

-11!(-2;`:/uts/tplog_2020.07.07) /- wsfull

How can we get the count of the tplog file in this case?
Is upserting the messages in chunks to the partition only option to splay the tplog file in this case?

Edit:

Kdb Version - 3.2
Log File size - 24G
Memory Stats -
q).Q.w[] 
used| 35590543840
heap| 35500589056
peak| 35500589056
wmax| 0
mmap| 0
mphy| 270267080704
syms| 2807
symw| 158810

I tried running:

upd:insert(Version 3.2)
-11!(1000000;`:/uts/tplog_2020.07.07); /- output 1000000
count trade /- output count from table - 88471241
-11!(2000000;`:/uts/tplog_2020.07.07); /- output 2000000
count trade/- output count from table - 88471241
-11!(3000000;`:/uts/tplog_2020.07.07); /- wsfull

Tried with version 4.0
-11!(-2;`:/uts/tplog_2020.07.07); /- type error

When I try to get last few messages from table, it throws 'Segmentation Fault'

-11!(1000000;`:/uts/tplog_2020.07.07); /- output - 88471241
-5#trade /- Segmentation Fault, session closed

My first guess is that tplog file is corrupt. It will be great help if someone can tell me how can we get the faulty message from the tplog file which is causing type error?

Upvotes: 0

Views: 478

Answers (2)

terrylynch
terrylynch

Reputation: 13657

Following on from the discussion in the comments, I'm starting to think that something isn't right with how you're writing chunks to the log file. Replaying 100 chunks is supposed to return 100, replaying 200 chunks is supposed to return 200 chunks (assuming a well-formed log file). Here's a function you can use to inspect individual chunks:

{`counter set 0;.z.ps:{$[counter=desiredChunk;`savedChunk set x;counter+:1]};-11!(1+desiredChunk::y;x);.z.ps:{value x}}[`:tplog2020.07.01;5]

Pass the logfile and the chunk you want to extract (starting from 0 for the first chunk). Once you've run this you'll have a global variable "savedChunk" which contains your chunk and which you can inspect. A single chunk should look like:

q)savedChunk
`upd
`myTable
(0D05:34:00.186409000;`foo;1.23;1234;1b;.....)

/or if you write to the tplog in batches it could look like:
q)savedChunk
`upd
`myTable
(0D05:34:00.186409000 0D05:34:00.186409000;`foo`bar;1.23 4.56;1234 5678;10b;.....)

I would take a look at your chunks, starting at chunk zero, to see if they're well formed. Then look at chunk 12683 to see if anything isn't right there.

Is it possible that your chunks contain more than one upd/table? (which would be a custom implementation that I've never seen before).

Upvotes: 2

Adam Bonham
Adam Bonham

Reputation: 615

What KDB version are you using?

In 4.0 they added integrity checks for this.

"Further integrity checks have been added to streaming execute -11!x to avoid wsfull or segfault on corrupted/incomplete log files."

https://code.kx.com/q/releases/ChangesIn4.0/#miscellaneous

Upvotes: 0

Related Questions