What's wrong with my pdf writer?

Question

I'm writing code to produce pdfs (from postscript of course), and I've tried to follow the spec as best I could. But imagemagick's identify says there's something wrong with my xref table.

Can anyone see where/what my problem is?

$ echo quit | gsnd -q pw.ps dancingmen.ps | identify -
   **** Warning:  An error occurred while reading an XREF table.
   **** The file has been damaged.  This may have been caused
   **** by a problem while converting or transfering the file.
   **** Ghostscript will attempt to recover the data.

   **** This file had errors that were repaired or ignored.
   **** Please notify the author of the software that produced this
   **** file that it does not conform to Adobe's published PDF
   **** specification.

-=>/tmp/magick-16940kBciKvHuOrD3 PBM 612x792 612x792+0+0 16-bit Bilevel Gray 61KB 0.000u 0:00.000

My pdf (made with ghostscript on Linux, single LF eols):

%PDF-1.3

1 0 obj
<< /Type /Catalog 
/Pages 2 0 R 
>> 
endobj

2 0 obj
<< /Kids [ 3 0 R ] 
/Type /Pages 
/Count 1 
>> 
endobj

3 0 obj
<< /Contents [ 4 0 R ] 
/MediaBox [ 0.0 0.0 612.0 792.0 ] 
/Type /Page 
/Parent 2 0 R 
>> 
endobj

4 0 obj
<< /Length 1287 
>> 
stream
2.0 4.0 m 2.0 3.9 l 2.05516 3.9 2.1 3.94484 2.1 4.0 c 2.1 4.05516 2.05516 4.1 2.0 4.1 c 1.94484 4.1 1.9 4.05516 1.9 4.0 c 1.9 3.94484 1.94484 3.9 2.0 3.9 c f 2.0 3.6 m 2.5 3.1 l S -2.0 3.6 m -1.5 3.1 l S 2.0 3.1 m 2.4 2.8 l 2.1 2.4 l 2.2 2.35 l S -2.0 3.1 m -1.7 2.6 l -1.5 2.8 l S 2.0 3.9 m 2.0 3.6 l 2.0 3.1 l S 3.0 4.0 m 3.0 3.9 l 3.05516 3.9 3.1 3.94484 3.1 4.0 c 3.1 4.05516 3.05516 4.1 3.0 4.1 c 2.94484 4.1 2.9 4.05516 2.9 4.0 c 2.9 3.94484 2.94484 3.9 3.0 3.9 c f 3.0 3.6 m 3.5 3.1 l S -3.0 3.6 m -2.5 4.1 l S 3.0 3.1 m 3.0 2.3 l 3.15 2.3 l S -3.0 3.1 m -3.0 2.3 l -2.85 2.3 l S 3.0 3.9 m 3.0 3.6 l 3.0 3.1 l S 4.0 4.0 m 4.0 3.9 l 4.05516 3.9 4.1 3.94484 4.1 4.0 c 4.1 4.05516 4.05516 4.1 4.0 4.1 c 3.94484 4.1 3.9 4.05516 3.9 4.0 c 3.9 3.94484 3.94484 3.9 4.0 3.9 c f 4.0 3.6 m 4.5 4.1 l S -4.0 3.6 m -3.5 4.1 l S 4.0 3.1 m 4.3 2.6 l 4.5 2.8 l S -4.0 3.1 m -3.7 2.6 l -3.5 2.8 l S 4.0 3.9 m 4.0 3.6 l 4.0 3.1 l S 5.0 4.0 m 5.0 3.9 l 5.05516 3.9 5.1 3.94484 5.1 4.0 c 5.1 4.05516 5.05516 4.1 5.0 4.1 c 4.94484 4.1 4.9 4.05516 4.9 4.0 c 4.9 3.94484 4.94484 3.9 5.0 3.9 c f 5.0 3.6 m 5.5 4.1 l 5.5 4.3 l 5.6 4.3 l 5.6 4.2 l 5.5 4.2 l S -5.0 3.6 m -4.5 3.1 l S 5.0 3.1 m 5.4 2.8 l 5.1 2.4 l 5.2 2.35 l S -5.0 3.1 m -4.6 2.8 l -4.9 2.4 l -4.8 2.35 l S 5.0 3.9 m 5.0 3.6 l 5.0 3.1 l S
endstream
endobj

xref
0 4
0000000000 65535 f 
0000000010 00000 n 
0000000063 00000 n 
0000000127 00000 n 
0000000234 00000 n 
trailer
<<
  /Root 1 0 R
  /Size 4
>>
startxref
1581
%%EOF

For reference, this is the postscript drawing which is being converted.

Update: I've fixed several of the issues mentioned: missing xref keyword, %%EOF instead of $$EOF. Same error from identify, but chrome's viewer actually shows me an image (really small, in the lower left corner because I haven't dealt with graphics state yet).

link to file

link to newer file with single content stream

Output from ghostscript:

$ echo pstack quit | gsnd -q data/pw.ps data/dancingmen.ps | gsnd -sDEVICE=ps2write -dPDFDEBUG -dPDFSTOPONERROR -
GPL Ghostscript 9.18 (2015-10-05)
Copyright (C) 2015 Artifex Software, Inc.  All rights reserved.
This software comes with NO WARRANTY: see the file PUBLIC for details.
   **** Warning:  An error occurred while reading an XREF table.
   **** The file has been damaged.  This may have been caused
   **** by a problem while converting or transfering the file.
   **** Ghostscript will attempt to recover the data.
<<
/Root 1 0 R
/Size 4 >>
%Resolving: [1 0]
<<
/Type /Catalog /Pages 2 0 R
>>
endobj
%Resolving: [2 0]
<<
/Kids [
3 0 R
]
/Type /Pages /Count 1 >>
endobj
%Resolving: [3 0]
<<
/Contents [
4 0 R
]
/MediaBox [
0.0 0.0 612.0 792.0 ]
/Type /Page /Parent 2 0 R
>>
endobj
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [1 0]
%Resolving: [1 0]
%Resolving: [1 0]
%Resolving: [1 0]
%Resolving: [2 0]
Processing pages 1 through 1.
Page 1
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [3 0]
%Resolving: [3 0]
%Resolving: [3 0]
%Resolving: [3 0]
%Resolving: [3 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [1 0]
%Resolving: [2 0]
%Resolving: [4 0]
<<
/Length 1288 >>
stream
%FilePosition: 270
endobj
2.0 4.0 m
2.0 3.9 l
2.05516 3.9 2.1 3.94484 2.1 4.0 c
2.1 4.05516 2.05516 4.1 2.0 4.1 c
1.94484 4.1 1.9 4.05516 1.9 4.0 c
1.9 3.94484 1.94484 3.9 2.0 3.9 c
f
2.0 3.6 m
2.5 3.1 l
S
-2.0 3.6 m
-1.5 3.1 l
S
2.0 3.1 m
2.4 2.8 l
2.1 2.4 l
2.2 2.35 l
S
-2.0 3.1 m
-1.7 2.6 l
-1.5 2.8 l
S
2.0 3.9 m
2.0 3.6 l
2.0 3.1 l
S
3.0 4.0 m
3.0 3.9 l
3.05516 3.9 3.1 3.94484 3.1 4.0 c
3.1 4.05516 3.05516 4.1 3.0 4.1 c
2.94484 4.1 2.9 4.05516 2.9 4.0 c
2.9 3.94484 2.94484 3.9 3.0 3.9 c
f
3.0 3.6 m
3.5 3.1 l
S
-3.0 3.6 m
-2.5 4.1 l
S
3.0 3.1 m
3.0 2.3 l
3.15 2.3 l
S
-3.0 3.1 m
-3.0 2.3 l
-2.85 2.3 l
S
3.0 3.9 m
3.0 3.6 l
3.0 3.1 l
S
4.0 4.0 m
4.0 3.9 l
4.05516 3.9 4.1 3.94484 4.1 4.0 c
4.1 4.05516 4.05516 4.1 4.0 4.1 c
3.94484 4.1 3.9 4.05516 3.9 4.0 c
3.9 3.94484 3.94484 3.9 4.0 3.9 c
f
4.0 3.6 m
4.5 4.1 l
S
-4.0 3.6 m
-3.5 4.1 l
S
4.0 3.1 m
4.3 2.6 l
4.5 2.8 l
S
-4.0 3.1 m
-3.7 2.6 l
-3.5 2.8 l
S
4.0 3.9 m
4.0 3.6 l
4.0 3.1 l
S
5.0 4.0 m
5.0 3.9 l
5.05516 3.9 5.1 3.94484 5.1 4.0 c
5.1 4.05516 5.05516 4.1 5.0 4.1 c
4.94484 4.1 4.9 4.05516 4.9 4.0 c
4.9 3.94484 4.94484 3.9 5.0 3.9 c
f
5.0 3.6 m
5.5 4.1 l
5.5 4.3 l
5.6 4.3 l
5.6 4.2 l
5.5 4.2 l
S
-5.0 3.6 m
-4.5 3.1 l
S
5.0 3.1 m
5.4 2.8 l
5.1 2.4 l
5.2 2.35 l
S
-5.0 3.1 m
-4.6 2.8 l
-4.9 2.4 l
-4.8 2.35 l
S
5.0 3.9 m
5.0 3.6 l
5.0 3.1 l
S

   **** This file had errors that were repaired or ignored.
   **** Please notify the author of the software that produced this
   **** file that it does not conform to Adobe's published PDF
   **** specification.

%Resolving: [2 0]
%Resolving: [1 0]

Update: Sigh. I suppose it's best if I show the code. This program is intended to hook into certain drawing operators of postscript and capture paths and produce a pdf file of the contents. I'm ignoring the quality of the output, in particular transformation matrices, for now.

/prompt {} def
<<

/.create-pdf-data {  % called at start
    install-operator-overrides
}

/.create-pdf-page {  % called at showpage
    1 /PageNumber +=
    << /Type /Page
       /Parent pdf-object-names /Pages get create-ref
       /MediaBox [gsave newpath clippath pathbbox grestore]
       /Contents []
    >>
    current-page-name dup 3 1 roll create-object
    pdf-object-names exch get create-ref add-to-pages-kids

    [ display-list {
        exch pop
        create-content-stream
    } for-each ]
    { ( ) exch strcat strcat } reduce
    add-content-to-page
}
/current-page-name {
    (Page) PageNumber  as-string strcat
}
/current-page {
    pdf-objects pdf-object-names current-page-name get get
}

/.output-pdf {    % called at quit
    /OutputFileName where { pop OutputFileName }{ (%stdout) } ifelse
    (w) file write-pdf
    pstack
}

/operator-overrides <<
   %/start                   .create-pdf-data
    /stroke    ({ mark-path  /S cvx ] display  //super//call })
    /fill      ({ mark-path  /f cvx ] display  //super//call })
    /showpage  ({            .create-pdf-page  //super//call })
    /quit      ({            .output-pdf       //super//call })
>>

/install-operator-overrides {
    operator-overrides {
        1 index load
        dup /super exch def
            type /arraytype eq { /exec load }{ /dummyproc cvx } ifelse  
            /call exch def
        cvx exec  userdict 3 1 roll put
    } forall
    userdict /dummyproc {} put
}


/PageNumber 0
/+= { dup load 3 2 roll add store }

/write-pdf {
    /f exch def
    (1.3) write-header
    write-body
    write-xref-table
    write-trailer
}

/pdf-output-file-position 0
/write-header {
    /pdf-output-file-position 0 store
    (%PDF-) .w .w \n \n
}

/write-body {
    write-objects-and-save-positions
}

/write-objects-and-save-positions {
    pdf-objects {
        1 index save-position
        write-object
    } for-each
}

/write-xref-table {
    (xref) .w \n
    pdf-output-file-position /xref-position exch def
    (0 ) .w pdf-object-positions length 1 sub .n \n
    0 format-10 .w ( 65535 f ) .w \n
    pdf-object-positions {
        write-xref-table-row
    } for-each
}
/write-xref-table-row {
    exch pop format-10 .w
      ( 00000 n ) .w \n
}
/format-10-string 20 string
/format-10 {
    format-10-string cvs
    (0000000000) 0  10 3 index length sub getinterval
    exch strcat
}

/write-trailer {
    (trailer) .w \n
    (<<) .w \n
    (  /Root 1 0 R) .w \n
    (  /Size ) .w pdf-objects length 1 sub .n \n
    (>>) .w \n
    (startxref) .w \n
    xref-position .n \n
    (%%EOF) .w \n
}


/create-content-stream {
    to-string-with-spaces
    %dup length ==only ( ) print  ==
}

/write-object {
    exch .n ( 0 obj) .w \n
    dup write-dict
    pdf-streams exch 2 copy known { write-stream }{ pop pop } ifelse
    (endobj) .w \n \n
}

/write-stream {
    (stream) .w \n
    get .w \n
    (endstream) .w \n
}

/write-dict {
    (<< ) .w
    { exch write-thing write-thing \n } forall
    (>> ) .w \n
}

/write-thing {
    +is-ref   { write-ref      }{
    +is-name  { write-name     }{
    +is-array { write-array    }{
    +is-null  { pop (null ) .w }{
                .n ( ) .w
    } ifelse } ifelse } ifelse } ifelse
}

/write-ref {
    ref .n ( 0 R ) .w
}

/write-name {
    dup xcheck not { (/) .w } if
    .n ( ) .w
}

/write-array {
    ([ ) .w
    { write-thing } forall
    (] ) .w
}

/+is-ref   { dup is-ref   }
/+is-name  { dup is-name  }
/+is-array { dup is-array }
/+is-null  { dup is-null  }

/is-string { type /stringtype eq }
/is-array { type /arraytype eq }
/is-name  { type  /nametype eq }
/is-null  { type  /nulltype eq }
/is-ref   { +is-name { is-ref-format }{ pop false } ifelse }
/is-ref-format  { ref-check-string cvs 0 1 getinterval (&) eq }
/ref-check-string 20 string

/ref { 10 string cvs rest cvi }
/create-ref { (&) exch 10 string cvs strcat cvn }

/mark-path {    [    { /m } { /l } { /c } { /h } pathforall  }

/display {  add-to-display-list  }
/display-list <<
    0 null
>>
/add-to-display-list {  display-list dup 3 1 roll length exch put  }
/clear-display-list { /display-list << 0 null >> store }

/pdf-objects << % integer keys
    0 null
    1 << /Type /Catalog  /Pages /&2           >>
    2 << /Type /Pages    /Kids  []   /Count 0 >>
>>
/pdf-object-names << % integer values
    /Catalog 1
    /Pages   2
>>
/pdf-object-positions << % integer keys
    0 null
>>
/pdf-streams <<
>>

/create-object { % dict name
    exch pdf-objects dup length 3 2 roll put
    pdf-object-names exch pdf-objects length 1 sub put
}
/object { % name -> dict
    pdf-object-names exch get  pdf-objects exch get
}
/save-position {
    pdf-object-positions exch pdf-output-file-position put
}
/Pages {
    pdf-objects pdf-object-names /Pages get get
}


/add-content-to-page {
    << 
        /Length 2 index length 1 add
    >> dup 3 2 roll pdf-streams 3 1 roll put
    /current-content create-object
    pdf-object-names /current-content get create-ref
    current-page /Contents 2 copy get [ exch {}forall counttomark 4 add -1 roll ] put
}

/add-to-pages-kids { % ref
    Pages /Kids 2 copy get [ exch {}forall counttomark 4 add -1 roll ] put
    Pages /Count 2 copy get 1 add put
}



/.w { f exch  dup length /pdf-output-file-position +=  writestring }
/.n { dup is-string not { .n-string cvs } if  .w }
/.n-string 100 string
/\n { (\n) .w }
/to-string-with-spaces {  {as-string} map {( ) exch strcat strcat} reduce  }
/map { 1 index xcheck 3 1 roll [ 3 1 roll forall ] exch { cvx } if }
/reduce { exch dup first exch rest 3 -1 roll forall }
/first { 0 get }
/rest { 1 1 index length 1 sub getinterval }
/as-string { 20 string cvs dup length 13 gt { 0 7 getinterval } if }
/strcat { 2 copy length exch length add string dup 4 2 roll
  3 copy pop 0 exch  putinterval  exch length exch putinterval }
/for-each { % dict proc     key(int) value  *proc*
    1 1 3 index length 1 sub   % d p 1 1 lim
    [ 6 5 roll                   % p 1 1 lim [ d
    1 /index cvx /get cvx        % p 1 1 lim [ d 1 index get
    9 8 roll /exec cvx ] cvx       % 1 1 lim { d 1 index get p exec }
    for
}

>>
{ dup {
    dup type /arraytype ne {
        def
    }{  % Dict name proc
        [ 3 index /begin cvx
          3 -1 roll {} forall
          /end cvx
        ] cvx
        def
    } ifelse
} forall pop
} pop
begin



.create-pdf-data

Answer 1

Sigh, ran out of space in the comments again....

It would help to put the file somewhere, rather than pasting it. PDF files are binary and length calculations depend on things like CR/LF pairs, meaning that each /Length could potentially be incorrect and its not possible to tell from looking at the pasted file.

Similarly the xref table offsets could be incorrect. In fact the offset for entry 1 looks incorrect, even assuming LF EOLs, but its not possible to be certain from the pasted file.

Note the error message is from Ghostscript (which IM uses to deal with PDF files). You would probably get more information if you just fed the PDF file to Ghostscript in the first place. You could also try setting -DPDFDEBUG and -dPDFSTOPONERROR, the combination will print out which object GS is dealing with and what it thinks is the problem (if there's a PostScript error). Other PDF problems usually send some kind of back-channel output.

Notice that the Ghostscript message references the xref table as the problem:

**** Warning: An error occurred while reading an XREF table.

So I suspect your xref table is incorrect (also see below, object 0).

Non-breaking, but not best practice:

xref entry 0, the head of the linked list of free objects, has an offset of 0000000028 should be 0.

Your file seems to end $$EOF instead of %%EOF.

Its normal practice to place binary in a comment on line 2 in order to force applications to treat the file as binary when transmitting

Better to elide the Resources dictionary than use a null object, its smaller.

Similarly, better to have a single Contents stream (despite recent Adobe engines producing multiple streams) again because its smaller.

Obviously this is an early work in progress, I'm sure you will deal with these in time.

If you'll post the actual PDF file somewhere I can take a look.

[edit]

So the first problem is that the xref table subsection is incorrect. The subsection starts with 2 numbers, the initial index, and the number of entries in the table. The xref table has 5 entries starting from index 0 and going up to index 4. The subsection says

0 4

Correcting that to 0 5 leads us to the next problem, the Size entry in the trailer dictionary is 4, and should be 5.

But Ghostscript still complains.

The final problem is that the startxref offset is incorrect. Currently this is:

startxref 1581

But the actual byte offset of the 'xref' keyword is byte 1576.

If I correct all 3 of these problems then Ghostscript opens the file without complaint. It already did render the content of course (very tiny because there's no CTM operations) but now it doesn't have to fix the file.