Unpacking AS400 packed decimal (BCD) - possibly borked by EBCDIC conversion?

Question

I'm getting files transferred from an AS/400 to our Windows (SBS 2003) via FTP. The files are fixed-width data. The text appears fine, but some of the fields are packed decimals, which when unpacked give bad values. My assumption is that there's an implicit EBCDIC->ASCII conversion happening, which is converting the packed bytes too. However, doing the reverse conversion and unpacking them still gives bad values...sometimes.

My guess is the codepage they're using is slightly different (so when I convert back to EBCDIC it's not quite the same), but I've no idea how to find out what codepage they are using (their IT people are being...mildly recalcitrant...otherwise they could just do the FTP in binary mode and skip all these issues).

Here is some sample data - these are supposed to unpack to 8-digit numbers which are actually dates in YYYYMMDD format.

Received:
2,0,216,202,164
2,0,144,22,177
2,0,16,176,172
2,0,16,176,172
2,0,16,176,172
2,0,16,176,172
1,114,176,160,124
2,0,248,32,63
2,0,144,226,164
2,0,144,226,164
2,0,144,226,164
2,0,144,202,124
2,0,144,202,124
2,0,144,176,172
2,0,144,176,172
2,0,32,22,63
2,0,38,248,172
2,0,38,248,172
2,0,38,98,164
2,0,233,1,15
2,0,45,107,172
1,114,176,226,26
1,114,176,38,177
1,114,176,97,164
2,0,0,17,124
2,0,128,129,31
2,0,128,129,31
2,0,128,129,31
2,0,128,129,31
2,0,128,129,31
2,0,128,129,31
2,0,216,17,63
2,0,160,17,31
2,0,160,128,34
2,0,160,129,26
2,0,38,128,31
2,0,38,144,26
1,114,97,16,124
1,114,97,16,124
2,0,38,234,26
2,0,38,201,172
2,0,45,38,124
2,0,45,216,164
2,0,45,107,177
2,0,248,234,124
2,0,248,202,34
2,0,248,18,172
2,0,97,128,172
2,0,248,18,7
2,0,248,233,15
2,0,201,2,15
2,0,176,16,7
2,0,106,0,31
2,0,216,22,34
2,0,216,160,63
2,0,38,107,7
2,0,233,0,63
2,0,38,107,164
2,0,233,0,26
2,0,38,107,34
2,0,233,0,164
2,0,233,17,15
2,0,45,202,177
2,0,45,106,7
2,0,45,97,177
2,0,47,16,31
2,0,248,216,177
2,0,201,0,172
2,0,176,201,63
2,0,248,97,34
2,0,176,202,26
2,0,248,97,34
2,0,201,2,172
2,0,201,17,164
2,0,176,129,164
2,0,201,17,172
2,0,176,144,7
2,0,145,2,164
2,0,32,145,15
2,0,38,45,26
2,0,38,38,63
2,0,38,233,26
2,0,38,248,34
2,0,45,202,164
2,0,45,107,124
2,0,47,17,15
2,0,47,16,31
2,0,47,130,34
2,0,248,45,177
2,0,106,0,31
2,0,248,22,31
2,0,248,202,172
2,0,248,97,172
2,0,47,128,177
2,0,201,2,164
2,0,216,201,164
2,0,176,16,34
2,0,216,201,34

Here's the codepage to convert back to ebcdic that isn't quite working:

ascii = Array( _
&H0, &H1, &H2, &H3, &H37, &H2D, &H2E, &H2F, &H16, &H5, &H25, &HB, &HC, &HD, &HE, &HF, _
&H10, &H11, &H12, &H13, &H3C, &H3D, &H32, &H26, &H18, &H19, &H3F, &H27, &H1C, &H1D, &H1E, &H1F, _
&H40, &H4F, &H7F, &H7B, &H5B, &H6C, &H50, &H7D, &H4D, &H5D, &H5C, &H4E, &H6B, &H60, &H4B, &H61, _
&HF0, &HF1, &HF2, &HF3, &HF4, &HF5, &HF6, &HF7, &HF8, &HF9, &H7A, &H5E, &H4C, &H7E, &H6E, &H6F, _
&H7C, &HC1, &HC2, &HC3, &HC4, &HC5, &HC6, &HC7, &HC8, &HC9, &HD1, &HD2, &HD3, &HD4, &HD5, &HD6, _
&HD7, &HD8, &HD9, &HE2, &HE3, &HE4, &HE5, &HE6, &HE7, &HE8, &HE9, &H4A, &HE0, &H5A, &H5F, &H6D, _
&H79, &H81, &H82, &H83, &H84, &H85, &H86, &H87, &H88, &H89, &H91, &H92, &H93, &H94, &H95, &H96, _
&H97, &H98, &H99, &HA2, &HA3, &HA4, &HA5, &HA6, &HA7, &HA8, &HA9, &HC0, &H6A, &HD0, &HA1, &H7, _
&H20, &H21, &H22, &H23, &H24, &H15, &H6, &H17, &H28, &H29, &H2A, &H2B, &H2C, &H9, &HA, &H1B, _
&H30, &H31, &H1A, &H33, &H34, &H35, &H36, &H8, &H38, &H39, &H3A, &H3B, &H4, &H14, &H3E, &HE1, _
&H41, &H42, &H43, &H44, &H45, &H46, &H47, &H48, &H49, &H51, &H52, &H53, &H54, &H55, &H56, &H57, _
&H58, &H59, &H62, &H63, &H64, &H65, &H66, &H67, &H68, &H69, &H70, &H71, &H72, &H73, &H74, &H75, _
&H76, &H77, &H78, &H80, &H8A, &H8B, &H8C, &H8D, &H8E, &H8F, &H90, &H9A, &H9B, &H9C, &H9D, &H9E, _
&H9F, &HA0, &HAA, &HAB, &HAC, &HAD, &HAE, &HAF, &HB0, &HB1, &HB2, &HB3, &HB4, &HB5, &HB6, &HB7, _
&HB8, &HB9, &HBA, &HBB, &HBC, &HBD, &HBE, &HBF, &HCA, &HCB, &HCC, &HCD, &HCE, &HCF, &HDA, &HDB, _
&HDC, &HDD, &HDE, &HDF, &HEA, &HEB, &HEC, &HED, &HEE, &HEF, &HFA, &HFB, &HFC, &HFD, &HFE, &HFF)

Answer 1

Yes the EBCDIC -->> ASCII conversion will screw up packed decimal fields because some of the bytes in the Packed Decimal will get converted to ASCII.

As you found out doing a EBCDIC -->> ASCII -->> ASCII will not work because multiple EBCDIC chars can get mapped to a single ASCII char and multiple ASCII chars can get mapped to a single EBCDIC char. Typically this translation error occurs for no display characters that you will find in a packed decimal field.

The solutions are 1) Unpack on the AS400 and trnsmit 2) Do a Binary Transfer. You could use the RecordEditor (http://record-editor.sourceforge.net/) to view the file. You should then be able to cut and paste the data into a Text editor etc.

Note: For EBCDIC use CP037 (coded page 37 or whatever coded page you use). The RecordEditor will allow you define packed decimal Fields If you have a Cobol Copybook; You could try importing The Copybook as a Mainframe copybook (it may work ???)

Answer 2

Sorry, don't know a lot about VB, much more about iSeries FTP. The iSeries FTP is indeed doing automatic translation from EBCDIC to ASCII. FTP is using the system tables QEBCDIC and QASCII for this. You can find these tables with the command WRKTBL.

I suggest that you check the values in these tables and check that they match your files. If not, I am thinking that VB is doing some conversion too.

Important: If the issue is with the iSeries, then you can work with other translation tables, as well as another character set. You can change these values in the FTP command (if you send from iSeries to Windows, or the FTP server (if you are allowed to do that).

What you NEVER should do is change the system tables itself (only if you're not faint of heart ... ). Copy these tables, change whatever you want, and point to these new tables.

Also, studying the server side commands in your FTP session may be worth while. From you local FTP session you can ask the iSeries to do a lot of things for you. These commands are not standard FTP commands, but iSeries specific.

Hope this helps.

Answer 3

Try using the conversion routines and codepage query & conversion functions available to VB provided by the Programmer's Toolkit, part of Access for Windows.

These are ActiveX objects. Not too hard to use from VB.

(long) AS400System.HostCodePage  // tells you the host's code page  
(object) PackedConverter         // convert between numeric strings and  byte arrays  
(object) CodePageConverter       // convert text data between iSeries and PC code pages