Delimiters with bcp import

Question

I am trying to import bulk data from text files. The statement that i use is below:

DECLARE @s as varchar(300)

SET @s='bcp [hc]..[QRYM_COMPANIES] in ' + char(34) + 'C:\SQL Files\health canada allfiles\comp.txt' + char(34) + ' -C ACP -c -t \", -r\n -e' + char(34) + 'C:\SQL Files\health canada allfiles\err\comp.err.txt' + char(34) + ' -b250 -m50 -SServer -Usa -P' + char(34) + 'mypassword' + char(34) + ' -h' + char(34) + 'TABLOCK' + char(34) 


PRINT @s

EXEC master..xp_cmdshell @s

Now, normally I would have put -t, but my data contains commas in some of the data fields which results in the data being shifted. When I try to use -t \" I get the error "String data, right truncation", so that is why I am having to use the combination of -t ,\"

The problem here is that I get quotations in my transferred data. Below is a sample row of my data:

"20042","NSL95","4797","NU SKIN INTERNATIONAL, INC.","DIN OWNER","Y","N","N","N","","75 WEST CENTER STREET","PROVO","UTAH","UNITED STATES","84601",""

The table description is below. To avoid the "string truncation error" i have added a 1 to the left field size.

Can someone help me fix this problem?

Thanks

CREATE TABLE [health_canada].[dbo].[QRYM_COMPANIES] (

DRUG_CODE   VARCHAR(18) NOT NULL, --NUMERIC(8) NOT NULL,

MFR_CODE        VARCHAR(15),

COMPANY_CODE        VARCHAR(16), --NUMERIC(6),

COMPANY_NAME        VARCHAR(180),

COMPANY_TYPE        VARCHAR(140),

ADDRESS_MAILING_FLAG        VARCHAR(11),

ADDRESS_BILLING_FLAG        VARCHAR(11),

ADDRESS_NOTIFICATION_FLAG       VARCHAR(11),

ADDRESS_OTHER       VARCHAR(11),

SUITE_NUMERIC       VARCHAR(120),

STREET_NAME         VARCHAR(180),

CITY_NAME       VARCHAR(160),

PROVINCE        VARCHAR(140),

COUNTRY         VARCHAR(140),

POSTAL_CODE         VARCHAR(120),

POST_OFFICE_BOX         VARCHAR(115)

) ON [PRIMARY]

GO

Answer 1

Consider to use a format file with your bcp call. It is easy to configure, and you can fine tune it for your file. Just add -f and the path to use it: http://msdn.microsoft.com/en-us/library/ms162802.aspx

Steps

• Add the format file to the disk (make sure that the SQL server can reach the path).
• Update paths in query.
• Run query.

I used this file:

This code imported your example data without quotations with the format file:

DECLARE @importTable VARCHAR(128) = 'hc.dbo.QRYM_COMPANIES';
DECLARE @importPath VARCHAR(8000) = '\\YOURPATH\comp.txt';
DECLARE @formatFilePath VARCHAR(8000) = '\\YOURPATH\formatFile.txt';
DECLARE @separator VARCHAR(10) = '';

DECLARE @export VARCHAR(8000) = 'bcp "' + @importTable + '" in "' + @importPath + '" -T -f "' + @formatFilePath + '" -C RAW -S ' + @@SERVERNAME;

PRINT(@export)
EXEC xp_cmdshell @export 

Output:

SELECT * FROM hc.dbo.QRYM_COMPANIES:

Add this format file to your disk:

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <RECORD> 
  <FIELD ID="0" xsi:type="CharTerm" TERMINATOR='"' MAX_LENGTH="1" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="1" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="18" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="2" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="15" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="3" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="16" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="4" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="180" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="5" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="140" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="6" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="7" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="8" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="9" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="11" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="10" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="120" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="11" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="180" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="12" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="160" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="13" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="140" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="14" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="140" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="15" xsi:type="CharTerm" TERMINATOR='","' MAX_LENGTH="120" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="16" xsi:type="CharTerm" TERMINATOR='"\r\n' MAX_LENGTH="115" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
 </RECORD>
 <ROW>
  <COLUMN SOURCE="1" NAME="DRUG_CODE" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="2" NAME="MFR_CODE" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="3" NAME="COMPANY_CODE" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="4" NAME="COMPANY_NAME" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="5" NAME="COMPANY_TYPE" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="6" NAME="ADDRESS_MAILING_FLAG" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="7" NAME="ADDRESS_BILLING_FLAG" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="8" NAME="ADDRESS_NOTIFICATION_FLAG" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="9" NAME="ADDRESS_OTHER" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="10" NAME="SUITE_NUMERIC" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="11" NAME="STREET_NAME" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="12" NAME="CITY_NAME" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="13" NAME="PROVINCE" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="14" NAME="COUNTRY" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="15" NAME="POSTAL_CODE" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="16" NAME="POST_OFFICE_BOX" xsi:type="SQLVARYCHAR"/>
 </ROW>
</BCPFORMAT>

You can auto generate new format files with bcp if you need it for other files/tables (no need to do this now if the format file works)

Important: After format file export:

• Replace: TERMINATOR="\"," to TERMINATOR='","'

• Replace: TERMINATOR="\r\n" to TERMINATOR='"\r\n'*

• Add to <record> (will remove first quotation): <FIELD ID="0" xsi:type="CharTerm" TERMINATOR='"' MAX_LENGTH="1" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>

Auto generate query:

DECLARE @importTable VARCHAR(128) = 'hc.dbo.QRYM_COMPANIES';
DECLARE @formatFilePath VARCHAR(8000) = '\\YourFilePath\formatFile.txt';
DECLARE @bcpCall VARCHAR(8000);

SET @bcpCall = 'bcp "' + @importTable + '" format nul -c -t", -x -f "' + @formatFilePath + '" -T -S ' + @@SERVERNAME;

PRINT @bcpCall 

EXEC xp_cmdshell @bcpCall