CODEWITHSUNDEEP
t-sqlazureazure-sql-database
ADringer
ADringer

Reputation: 2834

Importing a BCP file in Azure database

I have an Azure function that retrieves a zip file that contains multiple BCP files which unzips them and adds them as blobs.

I now want to import the BCP files into my SQL database but not sure how to go about it. I know I can use following script and run an SqlCommand:

BULK INSERT RegPlusExtract.dbo.extract_class
FROM 'D:\local\data\extract_class.bsp'
WITH ( FIELDTERMINATOR = '@**@',ROWTERMINATOR = '*@@*')

But this obviously does not work as the SQL server doesn't have access to the local function's D: drive.

How should I go about loading the data? Is it possible to load the BCP file into memory and then pass the SQLCommand? Or can I pass the file direct to SQL server?

I've found out that for backup/restore I can do FROM URL = ''. If I could use this for bulk insert then I can just reference the blob url, but doesn't look like I can?

Upvotes: 0

Views: 1986

Answers (1)

TheGameiswar
TheGameiswar

Reputation: 28900

You will need to use BLOB storage..below are the steps and these are documented here Microsoft/sql-server-samples

--create an external data source

CREATE EXTERNAL DATA SOURCE MyAzureBlobStorage
WITH (  TYPE = BLOB_STORAGE, 
        LOCATION = 'https://sqlchoice.blob.core.windows.net/sqlchoice/samples/load-from-azure-blob-storage', 
--      CREDENTIAL= MyAzureBlobStorageCredential    --> CREDENTIAL is not required if a blob storage is public!
);

You also can upload files to a container and reference it like below.Here week3 is a container

CREATE EXTERNAL DATA SOURCE MyAzureInvoicesContainer
    WITH  (
        TYPE = BLOB_STORAGE,
        LOCATION = 'https://newinvoices.blob.core.windows.net/week3', 
        CREDENTIAL = UploadInvoices  
    );

Now you can use OpenRowset and BulkInsert like below

-- 2.1. INSERT CSV file into Product table
BULK INSERT Product
FROM 'product.csv'
WITH (  DATA_SOURCE = 'MyAzureBlobStorage',
        FORMAT='CSV', CODEPAGE = 65001, --UTF-8 encoding
        FIRSTROW=2,
        TABLOCK); 

-- 2.2. INSERT file exported using bcp.exe into Product table
BULK INSERT Product
FROM 'product.bcp'
WITH (  DATA_SOURCE = 'MyAzureBlobStorage',
        FORMATFILE='product.fmt',
        FORMATFILE_DATA_SOURCE = 'MyAzureBlobStorage',
        TABLOCK); 

-- 2.3. Read rows from product.dat file using format file and insert it into Product table
INSERT INTO Product WITH (TABLOCK) (Name, Color, Price, Size, Quantity, Data, Tags) 
SELECT Name, Color, Price, Size, Quantity, Data, Tags
FROM OPENROWSET(BULK 'product.bcp',
                DATA_SOURCE = 'MyAzureBlobStorage',
                FORMATFILE='product.fmt',
                FORMATFILE_DATA_SOURCE = 'MyAzureBlobStorage') as products; 

-- 2.4. Query remote file 
SELECT Color, count(*)
FROM OPENROWSET(BULK 'product.bcp',
                DATA_SOURCE = 'MyAzureBlobStorage',
                FORMATFILE='data/product.fmt',
                FORMATFILE_DATA_SOURCE = 'MyAzureBlobStorage') as data
GROUP BY Color;

Upvotes: 4

Related Questions