Read a CSV file that have indefinite number of columns every time and create a table based on column names in csv file

Question

I have a requirement to load the csv into DB using oracle apex or pl/sql code, but the problem is they are asking to load the csv file which will not come with same number of columns and column names .

I should create table & upload data dynamically based on the file name and data that i'm uploading. For every file i need to create a new table dynamically and insert data that are present in csv file.

For Example:

File1: col1 col2 col3 col4 (NOTE: If i upload File 1, Table should be created dynamically based on the file name and table should contain same column name and data same as column headers of csv file . )

file 2: col1 col2 col3 col4 col 5

file 3: col4 col2 col1 col3

Depending on the columns and file name i need to create table for every file upload. Can we load like this or not? If yes, Please help me on this.

Regards, Sachin.

Answer 1

((Where's the PL/SQL code in this solution!!??! Bear with me... the answer is buried in here somewhere... I introduced some considerations and assumptions you will need to think about before going into the task. In the end, you'll find that Oracle APEX actually has a built-in solution that satisfies exactly what you've specified... with some caveats.))

If you are working within the Oracle APEX platform, you will have some advantages. APEX Version 4.2 and higher has a new page element called "Data Loading". The disadvantage however is that the definition of the upload target is fixed and not dynamic. You will need to know how your table is structured prior to loading the data.

One approach to overcome this is to build a generic, two-column table as your target, which will serve for all uploads. Column 1 will be your file-name and column two will be a single clob data type, which will contain the entire data file's contents including the header row. The "Data Loading" element will give the user the opportunity to verify and select this mapping convention in a couple of clicks.

At this point, it's mostly PL/SQL backend work doing the heavy lifting to parse and transform the data uploaded. As far as the dynamic table creation, I have noticed that the Oracle package, DBMS_SQL allows the execution of DDL SQL commands, which could be the route to making custom tables.

Alex Poole's comment is important as well, you will need to make some blanket assumption about the data type or have a provision to give more clues about what kind of data is contained. Assuming you can rely on a sample of existing data values is not good... what if all the values in your upload are null? I recommend perhaps a second column in the data input with a clue about the type of data for each column... just like the intended header names, maybe: AAAAA = for a five character column, # = for a numeric, MM/DD/YYYY = for a date with a specific masking.

The easier route: You will need to allow your end-user access to a developer-role account on a workspace of your APEX server. It is not as scary as you think. With careful instruction and some simple precautions, I have been able to make this work with even the most non-technical of users. The reason for this is that there is a more powerful upload tool found under the following menu item:

SQL Workshop --> Utilities --> Data Workshop

There is a choice under "Data Load" --> "Spreadsheet Data"

The data load tool will automatically do the following:

1. Accept a CSV formatted file through a browse function on your client machine
2. Upload the file and parse the first record for the column layout (names)
3. Allow the user to create a new table from the uploaded file, or to map to an existing one.
4. For new tables, each column data type can be declared and also a specific numeric/date mask if additional conversion from the uploaded data is necessary.
5. Delimiter type, optional enclosures (like double quotes), decimal conventions and currency types can also be declared prior to parsing the uploaded file.

Once the user has identified all these mappings and settings, the table is created with the uploaded data. Any errors in record upload are reported immediately afterwards with detailed feedback on the failed records.

A security consideration to note: You probably do not want to give end users access to your APEX server's backend... but you CAN create a new workspace... just for your end users... create a new database schema for receiving their uploads, maybe with some careful resource controls. Developer is the minimum role needed... but even if the end users see the other stuff there won't be access to anything important from an isolated workspace.

I have implemented the isolated workspace approach on a 4.0/4.1 release APEX platform a few years back, and it worked nicely. Our end user had control over the staging and quality checking of her data inputs (from excel spreadsheet/csv exports collected from a combination of sources). I suppose it may have been even better to cut her out of the picture entirely and focused on automating the export-review-upload process between our database and her other sources. In this case, the volume of data involved was not great enough (100's to 1000's of records) and the need for manual review and edit of the exported data was very important prior to pushing it into the database... so the human element was still important in this case - it is something you'll want to think about now.