CODEWITHSUNDEEP
cwindows
kampi
kampi

Reputation: 2484

How to handle a huge string correctly?

This may be a newbie question, but i want to avoid buffer overflow. I read very much data from the registry which will be uploaded to an SQL database. I read the data in a loop, and the data was inserted after each loop. My problem is, that this way, if i read 20 keys, and the values under is ( the number of keys is different on every computer ), then i have to connect to the SQL database 20 times.

However i found out, that there is a way, to create a stored procedure, and pass the whole data it, and so, the SQL server will deal with data, and i have to connect only once to the SQL server.

Unfortunately i don't know how to handle such a big string to avoid any unexpected errors, like buffer owerflow. So my question is how should i declare this string?

Should i just make a string like char string[ 15000 ]; and concatenate the values? Or is there a simplier way for doing this?

Thanks!

Upvotes: 5

Views: 1714

Answers (3)

Harry Johnston
Harry Johnston

Reputation: 36328

To do this properly in C you need to allocate the memory dynamically, using malloc or one of the operating system equivalents. The idea here is to figure out how much memory you actually need and then allocate the correct amount. The registry functions provide various ways you can determine how much memory you need for each read.

It gets a bit trickier if you're reading multiple values and concatenating them. One approach would be to read each value into a separately allocated memory block, then concatenate them to a new memory block once you've got them all.

However, it may not be necessary to go to this much trouble. If you can say "if the data is more than X bytes the program will fail" then you can just create a static buffer as you suggest. Just make sure that you provide the registry and/or string concatenation functions with the correct size for the remaining part of the buffer, and check for errors, so that if it does fail it fails properly rather than crashing.

One more note: char buf[15000]; is OK provided the declaration is in program scope, but if it appears in a function you should add the static specifier. Implicitly allocated memory in a function is by default taken from the stack, so a large allocation is likely to fail and crash your program. (Fifteen thousand bytes should be OK but it's not a good habit to get into.)

Also, it is preferable to define a macro for the size of your buffer, and use it consistently:

#define BUFFER_SIZE 15000
char buf[BUFFER_SIZE];

so that you can easily increase the size of the buffer later on by modifying a single line.

Upvotes: 0

Alex Kotliarov
Alex Kotliarov

Reputation: 183

You may read (key, value) pairs from a registry and store them into a preallocated buffer while there is sufficient space there. Maintain "write" position within the buffer. You could use it to check whether there is enough space for new key,value pair in the buffer. When there is no space left for new (key,value) pair - execute stored procedure and reset "write" position within the buffer. At the end of the "read key, value pairs" loop - check buffer's 'write" position and execute stored procedure if it is greater than 0. This way you will minimize number of times you execute stored procedure on a server.

const int MAX_BUFFER_SIZE = 15000;
char buffer[MAX_BUFFER_SIZE];
char buffer_pos = 0; // "write" position within the buffer.

...

// Retrieve key, value pairs and push them into the buffer.
while(get_next_key_value(key, value)) {
  post(key, value);
}

// Execute stored procedure if buffer is not empty.
if(buffer_pos > 0) {
  exec_stored_procedure(buffer);
}
...

bool post(const char* key, const char* value)
{
  int len = strlen(key) + strlen(value) + <length of separators>;

  // Execute stored procedure if there is no space for new key/value pair.
  if(len + buffer_pos >= MAX_BUFFER_SIZE) {
    exec_stored_procedure(buffer);
    buffer_pos = 0;  // Reset "write" position.
  }

  // Copy key, value pair to the buffer if there is sufficient space.
  if(len + buffer_pos < MAX_BUFFER_SIZE) {  
    <copy key, value to the buffer, starting from "write" position>
    buffer_pos += len; // Adjust "write" position.
    return true;
  }
  else {
    return false;
  }
}

bool exec_stored_procedure(const char* buf)
{
  <connect to SQL database and execute stored procedure.>
}

Upvotes: 0

Peon the Great
Peon the Great

Reputation: 1319

STL strings should do a much better job than the approach you have described.

You'll also need to build some thresholds. For example, if your string grew more than a mega bytes, it will be worth considering making different SQL connections since your transaction will be too long.

Upvotes: 2

Related Questions