Turning a Folder Into a Single File Without Zipping it?

Question

So my goal here is to created a self contained file that would include all the assets to load into a 3D scene. For instance there would be one text file detailing all the positions, rotations, scales, etc. of models and then folders with models and textures inside there. This wouldn't be hard to do in a folder, but for simplicity I want to have it all into one file, avoiding compressing it and uncompressing it. The reason why I don't want to compress and decompress it is to speed up loading times, because this is for a game.

Is there any possible way to do this? Or am I going to have to bite the bullet and make it into a zip.

Thanks.

Answer 1

The timing of the original question may be fortuitous. It turns out that in the future the Find/Load resource approach I suggested won't be an option for us either (the code that I used it in before is going cross-platform), so I came up with a solution that may also work for you. Full code is a bit much to post as an SO answer (but see my follow-on comment), although you can probably cook up a quick version of your own in no time.

Basically, I wrote a utility that just takes a basename for the output files as argv[1] ("output-base-name"), and argv[2...(argc-1)] are input files. I open output-base-name.cpp and output-base-name.h for writing (will overwrite if they exist).

For each input file, the generator utility does this:

1: generates a variable-name-friendly identifier based on the filename ("generated_name")
2: reads the input file
3: dumps the following into output-base-name.cpp:

static uint8_t generated_name[] = {
    // Initialize with contents of input file as a bunch of 0xNN, 0xNN... values.
};

and into output-base-name.h, a corresponding

static const unsigned IDR_generated_name = N; // start at 0, see below re:range check

4: After the last input file, finishes up output-base-name.cpp with something like

static struct {
    unsigned       ID;
    unsigned       size;  // I assume you aren't trying to embed files >4GB!
    const uint8_t* data;
} 
fileData[] = {
   // one array element for each input file
   { IDR_generated_name,  sizeof(generated_name), generated_name },
   ... 
};

const uint8_t* getResourceData(unsigned ID){
    // a real version would do a safer lookup/range check of some sort based on ID
    return fileData[ID].data;
}

unsigned getResourceSize(unsigned ID){
    // a real version would do a safer lookup/range check of some sort based on ID
    return fileData[ID].size;
}

5: Finishes output-base-name.h with prototypes for whatever your "get" functions look like, e.g.

extern const uint8_t* getResourceData(unsigned ID);
extern unsigned getResourceSize(unsigned ID);

Just use output-base-name.h wherever you need to access the data and add output-base-name.cpp to your makefile or project (make it dependent on the binary input files, using the generator utility in the build rule and make will automatically regenerate it for you when necessary). If you want them in a dynamic lib of some sort, just add appropriate export attributes to your "get" functions.

Have the generator wrap your output in a namespace, throw together a class-based interface to it if you want to (or generate one instead of the example "get" functions), and you should be good.

This approach has several advantages over using an uncompressed zip file (or even the embedded resource approach I suggested for Windows targets):

1. It's a lot simpler to deal with in your application code - no reading of the zip directory, extracting file data, etc. is required.
2. No added reliance on another third-party lib for lookup/extraction of file data (assuming you don't roll your own zip library, which would be even more work)
3. All the resolution and lookup work is done for you - the addresses for each byte array will already be resolved by the linker if sticking the data into a static library or your executable, or by the OS loader if a shared library or DLL
4. You know the data will exist or your binary cannot run (not the case with zip lib approach, or even Find/LoadResource on Windows)
5. Data is a bit more inaccessible to (therefore safer from) Joe User, compared to a zip file
6. Should be faster than either the zip or Windows resource approach due to the pre-resolution of data addresses during the load process for the application
7. It can be also used for any form of relatively static data you want to ship with (or make available to) your app - just package files into a shared/dynamic library with the generator and use the generated exported accessor functions. You can even create updated or completely new data packages later (after compilation/ship time) and load at runtime using config options.
8. It will work in older C++ compilers if necessary (or, sans generating any namespace or class wrappers, even plain-old-C)

This is only an example of the approach that I used, but it covers all the basics so if a solution sooner rather than later (again, see comment) is better for you, this might be enough (you'd want to range-check the values in the "get" functions - at least in debug builds, maybe use containers of some sort instead of using a c-style struct array, etc).

Answer 2

If this is for a Windows application, you can use user-defined resource statements in the .RC file for your EXE or included DLL(s) that will insert the data from the external files as binary resources. This is especially handy if you want to only distribute a single EXE file that relies on data contained in other files, or don't want to have the files you might otherwise archive and distribute with your app accessible/modifiable (without some work) by an end user.

It's extremely easy - the syntax is just

nameID typeID filename

for the resource file. You can access the data using ::FindResource() and ::LoadResource(). MS docs for user-defined resources are at http://msdn.microsoft.com/en-us/library/windows/desktop/aa381054(v=vs.85).aspx.

Answer 3

You can use ZIP without compression or TAR. But here's a better idea: compress it. CPU time is cheap. Disk transfers take forever. Most of the time, loading compressed data and decompressing it is faster than loading uncompressed data.

Answer 4

You can create a ZIP file with no compression. ("Stored" mode.) There are also similar packaging archivers that are useful for creating a single archive from multiple files. TAR files on *nix systems are popular for this, and there are many others, as well.

Also, balance the amount of time that will be spend decompressing against the amount of time that will be spent loading an (uncompressed) file from disk. A lot of times, the disk read times for an uncompressed file are longer than the time that would be spent loading a compressed file and uncompressing it in memory.