writing big tiff pictures

Question

I'm writing big tiff files with bigtif library. I use simple calling of "TIFFWriteScanline" to write tiff and it writes data correctly until the size of file becomes more than 3, 4 GB, and there the writing process becomes very slow but writes tiff correctly). I want a way to resolve it because it cause a bottleneck for my application speed. can I resolve this problem in libtiff or I must switch to another library like gdal,..?

Can gdal write huge tiff files(more than 4GB) with a good speed?

thanks in advance.

Answer 1

Here is a good example writing a bigTIFF using LibTIFF Library:

TIFF *tif = TIFFOpen(fileName, "w8");

Source and context: https://gist.github.com/jcupitt/ba60e4e1100607d7a5cc9c19e2ec11e8

Answer 2

GDAL can write very large BigTIFFs. You need to build GDAL with BigTIFF support.

You need to be using libTIFF version 4 or newer if you are building libTIFF externally from GDAL.

You will need to pass a set of options to GDAL to write BigTIFFs, see:

http://www.gdal.org/frmt_gtiff.html

• BIGTIFF=YES/NO/IF_NEEDED/IF_SAFER: Control whether the created file is a BigTIFF or a classic TIFF.

You can speed up the processing using the following option:

• NUM_THREADS=number_of_threads/ALL_CPUS: (From GDAL 2.1) Enable multi-threaded compression by specifying the number of worker threads. Worth for slow compressions such as DEFLATE or LZMA. Will be ignored for JPEG. Default is compression in the main thread.

If you are writing BigTIFFs yourself then when you open the TIFF file for writing you need to pass '8' (as ASCII) to the TIFFOpen call as part of the mode character string.

The Documentation in tif_open.c indicates the valid options:

/*
 * Process library-specific flags in the open mode string.
 * The following flags may be used to control intrinsic library
 * behaviour that may or may not be desirable (usually for
 * compatibility with some application that claims to support
 * TIFF but only supports some brain dead idea of what the
 * vendor thinks TIFF is):
 *
 * 'l' use little-endian byte order for creating a file
 * 'b' use big-endian byte order for creating a file
 * 'L' read/write information using LSB2MSB bit order
 * 'B' read/write information using MSB2LSB bit order
 * 'H' read/write information using host bit order
 * 'M' enable use of memory-mapped files when supported
 * 'm' disable use of memory-mapped files
 * 'C' enable strip chopping support when reading
 * 'c' disable strip chopping support
 * 'h' read TIFF header only, do not load the first IFD
 * '4' ClassicTIFF for creating a file (default)
 * '8' BigTIFF for creating a file
 *
 * The use of the 'l' and 'b' flags is strongly discouraged.
 * These flags are provided solely because numerous vendors,
 * typically on the PC, do not correctly support TIFF; they
 * only support the Intel little-endian byte order.  This
 * support is not configured by default because it supports
 * the violation of the TIFF spec that says that readers *MUST*
 * support both byte orders.  It is strongly recommended that
 * you not use this feature except to deal with busted apps
 * that write invalid TIFF.  And even in those cases you should
 * bang on the vendors to fix their software.
 *
 * The 'L', 'B', and 'H' flags are intended for applications
 * that can optimize operations on data by using a particular
 * bit order.  By default the library returns data in MSB2LSB
 * bit order for compatibility with older versions of this
 * library.  Returning data in the bit order of the native CPU
 * makes the most sense but also requires applications to check
 * the value of the FillOrder tag; something they probably do
 * not do right now.
 *
 * The 'M' and 'm' flags are provided because some virtual memory
 * systems exhibit poor behaviour when large images are mapped.
 * These options permit clients to control the use of memory-mapped
 * files on a per-file basis.
 *
 * The 'C' and 'c' flags are provided because the library support
 * for chopping up large strips into multiple smaller strips is not
 * application-transparent and as such can cause problems.  The 'c'
 * option permits applications that only want to look at the tags,
 * for example, to get the unadulterated TIFF tag information.
 */

Make sure that you are writing the tiff out as either strips or tiles. My preference is for tiles. This is also true when using GDAL. For BigTIFF images you have to process the images as tiles or strips.

Edit 18:19 24/7/2017

The reason I answered this question is because I am having to create huge pyramid GeoTIFFs for a client (11 plus levels potentially covering the whole world).

The largest image I have created so far is just slightly short of 4GB. I have just quadrupled the size of the highest resolution image (to 1638400x1638400 pixels RGBA, LZW compression). So far an hour has passed by and I have only generated 5% of this layer (on a 'MSI GP727RD Leopard', release build).

The time issue is complicated in that I am drawing vector data into each tile being generated.

I am partly using GDAL for creating the GeoTIFF tags from the Coordinate System WKT (had to hack it out of the driver).

I am writing out the TIFF my-self using libTIFF. Once this is all working I will be pushing the processing into as many threads a possible. However I will be creating separate GeoTIFFs for each thread as there is no simple way of combining the lot into one large TIFF and I'm not sure that that would be sensible in any case.

The memory use in a 32bit process is quite low. My process is using ~60Mb of memory.