How to determine extension from fileBytes

Question

My application allows users to download files. While creating headers I am using Tika to set extension as shown below. This works fine for pdf files. Fails for DOC and EXCEL files.

private HttpHeaders getHeaderData(byte[] fileBytes) throws IOException, MimeTypeException {
        final HttpHeaders headers = new HttpHeaders();

        TikaInputStream tikaStream = TikaInputStream.get(fileBytes);
        Tika tika = new Tika();
        String mimeType = tika.detect(tikaStream);
        headers.setContentType(MediaType.valueOf(mimeType));

        MimeTypes defaultMimeTypes = MimeTypes.getDefaultMimeTypes();
        String extension = defaultMimeTypes.forName(mimeType).getExtension();
        headers.add("file-ext", extension);

        return headers;
    }

I see that the mimeType is resolved to "application/pdf" for pdf files but resolves to "application/x-tika-ooxml" for excel and word files which is the problem. How can I get word(.docx) and excel (xlx, xlsx) formats if I have a file in bytes.

Why does this work for pdf?

Answer 1

Tika official usages: https://tika.apache.org/1.26/detection.html

Tika supported formats: https://tika.apache.org/1.26/formats.html

You could get the answers by simply reading the above 2 pages. Here are some key quotes:

Microsoft Office and some related applications produce documents in the generic OLE 2 Compound Document and Office Open XML (OOXML) formats. The older OLE 2 format was introduced in Microsoft Office version 97 and was the default format until Office version 2007 and the new XML-based OOXML format. The OfficeParser and OOXMLParser classes use Apache POI libraries to support text and metadata extraction from both OLE2 and OOXML documents.

That means you need to include also Apache POI jars or Maven dependencies for MS office files.

Tika provides a wrapping detector in the form of org.apache.tika.detect.DefaultDetector. This uses the service loader to discover all available detectors, including any available container aware ones, and tries them in turn. For container aware detection, include the Tika Parsers jar and its dependencies in your project, then use DefaultDetector along with a TikaInputStream.

That means you need to include the Tika Parsers jar or Maven dependencies.

Then use new DefaultDetector().detect(TikaInputStream.get(file), new Metadata());

Answer 2

Summary

The short answer is: You have to use Tika's detector with its MediaType class - not MimeTypes.

The slightly longer answer is: Even that will not get you all the way, because of how older MS-Office files are structured. For those you have to also parse the files, and inspect their metadata.

The term "media type" has replaced the term "MIME type" - see here:

[RFC2046] specifies that Media Types (formerly known as MIME types) and Media Subtypes will be assigned and listed by the IANA.

Office 97-2003

When Tika inspects Excel and Word 97-2003 files using its detector, it will return a media type of application/x-tika-msoffice. I assume (perhaps incorrectly) that this is its way of handling a file-type group, where the detector cannot determine the specific flavor of MS-Office 97-2003 file, based on its analysis. This is similar to the application/x-tika-ooxml in your question.

Expected Results

Based on the IANA list here, and a Mozilla list here, these are the media types we expect to get for the following file types:

• .pdf :: application/pdf
• .xls :: application/vnd.ms-excel
• .doc :: application/msword
• .xlsx :: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
• .docx :: application/vnd.openxmlformats-officedocument.wordprocessingml.document

The Program

The program shown below uses the following Maven dependencies:

    <dependencies>
        <dependency>
            <groupId>org.apache.tika</groupId>
            <artifactId>tika-core</artifactId>
            <version>1.23</version>
        </dependency>
        <dependency>
            <groupId>org.apache.tika</groupId>
            <artifactId>tika-parsers</artifactId>
            <version>1.23</version>
        </dependency>
        <dependency>
            <groupId>javax.ws.rs</groupId>
            <artifactId>javax.ws.rs-api</artifactId>
            <version>2.1.1</version>
        </dependency>
    </dependencies>

The program (just for this demo - not production ready) is shown below. Specifically, look at the tikaDetect() and tikaParse() methods.

import java.io.IOException;
import java.io.File;
import java.io.FileInputStream;
import java.io.BufferedInputStream;
import java.util.Set;
import java.util.HashSet;
import org.apache.tika.mime.MediaType;
import org.apache.tika.mime.MimeTypeException;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.detect.Detector;
import org.apache.tika.detect.DefaultDetector;
import org.apache.tika.exception.TikaException;
import org.apache.tika.parser.Parser;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.sax.BodyContentHandler;
import org.apache.tika.parser.ParseContext;
import org.xml.sax.SAXException;
import org.xml.sax.ContentHandler;

public class Main {

    private final Set<File> msOfficeFiles = new HashSet();

    public static void main(String[] args) throws IOException, MimeTypeException,
            SAXException, TikaException {
        Main main = new Main();
        main.doFileDetection();
    }

    private void doFileDetection() throws IOException, MimeTypeException, SAXException, TikaException {
        File file1 = new File("C:/tmp/foo.pdf");
        File file2 = new File("C:/tmp/baz.xlsx");
        File file3 = new File("C:/tmp/bat.docx");
        // Excel 97-2003 format:
        File file4 = new File("C:/tmp/bar.xls");
        // Word 97-2003 format:
        File file5 = new File("C:/tmp/daz.doc");
        Set<File> files = new HashSet();
        files.add(file1);
        files.add(file2);
        files.add(file3);
        files.add(file4);
        files.add(file5);

        for (File file : files) {
            try (BufferedInputStream bis = new BufferedInputStream(
                    new FileInputStream(file))) {
                tikaDetect(file, bis);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        for (File file : msOfficeFiles) {
            tikaParse(file);
        }
    }

    private void tikaDetect(File file, BufferedInputStream bis)
            throws IOException, SAXException, TikaException {
        Detector detector = new DefaultDetector();
        Metadata metadata = new Metadata();
        MediaType mediaType = detector.detect(bis, metadata);
        if (mediaType.toString().equals("application/x-tika-msoffice")) {
            msOfficeFiles.add(file);
        } else {
            System.out.println("Media Type for " + file.getName()
                    + " is: " + mediaType.toString());
        }
    }

    private void tikaParse(File file) throws SAXException, TikaException {
        Parser parser = new AutoDetectParser();
        ContentHandler handler = new BodyContentHandler();
        Metadata metadata = new Metadata();
        ParseContext context = new ParseContext();
        try (BufferedInputStream bis = new BufferedInputStream(
                new FileInputStream(file))) {
            parser.parse(bis, handler, metadata, context);
            tikaDetect(file, bis);
        } catch (IOException e) {
            e.printStackTrace();
        }
        System.out.println("Media Type for " + file.getName()
                + " is: " + metadata.get("Content-Type"));
    }
}

Actual Results

The program generates some warnings and information messages. If we ignore these for this exercise, we get the following print statements:

Media Type for bat.docx is: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Media Type for baz.xlsx is: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Media Type for foo.pdf is: application/pdf
Media Type for bar.xls is: application/vnd.ms-excel
Media Type for daz.doc is: application/msword

These match the expected official media (MIME) types.