Azure Form Recognizer Name/Value Pairs

Question

I am currently working with Azure Form Recognizer and had a question. I am using

https://<>.cognitiveservices.azure.com/formrecognizer/v2.0-preview/layout/analyzeResults/2e0a2322-65bb-4fd2-a3bf-98f70b36641e

The JSON returned seems to be using basic OCR. I was wondering if its possible (easily)

to take this

{
                    "boundingBox": [
                        4.4033,
                        1.5114,
                        6.5483,
                        1.5114,
                        6.5483,
                        1.6407,
                        4.4033,
                        1.6407
                    ],
                    "text": "Invoice For: First Up Consultants",
                    "words": [
                        {
                            "boundingBox": [
                                4.4033,
                                1.5143,
                                4.8234,
                                1.5143,
                                4.8234,
                                1.6155,
                                4.4033,
                                1.6155
                            ],
                            "text": "Invoice",
                            "confidence": 1
                        },
                        {
                            "boundingBox": [
                                4.8793,
                                1.5143,
                                5.1013,
                                1.5143,
                                5.1013,
                                1.6154,
                                4.8793,
                                1.6154
                            ],
                            "text": "For:",
                            "confidence": 1
                        },
                        {
                            "boundingBox": [
                                5.2048,
                                1.5130,
                                5.4927,
                                1.5130,
                                5.4927,
                                1.6151,
                                5.2048,
                                1.6151
                            ],
                            "text": "First",
                            "confidence": 1
                        },
                        {
                            "boundingBox": [
                                5.5427,
                                1.5130,
                                5.7120,
                                1.5130,
                                5.7120,
                                1.6407,
                                5.5427,
                                1.6407
                            ],
                            "text": "Up",
                            "confidence": 1
                        },
                        {
                            "boundingBox": [
                                5.7621,
                                1.5114,
                                6.5483,
                                1.5114,
                                6.5483,
                                1.6151,
                                5.7621,
                                1.6151
                            ],
                            "text": "Consultants",
                            "confidence": 1
                        }
                    ]
                }

but return it as

"boundingBox": [
                        4.4033,
                        1.5114,
                        6.5483,
                        1.5114,
                        6.5483,
                        1.6407,
                        4.4033,
                        1.6407
                    ],
                    "text": "Invoice For:",
                    "value": "First Up Consultants"

}

If this is not something that I can do in azure form recognizer, then no worries. I just wanted to see.

Thank you in advance! Michael

Answer 1

It sounds like you're looking to extract semantic meaning from your document. In that case, you might want to look at using a custom Form Recognizer model.

You can start by training a custom model to extract key value pairs: https://learn.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/curl-train-extract

Sample key value pair:

{
    "key": {
        "text": "Address:",
        "boundingBox": [ 0.7972, 1.5125, 1.3958, 1.5125, 1.3958, 1.6431, 0.7972, 1.6431 ]
    },
    "value": {
        "text": "1 Redmond way Suite 6000 Redmond, WA 99243",
        "boundingBox": [ 0.7972, 1.6764, 2.15, 1.6764, 2.15, 2.2181, 0.7972, 2.2181 ]
    },
    "confidence": 0.86
}

Or you can train a custom model using labels that you provide: https://learn.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/label-tool

Sample field output:

{
    "total": {
        "type": "string",
        "valueString": "$22,123.24",
        "text": "$22,123.24",
        "boundingBox": [ 5.29, 3.41, 5.975, 3.41, 5.975, 3.54, 5.29, 3.54 ],
        "page": 1,
        "confidence": 1
    }
}