CODEWITHSUNDEEP
nlpspacynamed-entity-recognition
TanrCans
TanrCans

Reputation: 213

How can i work with Example for nlp.update problem with spacy3.0

i am trying to train my data with spacy v3.0 and appareantly the nlp.update do not accept any tuples. Here is the piece of code:

import spacy
import random
import json
nlp = spacy.blank("en")
ner = nlp.create_pipe("ner")
nlp.add_pipe('ner')
ner.add_label("label")
# Start the training
nlp.begin_training()
# Loop for 40 iterations
for itn in range(40):
    # Shuffle the training data
    random.shuffle(TRAINING_DATA)
    losses = {}
# Batch the examples and iterate over them
    for batch in spacy.util.minibatch(TRAINING_DATA, size=2):
        texts = [text for text, entities in batch]
        annotations = [entities for text, entities in batch]
# Update the model
        nlp.update(texts, annotations, losses=losses, drop=0.3)
    print(losses)

and i am receiving error

ValueError                                Traceback (most recent call last)
<ipython-input-79-27d69961629b> in <module>
     18         annotations = [entities for text, entities in batch]
     19 # Update the model
---> 20         nlp.update(texts, annotations, losses=losses, drop=0.3)
     21     print(losses)

~\Anaconda3\lib\site-packages\spacy\language.py in update(self, examples, _, drop, sgd, losses, component_cfg, exclude)
   1086         """
   1087         if _ is not None:
-> 1088             raise ValueError(Errors.E989)
   1089         if losses is None:
   1090             losses = {}

ValueError: [E989] `nlp.update()` was called with two positional arguments. This may be due to a backwards-incompatible change to the format of the training data in spaCy 3.0 onwards. The 'update' function should now be called with a batch of Example objects, instead of `(text, annotation)` tuples.

I set my train data format:

TRAINING_DATA = []
for entry in labeled_data:
    entities = []
    for e in entry['labels']:
        entities.append((e[0], e[1],e[2]))
    spacy_entry = (entry['text'], {"entities": entities})
    TRAINING_DATA.append(spacy_entry)

My train data looks like this:

[('Part List', {'entities': []}), ('pending', {'entities': []}), ('3D Printing', {'entities': [(0, 11, 'Process')]}), ('Recommended to use a FDM 3D printer with PLA material.', {'entities': [(25, 36, 'Process'), (41, 44, 'Material')]}), ('', {'entities': []}), ('No need supports or rafts.', {'entities': []}), ('Resolution: 0.20mm', {'entities': []}), ('Fill density 20%', {'entities': []}), ('As follows from the analysis, part of the project is devoted to 3D', {'entities': [(64, 66, 'Process')]}), ('printing, as all static components were created using 3D modelling and', {'entities': [(54, 66, 'Process')]}), ('subsequent printing.', {'entities': []}), ('', {'entities': []}), ('In our project, we created several versions of the', {'entities': []}), ('model during modelling, which we will describe and document in the', {'entities': []}), ('following subchapters. As a tool for 3D modelling, we used the Sketchup', {'entities': [(37, 49, 'Process')]}), ('Make tool, version from 2017. The main reason was the high degree of', {'entities': []}), ('intuitiveness and simplicity of the tool, as we had not encountered 3D', {'entities': [(68, 70, 'Process')]}), ('modelling before and needed a relatively flexible and efficient tool to', {'entities': []}), ('guarantee the desired result. with zero previous experience.', {'entities': []}), ('In this version, which is shown in the figures Figure 13 - Version no. 2 side view and Figure 24 - Version no. 2 - front view, for the first time, the specific dimensions of the infuser were clarified and', {'entities': []}), ('modelled. The details of the lower servo attachment, the cable hole in', {'entities': []}), ('the main mast, the winding cylinder mounting, the protrusion on the', {'entities': [(36, 44, 'Process')]}), ('winding cylinder for holding the tea bag, the preparation for fitting', {'entities': []}), ('the wooden and aluminium plate and the shape of the cylinder end that', {'entities': [(15, 25, 'Material')]}), ('exactly fit the servo were also reworked.', {'entities': []}), ('After the creation of this', {'entities': []}), ('version of the model, this model was subsequently officially consulted', {'entities': []}), ('and commented on for the first time.', {'entities': []}), ('In this version, which is shown in the figures Figure 13 - Version no. 2 side view and Figure 24 - Version no. 2 - front view, for the first time, the specific dimensions of the infuser were clarified and', {'entities': []}), ('modelled. The details of the lower servo attachment, the cable hole in', {'entities': []}), ('the main mast, the winding cylinder mounting, the protrusion on the', {'entities': [(36, 44, 'Process')]})]

I would appreciate your help as a new contributor. Thanks a lot!

Upvotes: 21

Views: 20688

Answers (4)

Sayidina ahmadal qososyi Qosyi
Sayidina ahmadal qososyi Qosyi

Reputation: 71

i think you still try using version 2xx approach, you can try this example, it's work with current version of spacy:

import spacy
from spacy.training.example import Example

nlp = spacy.load("en_core_web_sm")

# Data pelatihan dalam bentuk batch dari objek Example
train_data = [
    (text1, {"entities": [(start1, end1, "LABEL1"), (start2, end2, "LABEL2")]}),
    (text2, {"entities": [(start3, end3, "LABEL1"), (start4, end4, "LABEL3")]})
]

# Konversi data pelatihan menjadi batch dari objek Example
examples = []
for text, annotations in train_data:
    example = Example.from_dict(nlp.make_doc(text), annotations)
    examples.append(example)

# Pembaruan model dengan batch dari objek Example
nlp.update(examples, drop=0.5, losses={})

Upvotes: 1

Chinmay Das
Chinmay Das

Reputation: 410

for batch in batches:
    texts, annotations = zip(*batch)
    
    example = []
    # Update the model with iterating each text
    for i in range(len(texts)):
        doc = nlp.make_doc(texts[i])
        example.append(Example.from_dict(doc, annotations[i]))
    
    # Update the model
    nlp.update(example, drop=0.5, losses=losses)

this code is running successfully with Spacy 3. Note that here I had a tuple of string if you want to use only string don't need to use the for loop.

Upvotes: 7

Talal
Talal

Reputation: 89

Since spaCy version 3.0, they have migrated from older “simple training style” to using Example object.

from spacy.training import Example

example = Example.from_dict(nlp.make_doc(text), annotations)
nlp.update([example])

You can refer to this page on official spaCy's website.

https://spacy.io/usage/training

Upvotes: 5

krisograbek
krisograbek

Reputation: 1782

You didn't provide your TRAIN_DATA, so I cannot reproduce it. However, you should try something like this:

from spacy.training.example import Example

for batch in spacy.util.minibatch(TRAINING_DATA, size=2):
    for text, annotations in batch:
        # create Example
        doc = nlp.make_doc(text)
        example = Example.from_dict(doc, annotations)
        # Update the model
        nlp.update([example], losses=losses, drop=0.3)

Upvotes: 38

Related Questions