CODEWITHSUNDEEP
pythonhuggingface-transformersgpt-2gpt-3
Mobeus Zoom
Mobeus Zoom

Reputation: 608

Huggingface Transformer Priming

I am trying to replicate the results of this demo, whose author primes GPT-3 with just the following text:

gpt.add_example(Example('apple', 'slice, eat, mash, cook, bake, juice'))
gpt.add_example(Example('book', 'read, open, close, write on'))
gpt.add_example(Example('spoon', 'lift, grasp, scoop, slice'))
gpt.add_example(Example('apple', 'pound, grasp, lift'))

I only have access to GPT-2, via the Huggingface Transformer. How can I prime GPT-2 large on Huggingface to replicate the above examples? The issue is that, with this, one doesn't get to prime with the input and corresponding output separately (as the author of the GPT-3 demo did above).

Similarly, this tutorial describes using Huggingface, but there's no example which clearly shows how you can prime it using input vs output examples.

Does anyone know how to do this?

Desired output: use GPT-2 to return something like, for input "potato", output "peel, slice, cook, mash, bake" (as in the GPT-3 demo: https://www.buildgpt3.com/post/41/). Obviously the exact list of output verbs won't be the same as GPT-2 and GPT-3 are not identical models.

Upvotes: 1

Views: 263

Answers (1)

Jindřich
Jindřich

Reputation: 11220

The only thing the GPT model can do is predicting what word should follow. Technically, there is no input and output, it is a decoder-only model, so it only has output. Priming the model means that you force the output of the model to something that you want and then you let the model continue generating more text.

What happens in the demo is:

  1. You provide GPT-3 with natural language examples of what it should do. Something like this:
What can I do with an apple? slice eat, mash, cook, bake, juice
What can I do with a book? read, open, close, write on
What can I do with a spoon? lift, grasp, scoop, slice
  1. When a query comes (e.g., knife), you create a similar sentence to the examples:
What can I do with a knife?
  1. Let the model continue generation until it starts a new line that starts with What or until it breaks in a strange way which can always happen with a stochastic model. (And hope, the model got the pattern that you meant in the priming examples.)

Here is an example from the HuggingFace's demo of what happens with GPT-2. The text in bold was generated by the model.

enter image description here

Upvotes: 2

Related Questions