CODEWITHSUNDEEP
javascriptphpjsonopenai-apigpt-3
Mostafa Ezzat
Mostafa Ezzat

Reputation: 29

OpenAI GPT-3 API error: "This model's maximum context length is 2049 tokens"

I have two issues relating to the response result from OpenAI completion.

The following result doesn't return back the full text when I give a content of 500 words and prompt with "Fix grammar mistakes" (Is tokens issue?)

Enter image description here

The second issue is when the text sometimes have some double quotes or single quotes. It messes with the JSON format. So I delete any type of quotes from the content (I am not sure if it's the best solution, but I may prefer doing it in JavaScript, not PHP).

curl_setopt($ch, CURLOPT_POSTFIELDS, "{\n  \"model\": \"text-davinci-001\",\n  \"prompt\": \"" . $open_ai_prompt  . ":nn" . $content_text  . "\",\n  \"temperature\": 0,\n  \"top_p\": 1.0,\n  \"frequency_penalty\": 0.0,\n  \"presence_penalty\": 0.0\n}");

"message": "We could not parse the JSON body of your request. (HINT: This likely means you aren't using your HTTP library correctly. The OpenAI API expects a JSON payload, but what was sent was not valid JSON.

Upvotes: 0

Views: 2946

Answers (2)

Luke
Luke

Reputation: 8407

If you have tasks like "Fix grammar" or other similar text-replacement tasks for long documents, you can try to implement a find and replace method, where you ask the LLM for patches instead of the whole text.

So for example, you can prefix the input with line numbers, and ask the LLM to give you back only the patches.

There is also an library for that, which I am the author of. It is called LLM Patcher and is made for very fast text manipulation tasks, like keyword replacement or grammar fixes.

How does it work?

  1. The user provides a text and a find-and-replace query.
  2. The text is split into lines and sentences.
  3. Each line and sentence is then prefixed with a identifier that looks like <l1s1> for line 1, sentence 1.
  4. The LLM is then asked to find-and-replace the query in each line and sentence.
  5. The changes are then streamed back to the user in the form of a diff. The diff looks like <r:l1s1> string to find || string to replace.

Demo

img

Upvotes: 0

Rok Benko
Rok Benko

Reputation: 22900

Regarding token limits

First of all, I think you don't understand how tokens work: 500 words is more than 500 tokens. Use the Tokenizer to calculate the number of tokens.

As stated in the official OpenAI article:

Depending on the model used, requests can use up to 4097 tokens shared between prompt and completion. If your prompt is 4000 tokens, your completion can be 97 tokens at most.

The limit is currently a technical limitation, but there are often creative ways to solve problems within the limit, e.g. condensing your prompt, breaking the text into smaller pieces, etc.

Switch text-davinci-001 for a GPT-3 model because the token limits are higher.

GPT-3 models:

Table


Regarding double quotes in JSON

You can escape double quotes in JSON by using \ in front of double quotes like this:

"This is how you can escape \"double quotes\" in JSON."

But... This is more of a quick fix. For proper solution, see @ADyson's comment above:

Don't build your JSON by hand like that. Make a PHP object / array with the correct structure, and then use json_encode() to turn it into valid JSON, it will automatically handle any escaping etc which is needed, and you can also use the options to tweak certain things about the output - check the PHP documentation.

EDIT 1

You need to set the max_tokens parameter higher. Otherwise, the output will be shorter than your input. You will not get the whole fixed text back, but just a part of it.

EDIT 2

Now you set the max_tokens parameter too high! If you set max_tokens = 5000, this is too much even for the most capable GPT-3 model (i.e., text-davinci-003). The prompt and the completion together can be 4097 tokens.

You can figure this out if you take a look at the error you got:

"error": {"message": "This model's maximum context length is 4097 tokens, however you requested 6450 tokens (1450 in your prompt; 5000 for the completion). Please reduce your prompt; or completion length."}

Upvotes: 3

Related Questions