CODEWITHSUNDEEP
javascriptkerastensorflow.js
Oscar
Oscar

Reputation: 520

Tokeniser from Keras to Tensorflowjs

I know this might seem duplicate, however I am not able to find the solution suitable for me. Or maybe I just need a complete example.

Here is the problem: I want to implement a webpage predicting the class of an input text, thanks to a pre-trained model. I have the json file corresponding to tensorflowjs model and both

now, I know how to load the model in a javascript object, with the async function of tensorflowjs. How can I do the same for the tokeniser? and how I can then tokenise (under the imported tokeniser) the input text?

======================= Clarification ===========================

The example of my json files can be found at these links

I tried the following code

// loadVocab function to get the vocabulary from json.
async function loadVocab() {
  var word2index = await JSON.parse(await JSON.stringify(vocabPath));
  return word2index;
}

where vocabPath is a string containing the url above.

at the end of my script I call a function init()

async function init(){
    model = await loadModel();
    word2index = await loadVocab();
    console.log(word2index["the"]); // I expect 1
}

but of course I got undefined since I guess it takes the real string of the url as a json, not the json at that url.

any idea?

Upvotes: 0

Views: 522

Answers (2)

Oscar
Oscar

Reputation: 520

I solved the issue in the following way finally,

let vocabPath = '/url/to/my/vocab.json';

async function loadVocab() {
    let vocab = await (await fetch(vocabPath)).json();
    return vocab;
}

Upvotes: 0

Sébastien Campion
Sébastien Campion

Reputation: 59

To load a vocabulary saved from python like that :

import json 
with open( 'word_dict.json' , 'w' ) as file:    
    json.dump( tokenizer.word_index , file )

You must load the JSON with an AJAX call like that:

function getJSON(url) {
    var resp ;
    var xmlHttp ;

    resp  = '' ;
    xmlHttp = new XMLHttpRequest();

    if(xmlHttp != null)
    {
        xmlHttp.open( "GET", url, false );
        xmlHttp.send( null );
        resp = xmlHttp.responseText;
    }

    return resp ;
}

var vocab = JSON.parse(getJSON('./word_dict.json'));

The python side is well explained here : Converting Python Keras NLP Model to Tensorflowjs

And a related question for the next step, how to vectorized it is here : Tensorflow.js tokenizer

Upvotes: 1

Related Questions