Questions generation in question answering +NLP

Question

I have a dataset (around 3K to 4K) excel files, each of them has around more or less 12K records which are combinations of FAQs, Email Conversations, comments from the blogs, chats etc.

Best part is, it has 2 columns one for Questions and another for Answers.

One Sample record from an excel- (Note - can't expose client data so creating only one record at my own to explain the scenario).

eg. Sample Question - What are IIT colleges in India?

Sample Answer - The Indian Institutes of Technology (IITs) are autonomous public institutes of higher education, located in India. They are governed by the Institutes of Technology Act, 1961 which has declared them as institutions of national importance and lays down their powers, duties, and framework for governance. The Institutes of Technology Act, 1961 lists twenty-three institutes.Each IIT is autonomous, linked to the others through a common council (IIT Council), which oversees their administration. The Minister of Human Resource Development is the ex officio Chairperson of the IIT Council. As of 2018, the total number of seats for undergraduate programs in all IITs is 11,279.

Client's requirement is-

Generate as many as simple questions from (above sample answer) paragraph along with their answers and append it in the same excel.

(he will then process each excel further by feeding it to his some tool which generates chat-bot stories).

eg.

• Are IITs autonomous? (Answer: Yes)
• What governs the IITs? (Answer: The Institutes of Technology Act, 1961)
• In which country IITs are located? (Answer: India)
• How many institutes does The Institutes of Technology Act, 1961 lists? (Answer:twenty-three) etc.

Answers generation I can do it using AllenAI, but not sure how to generate questions? I tried a repo but it looks incomplete and need more efforts as I'm newbie to NLP or ML, so not getting how to do those changes.

Any help on generating questions in question answering?

Can I create any model on top of existing linguistic model such as spacy's models to generate entities and then generate the questions?

Answer 1

• Yes, there are services you can use to generate questions automatically e.g https://app.flexudy.com that also has a REST API.

• Yes you can build question generation models using HuggingFace Transformer Sequence to Sequence transformer models. https://huggingface.co/models For example, you can fine-tune Bert2Bert or similar models. E.g this notebook on translation: https://github.com/huggingface/notebooks/blob/master/examples/translation.ipynb

Instead of using source and target languages as input and output respectively, use your passages and questions instead.