CODEWITHSUNDEEP
mediawikimediawiki-api
mcfly soft
mcfly soft

Reputation: 11665

get all sections separately with wikimedia api

I try to get all seperate sections of a wikipedia article through the api.

I know already :

Howto retrieve all sections seperately with one request ? (JSON Array for example)

Upvotes: 8

Views: 6985

Answers (3)

chipfall
chipfall

Reputation: 360

As has been pointed out, you can't do it all on one request but you can do it all in one line, broken out:

curl -s "http://myserver/mywiki/api.php?action=parse&format=json&page=Testpage&prop=sections" |\
 jq -r '.parse.sections[] | .index' |\
 xargs -I {} -n 1  curl -s "http://myserver/mywiki/api.php?action=parse&page=Testpage&format=json&prop=wikitext&section={}" |\
 jq '.parse.wikitext."*"' | xargs -I {} -0 -n 1 echo -e {}

explanation:

  1. curl -s keeps it quiet. you may need -k for https
  2. the first jq grabs the indexes of the returned array, aka sections
  3. we use xargs to grab each section as json
  4. and pass that back to get the wikitext of each section
  5. finally passing each to echo -e to interpret escapes
  6. the -0 stops some metacharacter being interpreted by xargs

This of course does not look much different from grabbing the page but you can change the first jq slightly to

jq -r ".parse.sections[] | select(.line == \"$section\") | .index"

and limit to one section. You did not ask this but it's a useful as a poor man's supplement to man pages. Written as a bash function one could recall a specifically named condensed section of a larger self-linked page at the command line. Man doesn't cover everything and it's been around since the start of Unix exactly because no one can remember everything and get it right, especially not chatGPT. Thanks Nemo for your original answer.

Upvotes: 0

mydoghasworms
mydoghasworms

Reputation: 18613

I realize this question was asked four years ago, so possibly the following was not available then:

You can use the REST API described here: https://www.mediawiki.org/wiki/REST_API

The REST endpoints are described/documented here: https://en.wikipedia.org/api/rest_v1/#/

The mobile-sections endpoint (intended for consuming info for a mobile device) gives you a nice breakdown with headings, which sounds like what you are asking for.

Alternatively, the metadata endpoint returns a toc (table of contents) section which contains the same breakdown of headings.

Here is an example URL, fetching the mobile sections for the "Egyptian pyramids" page: https://en.wikipedia.org/api/rest_v1/page/mobile-sections/Egyptian_pyramids

The advantage is that the response is in JSON format (which is what you were asking for).

Upvotes: 6

Nemo
Nemo

Reputation: 2572

What you ask is called parsing, because it requires interpretation of the wikitext source to split the page by sections etc. So the solution is given in https://www.mediawiki.org/wiki/API:Parsing_wikitext

1) Get the list of sections: https://www.mediawiki.org/w/api.php?action=parse&page=API:Parsing_wikitext&prop=sections

2) Ask the parsed wikitext of that section: https://www.mediawiki.org/w/api.php?action=parse&page=API:Parsing_wikitext&section=1&prop=text

Upvotes: 8

Related Questions