Python/Beautiful soup. Extracting all the text from <li> tags between a h2 and h3 tag

Question

What I am trying to do: There are 3 lists of food additives on this website and I am trying to extract them to get 3 different lists. they are in <ul> and <li> tags, which are between <h2> and <h3> tags. I want to find the first h2, extract all the lis below it into a list, when the next h tag (h3) is reached, start a new list and extract all the lis below that and continue this for the third list.

What I've already tried: I have read around and found a question very similar to mine. BeautifulSoup: How do I extract all the <li>s from a list of <ul>s that contains some nested <ul>s? I've tried to apply the logic of that answer, but it hasn't worked for me.

Before I start making lists, I am running print statements to see what the output is.

import urllib.request as request
import bs4 as bs

sauce = request.urlopen("https://www.foodadditivesworld.com/articles/banned-food-additives.html").read()
soup = bs.BeautifulSoup(sauce, 'lxml')


firstH2 = soup.find('h2') # Start here
# print(firstH2.text)
# print(firstH2.findNextSiblings())
uls = []
for sib in firstH2.findNextSiblings():
#     print(child.name)
    if sib.name=='h3':
        print(sib)
        break
    elif sib.name == 'div':
        print(sib.text)
        continue
        for c in sib.descendants:
            if c.name=='li':
                print (c)

What is happening: the code is doing basically what I want, but it should break the first time it runs into a h3 tag, but it doesn't, it continues to the second h3 tag before stopping. Why is it missing this first occurrence?

Answer 1

You can scrape the h2 and ul tags, and then use itertools.groupby:

import requests, itertools, re
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.foodadditivesworld.com/articles/banned-food-additives.html').text, 'html.parser')
_, *data = [[i.name, i] for i in d.find_all(re.compile('h2|h3|ul'))]
new_data = [[a, list(b)] for a, b in itertools.groupby(data, key=lambda x:x[0] == 'h2' or x[0] == 'h3')]
new_result = [[new_data[i][-1][0][-1].text, [c.text for b in new_data[i+1][-1] for c in b[-1].find_all('li')]] for i in range(0, len(new_data), 2)]

Output:

[['Banned Food Additives in US', ['Calamus extract', 'Calamus oil', 'Calcium cyclamate', 'Chlorofluorocarbons', 'cinnamyl anthranilate', 'Cobaltous chloride', 'Cobalt sulfate', 'Coumarin', 'Cyclamate', 'Diethyl pyrocarbonatec', 'Dulcin', 'Fd&c green no. 1', 'Fd&c green no. 2', 'Fd&c red no. 3, aluminum lake', 'CFd&c red no. 3, calcium lake', 'Fd&c red no. 1', 'Fd&c red no. 2', 'Fd&c red no. 4', 'Fd&c violet no. 1', 'Magnesium cyclamate', 'Nordihydroguaiaretic acid', 'Potassium cyclamate', 'P-4000', 'Safrole', 'Sodium cyclamate', 'Thiourea']], ['UK Food Additives Banned in Australia and New Zealand', ['E131 Patent Blue V', 'E154 Brown FK', 'E161g Canthaxanthin', 'E180 Litholrubine BK']], ['Preservatives', ['E214 Â\xa0 Ethyl p-hydroxybenzoate', 'E215 Â\xa0 Sodium ethyl p-hydroxybenzoate', 'E219 Â\xa0 Sodium methyl p-hydroxybenzoate', 'E226 Â\xa0 Calcium sulphite', 'E227 Â\xa0 Calcium hydrogen sulphite', 'E230 Â\xa0 Biphenyl; diphenyl', 'E231 Â\xa0 Orthophenyl phenol', 'E232 Â\xa0 Sodium orthophenyl phenol', 'E239 Â\xa0 Hexamethylene tetramine', 'E284 Â\xa0 Boric acid', 'E285 Â\xa0 Sodium tetraborate; borax', 'E356 Â\xa0 Sodium adipate antioxidant']], ['Stabilisers, Thickeners and Gelling Agents Emulsifiers', ['E417 Â\xa0 Tara gum', 'E425 Â\xa0 Konjac', 'E426 Â\xa0 Soybean hemicellulose', 'E226 Â\xa0 Calcium sulphite', 'E432 Polyoxyethylene sorbitan monolaurate; Polysorbate 20', 'E434 Polyoxyethylene sorbitan monopalmitate; Polysorbate 40', 'E459 Â\xa0 Beta-cyclodextrin', 'E462 Â\xa0 Ethyl cellulose', 'E468 Â\xa0 Crosslinked sodium carboxy methyl cellulose', 'E472d Â\xa0 Tartaric acid esters of mono- and diglycerides of fatty acids', 'E474 Â\xa0 Sucroglycerides', 'E483 Â\xa0 Stearyl tartrate', 'E493 Â\xa0 Sorbitan monolaurate', 'E494 Â\xa0 Sorbitan monooleate', 'E495 Â\xa0 Sorbitan monopalmitate', 'E513 Â\xa0 Sulphuric acid', 'E517 Â\xa0 Ammonium sulphate', 'E520 Â\xa0 Aluminium sulphate', 'E521 Â\xa0 Aluminium sodium sulphate', 'E522 Â\xa0 Aluminium potassium sulphate', 'E523 Â\xa0 Aluminium ammonium sulphate', 'E524 Â\xa0 Sodium hydroxide', 'E525 Â\xa0 Potassium hydroxide', 'E527 Â\xa0 Ammonium hydroxide', 'E528 Â\xa0 Magnesium hydroxide', 'E538 Â\xa0 Calcium ferrocyanide', 'E553a Â\xa0 (i) Magnesium silicate', 'E553b Â\xa0 Talc E574 Â\xa0 Gluconic acid', 'E576 Â\xa0 Sodium gluconate', 'E585 Â\xa0 Ferrous lactate', 'E626 Â\xa0 Guanylic acid', 'E628 Â\xa0 Dipotassium guanylate', 'E629 Â\xa0 Calcium guanylate', 'E630 Â\xa0 lnosinic acid', 'E632 Â\xa0 Dipotassium inosinate', 'E633 Â\xa0 Calcium inosinate', "E634 Â\xa0 Calcium 5'-ribonucleotides", 'E650 Â\xa0 Zinc acetate', 'E900 Â\xa0 Dimethylpolysiloxane', 'E902 Â\xa0 Candelilla wax', 'E905 Â\xa0 Microcrystalline wax', 'E912 Â\xa0 Montan acid esters', 'E927b Â\xa0 Carbamide', 'E938 Â\xa0 Argon', 'E939 Â\xa0 Helium', 'E948 Â\xa0 Oxygen', 'E949 Â\xa0 Hydrogen', 'E959 Â\xa0 Neohesperidine DC', 'E962 Â\xa0 Salt of aspartame-acesulfame', 'E999 Â\xa0 Quillaia extract', 'E1103 Â\xa0 Invertase', 'E1202 Â\xa0 Polyvinylpolypyrrolidone', 'E1204 Â\xa0 Pullulan', 'E1451 Â\xa0 Acetylated oxidised starch', 'E1452 Â\xa0 Starch aluminium Octenyl succinate', 'Annatto ExtractM', 'Anthocyanins', 'Lake Allura Red', 'Lake Amaranth', 'Solvent Black 5', 'Solvent Black 7', 'Pigment Fast Yellow G', 'Pigment Green B', 'FD&C; Blue No.2 ', 'FD&C; Blue No.1 ', 'Beverages ', 'Confectionery ', 'Anticaking Agents ', 'Color Retention Agents ']]]

printing result:

print('\n\n'.join('  {}\n{}'.format(a, '\n'.join(f'\t-{i}' for i in b)) for a, b in new_result))

Output:

Banned Food Additives in US
-Calamus extract
-Calamus oil
-Calcium cyclamate
-Chlorofluorocarbons
-cinnamyl anthranilate
-Cobaltous chloride
-Cobalt sulfate
-Coumarin
-Cyclamate
-Diethyl pyrocarbonatec
-Dulcin
-Fd&c green no. 1
-Fd&c green no. 2
-Fd&c red no. 3, aluminum lake
-CFd&c red no. 3, calcium lake
-Fd&c red no. 1
-Fd&c red no. 2
-Fd&c red no. 4
-Fd&c violet no. 1
-Magnesium cyclamate
-Nordihydroguaiaretic acid
-Potassium cyclamate
-P-4000
-Safrole
-Sodium cyclamate
-Thiourea

UK Food Additives Banned in Australia and New Zealand
-E131 Patent Blue V
-E154 Brown FK
-E161g Canthaxanthin
-E180 Litholrubine BK

Preservatives
-E214   Ethyl p-hydroxybenzoate
-E215   Sodium ethyl p-hydroxybenzoate
-E219   Sodium methyl p-hydroxybenzoate
-E226   Calcium sulphite
-E227   Calcium hydrogen sulphite
-E230   Biphenyl; diphenyl
-E231   Orthophenyl phenol
-E232   Sodium orthophenyl phenol
-E239   Hexamethylene tetramine
-E284   Boric acid
-E285   Sodium tetraborate; borax
-E356   Sodium adipate antioxidant

Stabilisers, Thickeners and Gelling Agents Emulsifiers
-E417   Tara gum
-E425   Konjac
-E426   Soybean hemicellulose
-E226   Calcium sulphite
-E432 Polyoxyethylene sorbitan monolaurate; Polysorbate 20
-E434 Polyoxyethylene sorbitan monopalmitate; Polysorbate 40
-E459   Beta-cyclodextrin
-E462   Ethyl cellulose
-E468   Crosslinked sodium carboxy methyl cellulose
-E472d   Tartaric acid esters of mono- and diglycerides of fatty acids
-E474   Sucroglycerides
-E483   Stearyl tartrate
-E493   Sorbitan monolaurate
-E494   Sorbitan monooleate
-E495   Sorbitan monopalmitate
-E513   Sulphuric acid
-E517   Ammonium sulphate
-E520   Aluminium sulphate
-E521   Aluminium sodium sulphate
-E522   Aluminium potassium sulphate
-E523   Aluminium ammonium sulphate
-E524   Sodium hydroxide
-E525   Potassium hydroxide
-E527   Ammonium hydroxide
-E528   Magnesium hydroxide
-E538   Calcium ferrocyanide
-E553a   (i) Magnesium silicate
-E553b   Talc E574   Gluconic acid
-E576   Sodium gluconate
-E585   Ferrous lactate
-E626   Guanylic acid
-E628   Dipotassium guanylate
-E629   Calcium guanylate
-E630   lnosinic acid
-E632   Dipotassium inosinate
-E633   Calcium inosinate
-E634   Calcium 5'-ribonucleotides
-E650   Zinc acetate
-E900   Dimethylpolysiloxane
-E902   Candelilla wax
-E905   Microcrystalline wax
-E912   Montan acid esters
-E927b   Carbamide
-E938   Argon
-E939   Helium
-E948   Oxygen
-E949   Hydrogen
-E959   Neohesperidine DC
-E962   Salt of aspartame-acesulfame
-E999   Quillaia extract
-E1103   Invertase
-E1202   Polyvinylpolypyrrolidone
-E1204   Pullulan
-E1451   Acetylated oxidised starch
-E1452   Starch aluminium Octenyl succinate
-Annatto ExtractM
-Anthocyanins
-Lake Allura Red
-Lake Amaranth
-Solvent Black 5
-Solvent Black 7
-Pigment Fast Yellow G
-Pigment Green B
-FD&C; Blue No.2 
-FD&C; Blue No.1 
-Beverages 
-Confectionery 
-Anticaking Agents 
-Color Retention Agents