Python: list index out of range selenium webdriver

Question

I tried to scrape the coinmarketcap dot com with selenium webdriver, but found an error of list index of range and did not understand why this happened.

driver.get('https://coinmarketcap.com/')

#stored the all rows of coins (that is 100 on first page)
all_tr = driver.find_elements(By.XPATH,'//tbody/tr')

#scrape coin name and coin url
for index, i in enumerate(all_tr):
    coin_name = driver.find_elements(By.XPATH,'//td[3]/div/a/div/div/p')[index].text
    coin_url = driver.find_elements(By.XPATH,'//td[3]/div/a')[index].get_attribute('href')
    print(coin_name,coin_url)

It goes fine till 18 results and then generates an error of list index out of range.

Answer 1

The reason that happens is because the elements don't become visible until you scroll down on chrome. You can use a script that will scroll down and up after every loop in order to keep the elements visible. I luckily have one that I use for similar scenarios, here is what your code will look like after:

def make_elements_visible():
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        new_height =driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height
    driver.find_element(By.TAG_NAME, 'body').send_keys(Keys.HOME)

driver.get('https://coinmarketcap.com/')

all_tr = driver.find_elements(By.XPATH,'//tbody/tr')

for index, i in enumerate(all_tr):
    make_elements_visible()
    coin_name = driver.find_elements(By.XPATH, '//td[3]/div/a/div/div/p')[index].text
    coin_url = driver.find_elements(By.XPATH, '//td[3]/div/a')[index].get_attribute('href')
    print(coin_name, coin_url)

Output:

Bitcoin https://coinmarketcap.com/currencies/bitcoin/
Ethereum https://coinmarketcap.com/currencies/ethereum/
Tether https://coinmarketcap.com/currencies/tether/
BNB https://coinmarketcap.com/currencies/bnb/
USD Coin https://coinmarketcap.com/currencies/usd-coin/
XRP https://coinmarketcap.com/currencies/xrp/
Solana https://coinmarketcap.com/currencies/solana/
Terra https://coinmarketcap.com/currencies/terra-luna/
Cardano https://coinmarketcap.com/currencies/cardano/
TerraUSD https://coinmarketcap.com/currencies/terrausd/
Binance USD https://coinmarketcap.com/currencies/binance-usd/
Dogecoin https://coinmarketcap.com/currencies/dogecoin/
Avalanche https://coinmarketcap.com/currencies/avalanche/
Polkadot https://coinmarketcap.com/currencies/polkadot-new/
Shiba Inu https://coinmarketcap.com/currencies/shiba-inu/
Wrapped Bitcoin https://coinmarketcap.com/currencies/wrapped-bitcoin/
Dai https://coinmarketcap.com/currencies/multi-collateral-dai/
Polygon https://coinmarketcap.com/currencies/polygon/
NEAR Protocol https://coinmarketcap.com/currencies/near-protocol/
Litecoin https://coinmarketcap.com/currencies/litecoin/
Cronos https://coinmarketcap.com/currencies/cronos/
TRON https://coinmarketcap.com/currencies/tron/
UNUS SED LEO https://coinmarketcap.com/currencies/unus-sed-leo/
FTX Token https://coinmarketcap.com/currencies/ftx-token/
Bitcoin Cash https://coinmarketcap.com/currencies/bitcoin-cash/
Chainlink https://coinmarketcap.com/currencies/chainlink/
Cosmos https://coinmarketcap.com/currencies/cosmos/
Uniswap https://coinmarketcap.com/currencies/uniswap/
Stellar https://coinmarketcap.com/currencies/stellar/
ApeCoin https://coinmarketcap.com/currencies/apecoin-ape/
Algorand https://coinmarketcap.com/currencies/algorand/
Ethereum Classic https://coinmarketcap.com/currencies/ethereum-classic/
Monero https://coinmarketcap.com/currencies/monero/
VeChain https://coinmarketcap.com/currencies/vechain/
Filecoin https://coinmarketcap.com/currencies/filecoin/
Internet Computer https://coinmarketcap.com/currencies/internet-computer/
Hedera https://coinmarketcap.com/currencies/hedera/
Decentraland https://coinmarketcap.com/currencies/decentraland/
Elrond https://coinmarketcap.com/currencies/elrond-egld/
The Sandbox https://coinmarketcap.com/currencies/the-sandbox/
Theta Network https://coinmarketcap.com/currencies/theta-network/
PancakeSwap https://coinmarketcap.com/currencies/pancakeswap/
Tezos https://coinmarketcap.com/currencies/tezos/
THORChain https://coinmarketcap.com/currencies/thorchain/
EOS https://coinmarketcap.com/currencies/eos/
Aave https://coinmarketcap.com/currencies/aave/
Klaytn https://coinmarketcap.com/currencies/klaytn/
Fantom https://coinmarketcap.com/currencies/fantom/
STEPN https://coinmarketcap.com/currencies/green-metaverse-token/
Axie Infinity https://coinmarketcap.com/currencies/axie-infinity/
KuCoin Token https://coinmarketcap.com/currencies/kucoin-token/
Zcash https://coinmarketcap.com/currencies/zcash/
Helium https://coinmarketcap.com/currencies/helium/
Flow https://coinmarketcap.com/currencies/flow/
BitTorrent-New https://coinmarketcap.com/currencies/bittorrent-new/
Huobi Token https://coinmarketcap.com/currencies/huobi-token/
IOTA https://coinmarketcap.com/currencies/iota/
The Graph https://coinmarketcap.com/currencies/the-graph/
Maker https://coinmarketcap.com/currencies/maker/
Bitcoin SV https://coinmarketcap.com/currencies/bitcoin-sv/
eCash https://coinmarketcap.com/currencies/ecash/
TrueUSD https://coinmarketcap.com/currencies/trueusd/
Convex Finance https://coinmarketcap.com/currencies/convex-finance/
Waves https://coinmarketcap.com/currencies/waves/
Nexo https://coinmarketcap.com/currencies/nexo/
Stacks https://coinmarketcap.com/currencies/stacks/
Neo https://coinmarketcap.com/currencies/neo/
Quant https://coinmarketcap.com/currencies/quant/
OKB https://coinmarketcap.com/currencies/okb/
Kusama https://coinmarketcap.com/currencies/kusama/
Celo https://coinmarketcap.com/currencies/celo/
Chiliz https://coinmarketcap.com/currencies/chiliz/
Loopring https://coinmarketcap.com/currencies/loopring/
Harmony https://coinmarketcap.com/currencies/harmony/
Curve DAO Token https://coinmarketcap.com/currencies/curve-dao-token/
Pax Dollar https://coinmarketcap.com/currencies/paxos-standard/
Dash https://coinmarketcap.com/currencies/dash/
Gala https://coinmarketcap.com/currencies/gala/
Neutrino USD https://coinmarketcap.com/currencies/neutrino-usd/
Enjin Coin https://coinmarketcap.com/currencies/enjin-coin/
Zilliqa https://coinmarketcap.com/currencies/zilliqa/
Basic Attention Token https://coinmarketcap.com/currencies/basic-attention-token/
Mina https://coinmarketcap.com/currencies/mina/
NEM https://coinmarketcap.com/currencies/nem/
Arweave https://coinmarketcap.com/currencies/arweave/
Decred https://coinmarketcap.com/currencies/decred/
Amp https://coinmarketcap.com/currencies/amp/
XDC Network https://coinmarketcap.com/currencies/xinfin/
Kava https://coinmarketcap.com/currencies/kava/
Kyber Network Crystal v2 https://coinmarketcap.com/currencies/kyber-network-crystal-v2/
Compound https://coinmarketcap.com/currencies/compound/
Holo https://coinmarketcap.com/currencies/holo/
Theta Fuel https://coinmarketcap.com/currencies/theta-fuel/
Kadena https://coinmarketcap.com/currencies/kadena/
yearn.finance https://coinmarketcap.com/currencies/yearn-finance/
Moonbeam https://coinmarketcap.com/currencies/moonbeam/
PAX Gold https://coinmarketcap.com/currencies/pax-gold/
Gnosis https://coinmarketcap.com/currencies/gnosis-gno/
Secret https://coinmarketcap.com/currencies/secret/
Audius https://coinmarketcap.com/currencies/audius/

Process finished with exit code 0