Unable To Extract Tables

Beginner here. I'm having issues while trying to extract data from the second (Team Statistics) and third (Team Analytics 5-on-5) Table on this page: https://www.hockey-reference.c

Solution 1:

You get this error because the read_html() method returns a list of 1 element and that element is at position 0

instead of

df = df_list[1]

use this

df = df_list[0]

You get combined table of all teams from your mentioned site so if you want to extract the table of 2nd and 3rd team use loc[] accessor:-

east_division=df.loc[9:17]
north_division=df.loc[18:25]

Solution 2:

Use the URL directly in pandas.read_html

df = pd.read_html('https://www.hockey-reference.com/leagues/NHL_2021.html')

Solution 3:

The tables are in fact there in the html (within the comments). Use BeautifulSoup to pull out the comments and parse those tables as well. The code below will pull all (both commented and uncommented tables). and put it into a list. Just a matter of pulling out the table by index that you want, in this case indices 1 and 2.

import requests
from bs4 import BeautifulSoup, Comment
import pandas as pd

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'}

url = "https://www.hockey-reference.com/leagues/NHL_2021.html"# Gets all uncommented tables
tables = pd.read_html(url, header=1)

# Get the html source
response = requests.get(url, headers=headers)

# Creat soup object form html
soup = BeautifulSoup(response.content, 'html.parser')


# Get the comments in html
comments = soup.find_all(string=lambda text: isinstance(text, Comment))

# Iterate thorugh each comment and parse the table if found# # Append the table to the tables listfor each in comments:
    if'table'instr(each):
        try:
            tables.append(pd.read_html(each, header=1)[0])
            tables = tables[tables['Rk'].ne('Rk')]
            tables = tables.rename(columns={'Unnamed: 1':'Team'})
        except:
            continue

Output:

fortablein tables[1:3]:
    print(table)
      Rk             Unnamed: 1  AvAge  GP   W  ...    S    S%    SA    SV%  SO
01.0New York Islanders   29.02818  ...  8419.87670.920512.0    Tampa Bay Lightning   28.32619  ...  79812.27250.919323.0       Florida Panthers   28.12718  ...  91810.08400.910034.0    Toronto Maple Leafs   28.92919  ...  88311.28280.909245.0    Carolina Hurricanes   27.22619  ...  81610.97590.912356.0    Washington Capitals   30.42717  ...  76812.08080.895067.0   Vegas Golden Knights   29.12518  ...  75211.06910.920478.0        Edmonton Oilers   28.43018  ...  94510.69380.907289.0          Winnipeg Jets   28.02717  ...  79511.48560.9101910.0    Pittsburgh Penguins   28.12717  ...  77911.07840.89911011.0     Chicago Blackhawks   27.22914  ...  86310.19970.91021112.0         Minnesota Wild   28.82516  ...  76410.37230.91321213.0        St. Louis Blues   28.22814  ...  83610.48350.89201314.0          Boston Bruins   28.82514  ...  7728.86650.91321415.0     Colorado Avalanche   26.82515  ...  8468.76220.90541516.0     Montreal Canadiens   28.82712  ...  8909.77820.90901617.0    Philadelphia Flyers   27.52513  ...  69911.77530.89231718.0         Calgary Flames   28.02813  ...  8388.98450.90431819.0      Los Angeles Kings   27.72611  ...  74810.38140.91021920.0      Vancouver Canucks   27.73113  ...  9518.810350.90312021.0  Columbus Blue Jackets   27.02911  ...  8399.39020.89512122.0        Arizona Coyotes   28.52712  ...  6899.78510.90712223.0        San Jose Sharks   29.32511  ...  7499.58000.89012324.0New York Rangers   25.72611  ...  7739.27460.90622425.0    Nashville Predators   28.92811  ...  8807.48370.88512526.0          Anaheim Ducks   28.4298  ...  8047.78520.89132627.0           Dallas Stars   28.3238  ...  65710.26260.90432728.0      Detroit Red Wings   29.4288  ...  7858.08700.89102829.0        Ottawa Senators   26.4309  ...  9428.29600.87402930.0New Jersey Devils   26.2248  ...  7088.57410.89623031.0         Buffalo Sabres   27.4266  ...  7287.78040.893031   NaN         League Average   28.12713  ...  8089.88080.9022

[32rows x 32 columns]
    Rk             Unnamed: 1    S%    SV%  ...  HDGF  HDC%  HDGA  HDCO%01New York Islanders   8.30.931  ...    1112.21111.812    Tampa Bay Lightning   8.70.933  ...    1114.966.323       Florida Panthers   7.90.926  ...    1514.41217.634    Toronto Maple Leafs   8.80.933  ...    1613.4811.145    Carolina Hurricanes   7.50.932  ...    1212.879.356    Washington Capitals   9.80.919  ...    1010.957.867   Vegas Golden Knights   9.30.927  ...    2015.91114.578        Edmonton Oilers   8.20.920  ...     911.3139.889          Winnipeg Jets   8.50.926  ...    1515.087.8910    Pittsburgh Penguins   8.80.922  ...    1014.51513.51011     Chicago Blackhawks   7.30.925  ...    1010.51415.11112         Minnesota Wild   9.90.930  ...    1614.2811.91213        St. Louis Blues   8.40.914  ...    1518.11515.81314          Boston Bruins   6.60.922  ...     57.41112.21415     Colorado Avalanche   6.70.916  ...     88.1813.31516     Montreal Canadiens   7.80.935  ...    1512.0811.31617    Philadelphia Flyers  10.10.907  ...    1815.9912.91718         Calgary Flames   7.60.929  ...     66.989.21819      Los Angeles Kings   7.50.925  ...    1113.189.81920      Vancouver Canucks   7.30.919  ...    1713.22017.42021  Columbus Blue Jackets   8.10.918  ...     59.61513.62122        Arizona Coyotes   7.70.924  ...    1114.71412.82223        San Jose Sharks   8.10.909  ...    1214.61614.02324New York Rangers   7.80.921  ...    1714.0812.72425    Nashville Predators   5.70.918  ...     510.61113.42526          Anaheim Ducks   7.40.909  ...    1213.32516.82627           Dallas Stars   7.40.929  ...    1113.3512.82728      Detroit Red Wings   7.50.923  ...    1315.31216.72829        Ottawa Senators   7.10.894  ...     78.62014.32930New Jersey Devils   7.20.923  ...    1014.31213.23031         Buffalo Sabres   5.80.911  ...     68.21614.0