ENH: Add support for storage_options in pd.read_html

[serhatgktp]

• closes ENH: support storage_options in pandas.read_html #49944
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.1.0.rst file if fixing a bug or adding a new feature.

Added support for the storage_options keyword for pandas.read_html , so that users may pass headers alongside the HTTP request when using pandas.read_html to read from a URL.

The solution involves two parts:

1. Adding storage_options as an optional parameter to pandas.read_html and propagating it down to the method(s) where an HTTP request is made (get_handle and urlopen in pandas.io.common)
2. If necessary, add support for storage_options in the function that makes an HTTP request and returns the response

1) Propagating storage_options from read_html down to the HTTP request

The sequence of functions leading to the HTTP request are as follows:

1. read_html                                    (pandas.io.html)
2. _parse                                       (pandas.io.html)
3. parser(args)         # initialize parser     (pandas.io.html)
4. parse_tables                                 (pandas.io.html)
5. _parse_tables                                (pandas.io.html)
6. _build_doc                                   (pandas.io.html)
  i)  urlopen                                   (pandas.io.common)
  ii) get_handle                                (pandas.io.common)

storage_options must be added as an optional argument to pandas.read_html, and should be passed down to each function (or object instance) until urlopen or get_handle is reached.

2) Adding support for storage_options in the HTTP request functions

There exist two types of parsers that are used by pandas.read_html: _BeautifulSoupHtml5LibFrameParser and _LxmlFrameParser. Both of these classes implement the abstract methods of their parent class _HtmlFrameParser.

The two parsers implement _build_doc differently such that _BeautifulSoupHtml5LibFrameParser uses get_handle to make the GET request whereas _LxmlFrameParser uses urlopen.

get_handle, as it is defined in pandas.io.common, already supports the storage_options keyword, and thus doesn't require any modifying.

urlopen, however, does not expect or handle the storage_options keyword. Therefore, we must add logic such that if urlopen is called with storage_options, then the contents of storage_options should be passed as headers in the outgoing HTTP request. Otherwise, urlopen should follow the exact same logic it normally would.

For a more detailed problem analysis and solution description, please see this notion document.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[serhatgktp]

@mroeschke do you know who I could ping for a review

[mroeschke]

Could you merge in main and resolve the merge conflict?

[serhatgktp]

@mroeschke done

[mroeschke]

This shouldn't be needed. This is already handled in _get_filepath_or_buffer

[serhatgktp]

Right, I missed that. I've reverted the implementation of urlopen() to its original form, and I've used common.get_handle() in html.py instead.

[mroeschke]

Instead of HTMLUserAgentResponder.read_html_first_df, could you specify a lambda function of read_html?

[serhatgktp]

Done 👍

Tests are passing upon running pytest pandas\tests\io\test_user_agent.py from the root directory.

[mroeschke]

Looks like some of the CI checks are still failing

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.

[serhatgktp]

Hi @mroeschke - I've attempted to address the failing tests and I've merged into main. Could you reopen this PR so that I can see the CI output? Thanks!

[serhatgktp]

@mroeschke All should be good now. Please let me know if anything seems off, thanks!

[mroeschke]

Thanks for sticking with this @serhatgktp. Nice work