Skip to content

Fix read_sql empty result with chunksize bug GH34411 #34429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

JohanKahrstrom
Copy link
Contributor

pd.read_sql was returning an empty generator when chunksize is set and the query returns zero results. Now it correctly returns a generator with a single empty DataFrame (:issue:34411).

@WillAyd WillAyd added the IO SQL to_sql, read_sql, read_sql_query label Jun 5, 2020
Copy link
Member

@WillAyd WillAyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WillAyd
Copy link
Member

WillAyd commented Jun 22, 2020

@JohanKahrstrom looks like a merge conflict - can you fix that up? ping @jorisvandenbossche or @jreback for second review

@JohanKahrstrom JohanKahrstrom force-pushed the fix-read-sql-empty-result-with-chunksize-bug-34411 branch from 019e1a1 to 27bc425 Compare June 22, 2020 18:04
@pep8speaks
Copy link

pep8speaks commented Jun 22, 2020

Hello @JohanKahrstrom! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-01-14 17:55:48 UTC

@JohanKahrstrom
Copy link
Contributor Author

@WillAyd, @jorisvandenbossche, @jreback, Merge conflict has been resolved, and commit history squashed.

@JohanKahrstrom JohanKahrstrom force-pushed the fix-read-sql-empty-result-with-chunksize-bug-34411 branch from 27bc425 to 3090358 Compare June 22, 2020 19:39
@JohanKahrstrom
Copy link
Contributor Author

... and fixed black formatting.

@WillAyd
Copy link
Member

WillAyd commented Jun 23, 2020

/azp run

@azure-pipelines
Copy link
Contributor

Azure Pipelines successfully started running 1 pipeline(s).

@JohanKahrstrom
Copy link
Contributor Author

Thanks Will, I was trying to figure out how to rerun the azure check...

@jreback
Copy link
Contributor

jreback commented Jul 17, 2020

can u merge master and we will look again

@simonjayhawkins
Copy link
Member

@JohanKahrstrom can you add a release note

…d.DataFrame if chunksize is set and the resultset is empty

Added release note
@JohanKahrstrom JohanKahrstrom force-pushed the fix-read-sql-empty-result-with-chunksize-bug-34411 branch from 3090358 to bda0257 Compare September 3, 2020 09:36
@JohanKahrstrom
Copy link
Contributor Author

@jreback @simonjayhawkins Added release note, merged with master. Unrelated checks are failing, not sure what to do about those...

@github-actions
Copy link
Contributor

github-actions bot commented Oct 8, 2020

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

@github-actions github-actions bot added the Stale label Oct 8, 2020
@dsaxton
Copy link
Member

dsaxton commented Oct 8, 2020

@JohanKahrstrom Can you merge master to fix conflicts? Looks like this might be close to ready?

@JohanKahrstrom JohanKahrstrom force-pushed the fix-read-sql-empty-result-with-chunksize-bug-34411 branch from c21d71b to 065a809 Compare October 21, 2020 11:11
@JohanKahrstrom
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link
Contributor

Commenter does not have sufficient privileges for PR 34429 in repo pandas-dev/pandas

@JohanKahrstrom
Copy link
Contributor Author

@dsaxton I've merged with master. There's 3 failing tests unrelated to my code change, and the Azure test suite failed again (I don't have sufficient privileges to re-run them).

@jreback are you happy with my argument for not implementing your suggested change (as far as I can see it won't work)?

@arw2019 arw2019 added Needs Review and removed Stale labels Nov 6, 2020
@arw2019
Copy link
Member

arw2019 commented Nov 6, 2020

@JohanKahrstrom can you merge master once more? (There's a conflict in whatsnew)

@JohanKahrstrom
Copy link
Contributor Author

@arw2019 Done.

@arw2019
Copy link
Member

arw2019 commented Nov 29, 2020

This looks ready to go.

cc @jreback @dsaxton

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JohanKahrstrom can you merge master and move the whatsnew note. i think this was ok.

@@ -669,6 +669,7 @@ I/O
- :func:`read_csv` was closing user-provided binary file handles when ``engine="c"`` and an ``encoding`` was requested (:issue:`36980`)
- Bug in :meth:`DataFrame.to_hdf` was not dropping missing rows with ``dropna=True`` (:issue:`35719`)
- Bug in :func:`read_html` was raising a ``TypeError`` when supplying a ``pathlib.Path`` argument to the ``io`` parameter (:issue:`37705`)
- :meth:`read_sql` returned an empty generator if ``chunksize`` was no-zero and the query returned no results. Now returns a generator with a single empty dataframe (:issue:`34411`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to 1.3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback Done, thanks.

@jreback jreback added this to the 1.3 milestone Jan 14, 2021
@jreback jreback merged commit eb86524 into pandas-dev:master Jan 14, 2021
@jreback
Copy link
Contributor

jreback commented Jan 14, 2021

thanks @JohanKahrstrom

luckyvs1 pushed a commit to luckyvs1/pandas that referenced this pull request Jan 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
IO SQL to_sql, read_sql, read_sql_query
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: pd.read_sql returns empty list if query has no results and chunksize is set
7 participants