Skip to content

Fix / implement GH-15287: add a lazy fetch to Pdo\PgSql #15750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

outtersg
Copy link
Contributor

@outtersg outtersg commented Sep 4, 2024

Make Pdo\PgSql accept Pdo::setAttribute(PDO::ATTR_PREFETCH, 0) to enter libpq's single row mode.
This avoids storing the whole result set in memory before being able to call the first fetch().

@outtersg
Copy link
Contributor Author

outtersg commented Sep 4, 2024

This branch is currently uncomplete as I am struggling to make it work on

$stmt = $pdo->query(…);

as opposed to

$stmt = $pdo->prepare(…);
$stmt->execute();

which works.

This seems to be related by the fact that prepare() preserves the stmt,
while query() releases it just after having executed it (and thus the second fetch() fails).

This is understandable in the traditional fetch mode (a prepared statement might be reused later, while a singly-executed one is of no interest anymore once the result has been received),
but in lazy fetch mode this is problematic (the first result fetch triggers the freeing of the statement before we can fetch more results),
and I don't understand which code triggers that.

Anyone pointing me to the (maybe obvious :-\) difference between execute (in pdo_stmt.c) and query (in pdo_dbh.c), or anywhere else in PDO, would be of a great and appreciated help.

@outtersg outtersg marked this pull request as draft September 4, 2024 21:40
@outtersg outtersg changed the title Fix GH-15287: Pdo\PgSql lacks a lazy fetch Fix / implement GH-15287: add a lazy fetch to Pdo\PgSql Sep 4, 2024
@devnexen
Copy link
Member

devnexen commented Sep 4, 2024

Nice attempt ! I had a quick glance, I ll have a better look in few days. Fetch with query not working indeed but also I get a crash because of an empty error message.

@outtersg outtersg force-pushed the gh-15287 branch 4 times, most recently from a994067 to 6f270c4 Compare September 5, 2024 00:13
@outtersg
Copy link
Contributor Author

outtersg commented Sep 5, 2024

@devnexen wrote:

[…] but also I get a crash because of an empty error message.

_pdo_pgsql_trim_message() seems a bit fragile (not able to handle empty strings, due to the unsigned size_t - 1).
Correcting it would be easy, but would mask the real problem (of something passing an empty error message).

But I don't reproduce it here: did you get it with the provided .phpt, or is it triggered by another test case I would be glad to work on?

@devnexen
Copy link
Member

devnexen commented Sep 5, 2024

@devnexen wrote:

[…] but also I get a crash because of an empty error message.

_pdo_pgsql_trim_message() seems a bit fragile (not able to handle empty strings, due to the unsigned size_t - 1). Correcting it would be easy, but would mask the real problem (of something passing an empty error message).

It is definitively just the symptom.

But I don't reproduce it here: did you get it with the provided .phpt, or is it triggered by another test case I would be glad to work on?

Your new test but I build with asan.

@outtersg
Copy link
Contributor Author

outtersg commented Sep 6, 2024

Multiple reasons required a reorganize of the process, as diagnosed thanks to the failing test's code structure:

$stmt = $pdo->query(…); $stmt->fetchAll();
$stmt = $pdo->query(…); $stmt->fetchAll();

Internally, after having called the second ->query() (which did a PQprepare + the first PQgetResult),
the "=" of the returned value overwrites $stmt: the old value gets GC-decremented and thus freed,
calling the destructor, which contains the DEALLOCATE to tell that the DB statement is of no use anymore.

But:

  • PQgetResult() works on the connection, not on the statement: the results cleaning I put in the dtor thus ate the results of the new query :-\
  • libpq (or the server?) doesn't like pdo_stmt_00000001 being DEALLOCATEd between pdo_stmt_00000002 preparation and first getResult(), and the subsequent getResult()s. Does it make libpq consider the deceased pdo_stmt_00000001 to be the current statement (and try to getResult() on it?). I don't know but this is what makes the next getResult return a 7 PGRES_FATAL_ERROR
  • additionnaly, that PGRES_FATAL_ERROR comes with no PQerrorMessage (hence the subsequent error on trim_message). At least for diagnosis purposes it would be a good idea to handle that specific error by setting a generic message instead of simply the code 7.

@outtersg outtersg marked this pull request as ready for review September 6, 2024 12:29
@devnexen
Copy link
Member

devnexen commented Sep 7, 2024

Looking good overall, I ve tested your branch for a while. That will be for after 8.4 tough. few nitpicks so far.

@outtersg
Copy link
Contributor Author

outtersg commented Sep 8, 2024

@devnexen wrote:

That will be for after 8.4 tough.

That's fine. In fact somewhere on StackOverflow I already wrote that "hopefully on PHP 8.5 this will be possible".

@outtersg outtersg force-pushed the gh-15287 branch 2 times, most recently from b011c0e to d8a6cd4 Compare September 8, 2024 06:30
@devnexen
Copy link
Member

Looking nice ! I ll give another round of tests in few days.

Copy link
Member

@devnexen devnexen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good implementation-wise.

set S->result to NULL after having freed it (not crucial now, as it is immediately recycled)
fetch part of the lazy fetch mode for Pdo\Pgsql
Starts fixing php#15287
use Pdo::setAttribute(PDO::ATTR_PREFETCH, 0)
memory & non-regression test
mention in the NEWS file
reindent (the previous commits tried to get the minimal diff by _not_ reindenting preserved code that got if(){}'ed)
@devnexen
Copy link
Member

Thanks !

jorgsowa pushed a commit to jorgsowa/php-src that referenced this pull request Oct 1, 2024
Make Pdo\PgSql accept Pdo::setAttribute(PDO::ATTR_PREFETCH, 0) to enter libpq's single row mode.
This avoids storing the whole result set in memory before being able to call the first fetch().

close phpGH-15750
jorgsowa pushed a commit to jorgsowa/php-src that referenced this pull request Oct 1, 2024
Make Pdo\PgSql accept Pdo::setAttribute(PDO::ATTR_PREFETCH, 0) to enter libpq's single row mode.
This avoids storing the whole result set in memory before being able to call the first fetch().

close phpGH-15750
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 5, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 5, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query;
additionnally, fix the condition to interrupt the previous request (we shall test if *it* was lazy, not if the *new* one will)
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 7, 2024
…l queries

not only the statements, but the driver, love to make internal queries. We make sure no unfinished query still runs when having to pass an internal one.
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 10, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query;
additionnally, fix the condition to interrupt the previous request (we shall test if *it* was lazy, not if the *new* one will)
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 10, 2024
…l queries

not only the statements, but the driver, love to make internal queries. We make sure no unfinished query still runs when having to pass an internal one.
by the way factorize the loops that consumed the preceding query's results
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 23, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query;
additionnally, fix the condition to interrupt the previous request (we shall test if *it* was lazy, not if the *new* one will)
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 23, 2024
…l queries

not only the statements, but the driver, love to make internal queries. We make sure no unfinished query still runs when having to pass an internal one.
by the way factorize the loops that consumed the preceding query's results
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 23, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query;
additionnally, fix the condition to interrupt the previous request (we shall test if *it* was lazy, not if the *new* one will)
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 23, 2024
…l queries

not only the statements, but the driver, love to make internal queries. We make sure no unfinished query still runs when having to pass an internal one.
by the way factorize the loops that consumed the preceding query's results
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 23, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query;
additionnally, fix the condition to interrupt the previous request (we shall test if *it* was lazy, not if the *new* one will)
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 23, 2024
…l queries

not only the statements, but the driver, love to make internal queries. We make sure no unfinished query still runs when having to pass an internal one.
by the way factorize the loops that consumed the preceding query's results
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 24, 2024
doing an (internal) query to fetch metadata from the server broke the currently-running query;
additionnally, fix the condition to interrupt the previous request (we shall test if *it* was lazy, not if the *new* one will)
outtersg added a commit to outtersg/php-src that referenced this pull request Oct 24, 2024
…l queries

not only the statements, but the driver, love to make internal queries. We make sure no unfinished query still runs when having to pass an internal one.
by the way factorize the loops that consumed the preceding query's results
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants