Intermittent "RuntimeError: no decoder for OID 1015"

[piotrbulinski]

• asyncpg version: 0.14.0
• PostgreSQL version: 9.6.5
• Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
  the issue with a local PostgreSQL install?: AWS RDS
• Python version: 3.6.3
• Platform: AWS ECS using python:3.6.3 image
• Do you use pgbouncer?: No
• Did you install asyncpg with pip?: Yes
• If you built asyncpg locally, which version of Cython did you use?: N/A
• Can the issue be reproduced under both asyncio and
  uvloop?: N/A

I'm seeing intermittent errors in my logs:

File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 379, in fetchrow
    data = await self._execute(query, args, 1, timeout)
  File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1289, in _execute
    query, args, limit, timeout, return_status=return_status)
  File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1297, in __execute
    return await self._do_execute(query, executor, timeout)
  File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 1319, in _do_execute
    result = await executor(stmt, None)
  File "asyncpg/protocol/protocol.pyx", line 182, in bind_execute
  File "asyncpg/protocol/prepared_stmt.pyx", line 105, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg
  File "asyncpg/protocol/prepared_stmt.pyx", line 187, in asyncpg.protocol.protocol.PreparedStatementState._ensure_rows_decoder
RuntimeError: no decoder for OID 1015"

The query is very simple:

SELECT
        some_id,
        some_text,
       COALESCE(some_json->>'key', some_json->>'fallback_key') some_value,
       some_array_of_strings
FROM a_table

table:

some_id integer
some_text character varying(32)
some_json jsonb
some_array_of_strings character varying(16)[]

I've checked and OID 1015 is not defined in asyncpg (https://github.com/MagicStack/asyncpg/blob/master/asyncpg/protocol/pgtypes.pxi), but what bothers me is that I'm not able to reproduce the problem myself. I'm trying to send the same request to our API which caused this error and I don't get any errors in return. So it might not be caused by undefined OID for varchar_array (as found https://jdbc.postgresql.org/development/privateapi/constant-values.html), but something else. Unfortunately I'm out of ideas how to debug it and reproduce, so maybe you could help me with finding the root cause for those errors - please let me know what additional information would be useful to debug it.

[elprans]

OID 1015 is varchar[]. Array codecs are initialized dynamically when the query is introspected. It looks like either the introspection fails (unlikely), or the codec cache gets busted between after the introspection (likelier). There's been a change in 0.14.0 that increased the likelihood of this race. Have you been doing schema modifications in the temporal vicinity of the error?

[piotrbulinski]

@elprans no schema changes in a long time

[elprans]

Are you calling Connection.set_type_codec() or Connection.set_builtin_type_codec()?

[piotrbulinski]

class PostgreSQLRepository:
    MIN_POOL_SIZE = 10
    MAX_POOL_SIZE = 20

    MAX_CONNECTION_LIFETIME = 300  # seconds

    def __init__(self, db_host, db_port, db_name, db_user, db_password):
        self.pool = None
        self._host = db_host
        self._port = db_port
        self._database = db_name
        self._user = db_user
        self._password = db_password

    async def init_jsonb_codec(self, connection):
        def _encoder(value):
            return ujson.dumps(value)

        def _decoder(value):
            return ujson.loads(value)

        await connection.set_type_codec('jsonb', encoder=_encoder, decoder=_decoder, schema='pg_catalog', format='text')

    async def setup(self, loop):
        self.pool = await asyncpg.create_pool(
            host=self._host,
            port=self._port,
            database=self._database,
            user=self._user,
            password=self._password,
            loop=loop,
            min_size=self.MIN_POOL_SIZE,
            max_size=self.MAX_POOL_SIZE,
            max_inactive_connection_lifetime=self.MAX_CONNECTION_LIFETIME,
            init=self.init_jsonb_codec
        )

    async def teardown(self):
        await self.pool.close()

[elprans]

Yea. That seems to be the cause of the race. Sit tight, I'll have the fix soon.