Memory leak

[AnikinNN]

Bug Report

Seems like there is a leakage, when we get a lot of heavy nodes
To reproduce run the code below

from tqdm import tqdm
import psutil
import asyncio
import time
from neo4j import AsyncGraphDatabase
import matplotlib.pyplot as plt

async def log_memory_usage(interval=0.05, coro_complete_event=None):
    memory_usage = []
    time_ticks = []
    process = psutil.Process()  # Get the current process
    while not coro_complete_event.is_set():  # Keep running until the build completes
        mem_info = process.memory_info().rss / (1024 * 1024)  # Convert to MB
        memory_usage.append(mem_info)
        time_ticks.append(time.time())
        await asyncio.sleep(interval)

    return memory_usage, time_ticks


def plot_memory_usage(memory_usage, time_ticks):
    time_ticks = [t - time_ticks[0] for t in time_ticks]
    plt.plot(time_ticks, memory_usage)
    plt.xlabel("Time (sec)")
    plt.ylabel("Memory (MB)")
    plt.title("RAM usage over time")
    plt.grid(True)
    plt.show()


def measure_ram(func):
    async def measure_ram_wrapper(*args, **kwargs):
        coro_complete_event = asyncio.Event()
        logging_task = asyncio.create_task(log_memory_usage(coro_complete_event=coro_complete_event))

        result = await func(*args, **kwargs)
        await asyncio.sleep(1)
        coro_complete_event.set()
        ram_consumption = await logging_task
        plot_memory_usage(*ram_consumption)

        return result

    return measure_ram_wrapper

@measure_ram
async def show_ram_leak():
    uri = f"neo4j://neo4j:7687"
    auth = ('neo4j', '12345678')

    async with AsyncGraphDatabase.driver(uri, auth=auth) as driver:
        await driver.execute_query("MATCH (n) DETACH DELETE n")

        # create a lot of heavy nodes
        long_text = 'abcde' * 2 * 10000
        await driver.execute_query(
            f'''
            WITH range(1, 100) AS ids
            UNWIND ids AS i
            CREATE (n:SomeNode {{
              text: "{long_text}" + toString(i)
            }})
            '''
        )

        # request all nodes in the loop
        for i in tqdm(range(1000)):
            await driver.execute_query("MATCH (n:SomeNode) RETURN n")


if __name__ == '__main__':
    asyncio.run(show_ram_leak())

My Environment

Python Version: Python 3.11.10 (main, Sep 27 2024, 06:09:21) [GCC 12.2.0]
Driver Version: 5.25.0
Server Version and Edition: 5.19.0
Operating System: Linux 3b9a62e16a5d 6.8.0-45-generic #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2 x86_64 GNU/Linux

[robsdedude]

Hi and thanks for reaching out.

Also thank you for the nice reproducer. That surely makes it easier to investigate.

Let me give you a somewhat unrelated advice: I know this is a reproducer and no production code. But I highly recommend to never even get used to doing string concatenation/interpolation in any query string going to a database. That might get you into a query injection vulnerability one day by accident. Use something like this instead:

        await driver.execute_query(
            '''
            WITH range(1, 100) AS ids
            UNWIND ids AS i
            CREATE (n:SomeNode {text: $long_text + toString(i)})
            ''',
            long_text=long_text
        )

Now for the actual issue: I don't think there's a memory leak here. If you adjust your code to call gc.collect() before measuring the memory consumption you'll see that the graph stays nice and flat. Something like

import gc
...

async def log_memory_usage(interval=0.05, coro_complete_event=None):
    ...
    while not coro_complete_event.is_set():  # Keep running until the build completes
        gc.collect()
        ....

Alternatively, if you let the loop run for longer (without manually calling the garbage collector), I tried 10.000, you can see the memory consumption dropping every now and then after growing a fair bit initially and then meandering around a plateau value.

I'll close the issue for now as this does not seem to be a problem. Please re-open if you observe different behavior and I'll happily pick up the investigation again.