How do I use seleniumbase's innate parallelization (-n=NUMBER) alongside pandas to concurrently append (to an .xlsx)?

[amirthfultehrani]

First, thank you to all who have maintained and contributed to SeleniumBase. It is truly such a beautiful framework, and I am so glad to have found out it. May happiness be invoked on you all.

My gratitude aside, for the love of everything, I cannot figure this out (despite having spent consecutive days on it).

Example Code:

import os
from seleniumbase import BaseCase
import pytest
import pandas as pd

class MyTestClass(BaseCase):

    def process_item(self, item):
        """Simulate some processing that takes a bit of time."""
        time.sleep(0.1)  # Simulate processing delay
        return {"item": item, "result": item * 2} 

    def setUp(self):
        super().setUp()
        self.headless = True 

    @pytest.mark.parametrize("item", range(10))
    def test_multiworker_excel(self, item):
        # Simulate data extraction
        data = self.process_item(item)

        # Store data for later Excel creation
        # (Problem: This is local to each worker process)
        all_data = [] 
        all_data.append(data)

        # Each worker creates the Excel file separately!
        df = pd.DataFrame(all_data)
        df.to_excel("output/test_data.xlsx", index=False)

# Example call (run with multiple workers):
# python -m pytest my_test.py -n=2 

The Problem:

• When running this with pytest -n=2 (or any number of workers), each worker process will have its own separate copy of all_data.
• Consequently, each worker will create an "output/test_data.xlsx" containing only the data it processed, overwriting the results from other workers.
• It's basically a race between the workers as to which will have its data in the .xlsx first (and whoever gets it first, the others are ignored/not put in the .xlsx) as opposed to all workers data being processed. As an example, imagine 10 items to be processed with two workers (-n=2), only theoretically 5 of those items, if passed, will show up in the .xlsx as opposed to 10.

Desired Outcome:

• All workers should contribute their results to a single, shared "output/test_data.xlsx".

If anyone can please help me with this, I would be very grateful. I have already tried fidgeting with thread locks, multiprocessing, even using linux based modules (I am on Windows) but to no avail. Many threads I have looked at are either outdated or do not exactly deal with the issue I am facing. Thanks in advance for any help!