Error connecting to green grass core errors with RuntimeError: 30 (AWS_ERROR_PRIORITY_QUEUE_EMPTY)

[AndreyShprengel]

Not sure if this belongs under issues as a bug but figured I would start here. We have some code that gets the stream manager port at runtime to instantiate the stream manager client,

def _get_stream_manager_port() -> str:
    # Use IPC to get the port from the stream manager component configuration.
    ipc_client = awsiot.greengrasscoreipc.connect()
    request = GetConfigurationRequest()
    request.component_name = "aws.greengrass.StreamManager"
    request.key_path = ["port"]
    operation = ipc_client.new_get_configuration()
    operation.activate(request)
    future_response = operation.get_response()
    response = future_response.result(GET_STREAM_MANAGER_PORT_TIMEOUT_SECONDS)
    stream_manager_port = str(response.value["port"])
    return stream_manager_port

This works the majority of the time but occasionally getting the ipc client fails with the following stack trace:

  File "/usr/local/lib/python3.9/site-packages/cognito/edge_common/aws_client.py", line 79, in _get_stream_manager_port
   ipc_client = awsiot.greengrasscoreipc.connect()
 File "/usr/local/lib/python3.9/site-packages/awsiot/greengrasscoreipc/__init__.py", line 51, in connect
   elg = EventLoopGroup(num_threads=1)
 File "/usr/local/lib/python3.9/site-packages/awscrt/io.py", line 87, in __init__
   self._binding = _awscrt.event_loop_group_new(num_threads, is_pinned, cpu_group, on_shutdown)
RuntimeError: 30 (AWS_ERROR_PRIORITY_QUEUE_EMPTY): Attempt to pop an item from an empty queue.

Given that the error is coming from with the aws iot library and it is very intermittent is ihard to determine weather the error is from our usage of the library or something internal. Is there something we can do to further debug this or is should this be filed under bugs?

Version info:

awscrt==0.16.13
awsiotsdk==1.12.6

Nucleus Version: 2.10.1

[jmklix]

How often is this failing? (You said this occasionally fails)
Can you provide a debug log from when this is happening?