This is possibly more of a discussion than any kind of bug report.Also documenting this method for anyone searching for a similar solution in the future.

Using the upload_protocol = serial requires manually setting jumpers and hitting reset buttons, generally. The RTS/DTR pins on most USB-UART dongles are used to trigger proper reset states on the ESP8266/32 boards, so it seems reasonable to support doing that for STM32 boards as well.

For reference, the serial upload method calls this code path:
https://github.com/platformio/platform-ststm32/blob/develop/builder/main.py#L174

This, in turns calls (I believe) this bat file, on Windows:
https://github.com/rogerclarkmelbourne/Arduino_STM32/blob/master/tools/win/serial_upload.bat

The bat file calls stm32flash, PDF manual located here:
http://stm32flash.sourceforge.net/stm32flash-manual.pdf

The manual provides an example of how to use RTS/DTR in the way I propose:

• entry sequence: RTS=low, DTR=low, DTR=high
• exit sequence: RTS=high, DTR=low, DTR=high
stm32flash −R −i −rts,−dtr,dtr:rts,−dtr,dtr /dev/ttyS0

In this case, the DTR pins needs to be connected to the STM32's nRST / RESET pin, and the RTS needs to be connected to BOOT0.

Because of the weird argument format given to stm32flash, there doesn't appear to be any way to pass the arguments via PlatformIO. Any combination of the arguments in platformio.ini's upload_flags breaks the .bat file, in part due to the positional command %1 in line 11 of the .bat file.

This can be manually hacked around by changing line 11 of .platformio\packages\tool-stm32duino\serial_upload.bat to:
stm32flash -g 0x8000000 -R -i -rts,-dtr,dtr:rts,-dtr,dtr -b 115200 -w %str% %1

This works great for flashing over serial! Very happy with it.

But you know, not having to do a dirty hack would be nice. I'm really not sure the best approach for cleanly implementing it into the framework.