bug: tokenizer only matches exact words and no subwords

[Green-Sky]

eg spacestation should probably be tokenized as space + station</w>, but right now it is just an unhandled token.

[DEBUG] stable-diffusion.cpp:1077 - parse 'spacestation' to [['spacestation', 1], ]
[DEBUG] stable-diffusion.cpp:469  - split prompt "spacestation" to tokens ["<|endoftext|>", ]

(resulting images have nothing to do with a spacestation)

[Green-Sky]

Additionally, alot of loras define trigger words they are trained on. Not sure if they are in the lora and should be added to the vocab at runtime, or are just rare words not correctly tokenized (issue).

[leejet]

Yes, because I haven't implemented bpe yet. I marked this as TODO in the code before