Overview

It would be useful to add the ability for neural to get the token count for some given input. This would help prevent initiating requests that accidentally go over the maximum token count for some given model source.

This will also be useful in situations where we want to extract the maximum possible response from a model via request_token_num = model_max_token_len - context_tokens_len

Implementation

• The tokenizer should be appropriate for the respective model
• We should use an open-source (Ideally MIT) tokenizer that we can bundle to not require installing additional dependencies