Simple tokenizer
SimpleTokenizer
Bases: BaseTokenizer
SimpleTokenizer
Bases: BaseTokenizer
griptape/tokenizers/simple_tokenizer.py
characters_per_token: int = field(kw_only=True)
class-attribute
instance-attribute
model: str = field(kw_only=True, init=False)
class-attribute
instance-attribute
count_tokens(text)
griptape/tokenizers/simple_tokenizer.py