Persistent tokenization errors impacting n-gram analysis in our Keyword Density & Frequency Checker tool

hey folks, i'm running into a pretty stubborn technical challenge with our Keyword Density & Frequency Checker tool, specifically around the n-gram analysis part. we're seeing persistent, inconsistent tokenization failures that are just wrecking our n-gram generation, leading to malformed sequences and inaccurate density reports. it's not just simple whitespace or punctuation issues; we're talking about really tricky edge cases involving unicode characters, internal hyphens in compound words, and super long concatenated strings that somehow slip past our initial sanitization.

the problem manifests as tokens being incorrectly split or merged, completely throwing off the word boundaries needed for precise n-gram analysis. i've tried a bunch of regex permutations and even experimented with a few off-the-shelf NLP tokenizers like NLTK and spaCy (stripped down for performance where possible), but nothing seems to give us the robust, language-agnostic precision we need for all these edge cases. it feels like we're always one step behind, patching one issue only for another to pop up.

here's a snippet from our logs that kinda illustrates what i'm talking about:

ERROR: TokenizationException: Malformed sequence detected at index 45, input: "super-cali-fragilistic-expialidocious"
DEBUG: NGRAM_GENERATOR: Skipping invalid token: "fragilistic-expialidocious"
WARN: DensityCalculator: Skipping 2-gram due to invalid token sequence.

we need something that can handle complex word boundaries reliably across diverse text inputs, without excessive overhead. has anyone here successfully implemented or found a library for truly robust, perhaps even language-aware, tokenization that offers fine-grained control over how compound words, hyphenated terms, or even foreign language characters are treated for precise n-gram analysis? any recommendations for strategies or specific libraries that excel at this level of detail would be super helpful. help a brother out please...