Strips leading/trailing punctuation for word-level transcript alignment.
Keeps Latin and CJK letters/numbers so Mandarin captions are not treated as empty
(empty content words were falsely flagged as near-duplicates and dropped).
Parameters
token: string
A whitespace-separated token from the transcript.
Returns string
Lowercased core word, or empty when the token is punctuation-only.
Strips leading/trailing punctuation for word-level transcript alignment. Keeps Latin and CJK letters/numbers so Mandarin captions are not treated as empty (empty content words were falsely flagged as near-duplicates and dropped).