tldt — Too Long, Didn't Tokenize

tldt — Too Long, Didn't Tokenize — is a powerful CLI and library designed to optimize the way long texts are handled in AI workflows. Leveraging machine learning, it intelligently summarizes lengthy content while preserving context, making it ideal for reducing token usage in API calls, document uploads, and web crawling. Its suite of features includes support for LexRank and TextRank algorithms, HTML-to-Markdown conversion, Unicode confusables protection, text sanitization, and PII/API key cleaning, all without requiring API keys. This makes it especially suitable for developers, data scientists, and AI agents aiming to enhance privacy, security, and efficiency. Its Go library and coding skills support enable seamless integration into custom workflows and applications, offering a tailored approach to managing complex textual data. What sets tldt apart is its focus on protecting sensitive data and optimizing token consumption, ensuring cost-effective and secure AI interactions.