AI-Powered Chatbot Project Seeks Tokenizer Solution

A bank's AI-powered chatbot project requires a tokenizer solution to segment policy documents in PDF format. The goal is to accurately classify and retrieve policies based on categories, without relying on external services or manual labeling. Any suggestions, tools, or techniques for designing an in-house tokenizer would be valuable.

Source: https://dev.to/hassan_abbas_a984c9a3312b/need-a-tokenizer-logic-4a00