Implementing a Simple BPE Tokenizer in .NET
Implementing a Simple BPE Tokenizer in .NET
Mon Oct 07 2024 11 min read Yash Worlikar
Writing a basic Byte Pair Encoding (BPE) tokenizer using C# and .NET
Understanding Tokenization in Large Language Models
Understanding Tokenization in Large Language Models
Mon Sep 30 2024 6 min read Yash Worlikar
An in-depth exploration of tokenization, tokens, and their crucial role in modern LLMs