High-performance .NET wrapper for https://github.com/huggingface/tokenizers
dotnet add package Tokenizers.NET
- Run HuggingFace tokenizers on .NET
- High performance, zero GC-allocations
- I don't know, TrumpMcDonaldz rock!
- OSX ( ARM64 / x86_64 )
- Linux ( ARM64 / x86_64 )
- Windows ( ARM64 / x86_64 )
( Who uses Windows ARM lol )
https://budgetdevv.github.io/Tokenizers.NET/
- Navigate to project's root directory
dotnet clean
dotnet pack -c Release -v:m /p:DownloadNativeFiles=true