Py.Cafe

maartenbreddels/

tokenizers-demo

Streamlit Tokenizer Analysis

DocsPricing
  • app.py
  • requirements.txt
  • tokenizers-0.20.2.dev0-cp312-cp312-pyodide_2024_0_wasm32.whl
app.py
1
2
3
4
5
6
7
8
9
10
import streamlit as st
from tokenizers import Tokenizer
from tokenizers.models import BPE

tokenizer = Tokenizer(BPE())

from tokenizers.pre_tokenizers import Whitespace

tokenizer.pre_tokenizer = Whitespace()
st.write("This thing imports")
requirements.txt
1
2
3
4
5
6

streamlit==1.27.2  # currently pinned to this version
numpy
altair
pandas
https://py.cafe/files/maartenbreddels/tokenizers-demo/tokenizers-0.20.2.dev0-cp312-cp312-pyodide_2024_0_wasm32.whl