NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲Calculate the number of language model tokens for a string (blog.mastykarz.nl)
17 points by waldekm 370 days ago | 3 comments
sva_ 367 days ago [-]

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))
NitpickLawyer 367 days ago [-]
Does that work on anything other than openai?
topicseed 367 days ago [-]
No but most models have their own libs you can use
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 13:34:01 GMT+0000 (Coordinated Universal Time) with Vercel.