NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲Calculate the number of language model tokens for a string (blog.mastykarz.nl)
14 points by waldekm 3 days ago | 3 comments
sva_ 3 hours ago [-]

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))
NitpickLawyer 3 hours ago [-]
Does that work on anything other than openai?
topicseed 2 hours ago [-]
No but most models have their own libs you can use
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 16:53:10 GMT+0000 (Coordinated Universal Time) with Vercel.