tiktoken
Here are 44 public repositories matching this topic...
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
-
Updated
May 27, 2024 - C#
This project harnesses the power of Tiktoken and the OpenAI API to create a Python Streamlit web application with a primary focus on token counting and price estimation.
-
Updated
May 27, 2024 - Python
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
-
Updated
May 24, 2024 - Python
PHP port for openai/tiktoken (most)
-
Updated
May 18, 2024 - PHP
forked from @anse-app/chatgpt-demo. Index site at https://free-chat.asia
-
Updated
May 16, 2024 - TypeScript
📐 GPT token estimation and context size utilities without a full tokenizer
-
Updated
May 14, 2024 - TypeScript
Extracts Google Sheets to JSONL for fine-tuning, estimates task costs with tiktoken.
-
Updated
May 21, 2024 - Python
Web crawling GUI app for non-govermental events, easy integration with custom proxy, user agents, ApenAI Api and Redis Stack. Enjoy!
-
Updated
Apr 17, 2024 - Python
An Industry Standard Tokenizer, purposed for large-scale language models like OpenAI's GPT Series.
-
Updated
Apr 18, 2024 - Python
Logseq Markdown Tokenizer is a Python application that tokenizes and estimates prices for one to many markdown files.
-
Updated
Mar 7, 2024 - Python
Analysis of OpenAI's ChatGPT
-
Updated
Jan 28, 2024 - Jupyter Notebook
Improve this page
Add a description, image, and links to the tiktoken topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the tiktoken topic, visit your repo's landing page and select "manage topics."