sentencepiece agent skill by Orchestra-Research
Guides training language-independent SentencePiece tokenizers with BPE or Unigram directly on raw text, for multilingual, CJK, and reproducible tokenization needs.
At a glance
| Publisher | Orchestra-Research |
|---|---|
| Repository | Orchestra-Research/AI-Research-SKILLs (81 listed skills) |
| Category | AI Agents and LLM Engineering |
| License | MIT |
| Last commit | June 15, 2026 (repository) |
| SKILL.md | 225 lines, 566 words |
| Bundled files | 3 files, including references |
| Quality score | 77/100 · C Passes all seven graded checks |
| Security scan | Not scanned |
| README badge | Get the listing badge and raise the score |
Before you install
Read from the skill's files at the directory's October 19, 2026 snapshot: what ships with it, what its code reads, and the terms it is published under.
- Bundled scripts: none.
- Commands in SKILL.md: its code blocks run
pip install, which fetches or runs software from outside the skill folder. - License: MIT, an open-source or open-content license.
Install sentencepiece
The skills CLI works with Claude Code, Codex, Cursor and other agents; the other routes copy the folder by hand.
npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/02-tokenization/sentencepiecetmp="$(mktemp -d)"
git clone --depth 1 https://github.com/Orchestra-Research/AI-Research-SKILLs.git "$tmp/repo"
mkdir -p .claude/skills
cp -R "$tmp/repo/"02-tokenization/sentencepiece .claude/skills/
rm -rf "$tmp"python3 ~/.codex/skills/.system/skill-installer/scripts/install-skill-from-github.py \
--repo Orchestra-Research/AI-Research-SKILLs \
--path 02-tokenization/sentencepieceOrchestra-Research/AI-Research-SKILLs is third-party code. Read SKILL.md and any bundled scripts before you install it.
How the author describes it
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
From the description field of SKILL.md. Agents read this line to decide when to load the skill.
Sections in SKILL.md
- When to use SentencePiece
- Quick start
- Installation
- Train model
- Encode and decode
- Language-independent design
- Whitespace as symbol (▁)
- Tokenization algorithms
- BPE (Byte-Pair Encoding)
- Unigram (default)
- Training configuration
- Essential parameters
- Character coverage
- Encoding options
- Subword regularization
- Common patterns
Files in the skill folder
SKILL.mdreferences/algorithms.mdreferences/training.md
Suede Skill Check: 77/100 · C
Report snapshot · · graded-v2
The rubric totals 100 points; published scores are capped at 99. Security scan: not scanned.
Source revision: 773a52944ba4. Report version 5b1f9a38a016. Badge URLs follow the latest listing; save the report to keep this version.
Download report · Request a fresh review
What to review next
- Recent maintenance: 3 of 9 points. 9 for a repository commit within 30 days of the snapshot; 6 within 90; 3 within 180; 1 within 365.
- Boundary clause: 0 of 4 points. 4 when the description says when not to use the skill (“Not for …”, “Do not use …”, “Use … instead”).
- Example request: 0 of 3 points. 3 when the description quotes an example request (3–60 characters in quotation marks).
Add declarations and bundled files when they describe the skill’s actual needs. Update the source, then request a fresh review with the changed commit or PR.
sentencepiece passes the required frontmatter check and all seven graded checks, which cover the SKILL.md format, a declared license and a commit in the last year. Those checks earn 56 of the score’s points; the rest come from description quality, instruction depth, bundled files, maintenance and curated-list citations. How the score works.
See the checks and points
| Check | Result |
|---|---|
| Frontmatter parses and declares a name and a description | Pass |
| Name uses lowercase letters, digits and hyphens, 64 characters or fewer | Pass |
| Name matches the skill's folder name | Pass |
| Description is between 50 and 1,024 characters | Pass |
| Description says when an agent should use the skill | Pass |
| SKILL.md body is 500 lines or fewer | Pass |
| A license is declared in the frontmatter or the repository | Pass |
| Repository has a commit in the last 12 months | Pass |
| Score component | Points |
|---|---|
| Structural checks | 56 of 56 |
| Description length | 5 of 5 |
| Boundary clause | 0 of 4 |
| Example request | 0 of 3 |
| Instruction depth | 7 of 7 |
| Reference files | 3 of 3 |
| Bundled scripts | 0 of 2 |
| Recent maintenance | 3 of 9 |
| Curated-list citations | 3 of 7 |
| Tool permissions declared | 0 of 2 |
| Compatibility declared | 0 of 2 |
Show the listing and raise the score
sentencepiece is listed with 77/100 · C. This badge links to its Suede listing. Skills scoring 80–99 add a numeric and a letter quality shield; the largest open points for sentencepiece are recent maintenance (6), boundary clause (4) and example request (3). Send your repository and the points you want to raise, then request a fresh review after updating the skill.
[](https://skills.suedeai.ai/directory/orchestra-research/ai-research-skills/sentencepiece/)<a href="https://skills.suedeai.ai/directory/orchestra-research/ai-research-skills/sentencepiece/"><img src="https://skills.suedeai.ai/directory/badge.svg" alt="Suede Skills Directory: listed" height="20"></a>Where this skill is cited
Orchestra-Research/AI-Research-SKILLs appears in 2 curated lists: VoltAgent/awesome-agent-skills, skillmatic-ai/awesome-agent-skills.