/ ディレクトリ / プレイグラウンド / SkillCompass
← 全サーバー ↗ GitHub
● コミュニティ Evol-ai ⚡ 即起動

SkillCompass

作者 Evol-ai · Evol-ai/SkillCompass

Evaluate Agent Skill quality — find the weakest link, fix it, and prove the fix worked with before/after metrics.

SkillCompass scores your Agent Skills on clarity, activation rate, downstream correctness, and context cost. It highlights the skill most likely to be hurting your agent's performance, suggests a fix, and re-runs the evaluation so you can show the improvement. Useful when you have a shelf of skills and don't know which are actually earning their context weight.

▶ デモを見る 📦 インストール 💡 ユースケース

なぜ使うのか

主な機能

ライブデモ

実際の動作

skillcompass-skill.replay ▶ 準備完了
0/0

インストール

作者のリポジトリ · Claude Skills の公式ドキュメント

ユースケース

実用的な使い方： SkillCompass

Find the skill pulling your agent's performance down

👤 Skill authors with 5+ skills installed ⏱ ~45 min advanced

使うタイミング： You feel the agent has gotten worse, not better, as you added skills.

フロー
  1. Diagnose the loser
    For the weakest skill, what specifically is wrong?✓ コピーしました
    → Concrete critique (vague description, conflicting with other skill, etc.)
  2. Propose a fix
    Suggest a minimal edit to SKILL.md to fix it.✓ コピーしました
    → Small, reviewable diff
  3. Re-evaluate
    Re-run the eval and show before/after.✓ コピーしました
    → Metrics improved, with evidence

結果： A measurably better skill bundle, with a reproducible eval process.

注意点
  • Gaming the eval metric instead of helping real tasks — Include task-level downstream metrics (actual agent outcomes), not just text-level
組み合わせ： skill-optimizer-skill · filesystem

Review a new skill before you publish it

👤 Skill authors shipping their first bundle ⏱ ~20 min intermediate

使うタイミング： Before pushing to GitHub and telling the world about your skill.

フロー
  1. Score the draft
    Evaluate my draft skill at ./my-skill/.✓ コピーしました
    → Dimension scores
  2. Fix obvious issues
    Apply the low-hanging suggestions✓ コピーしました
    → Edits in SKILL.md

結果： A publication-ready skill rather than a rough draft.

注意点
  • Chasing a perfect score — Ship when scores plateau — diminishing returns

組み合わせ

他のMCPと組み合わせて10倍の力を

skillcompass-skill + skill-optimizer-skill

Two complementary tools: SkillCompass ranks, skill-optimizer drills into SKILL.md patterns

Use SkillCompass to pick the worst skill; use skill-optimizer to deeply analyze its SKILL.md.✓ コピーしました
skillcompass-skill + filesystem

Operate across the full ~/.claude/skills/ directory

Evaluate every skill in ~/.claude/skills/ and give me a CSV.✓ コピーしました

ツール

このMCPが提供する機能

ツール入力呼び出すタイミングコスト
skill-scoring skill path(s) Periodic audits eval compute
weakest-link-id bundle scores After scoring 0
fix-suggestion weak skill + critique Before editing 0
before-after-eval pre/post SKILL.md After applying fixes eval compute

コストと制限

運用コスト

APIクォータ
none beyond your LLM provider (evals use LLM calls)
呼び出しあたりのトークン
evals can be heavy — budget 20–100k tokens for a full bundle scan
金額
free, MIT
ヒント
Run on one skill at a time during iteration; bundle runs only for audits

セキュリティ

権限、シークレット、影響範囲

認証情報の保管： none at skill level
データ送信先： none beyond your LLM provider

トラブルシューティング

よくあるエラーと対処法

Node errors on install

Ensure Node 20+; npm i inside the skill directory.

確認： node -v
Evals are inconsistent run-to-run

Fix the task seed and use a non-stochastic sample; record provider+model.

代替案

SkillCompass 他との比較

代替案代わりに使う場面トレードオフ
skill-optimizer-skillYou want a single skill analyzed deeply rather than a bundle rankedDepth over breadth
manual reviewYou have 1–2 skills totalDoesn't scale

その他

リソース

📖 GitHub の公式 README を読む

🐙 オープンな issue を見る

🔍 400以上のMCPサーバーとSkillsを見る