BenchmarkFree
Start Eval
Public Eval Standardv0.0.1

Global Quality RankingEvaluation Method

A cross-scenario quality aggregation standard for the public prompt library: judge (evaluator model) scoring as the foundation, combined with user score acceptance, relative ranking within each eval run, then aggregated into a comparable global score. Rules are transparent and versioned.

  • Fair & Comparable

    Percentile ranking within each eval run eliminates judge scale differences; quick eval and batch eval are weighted equally.

  • Public Rules

    Algorithm version, weight thresholds, and listing conditions are fully published; the leaderboard follows the same rules as this article.

  • Human-in-the-Loop

    Judge scoring plus user accept/reject validation; manual scoring does not affect rankings, reducing subjective bias.

Chapter 01

Design Principles

Balancing fairness and interpretability: the primary score reflects quality only; speed and cost are shown separately.

  • Only judge scores are used; user accept/reject adjusts credibility, not a fifth dimension score
  • Judge dimension counts vary by scenario; each sample's dimensions are averaged equally before ranking
  • Percentile ranking within each eval run reduces impact of different judge scoring scales
  • Quick eval sessions and batch eval tasks enter the public board equally; private prompts produce a private index only, excluded from main sort
  • Speed and token cost are excluded from the primary score; available as secondary metrics or alternate sort keys

Method Overview

Five principles run through the full calculation pipeline below.

From single evaluation to public ranking

  1. Public libraryQuick eval + batch eval
  2. Judge scoringAuto dimensions
  3. User validationScore acceptance weight
  4. In-run rankingPercentile
  5. Global scoreCross-scenario

Calculation Flow

Six-step aggregation from a single evaluation result to the global primary score.

  1. 1

    Sample Filtering

    Only completed evaluations with valid judge scores and no explicit user rejection are included.

  2. 2

    Single Sample Quality Score

    Equal-weight mean across all judge dimensions for that sample; score range 1–5.

  3. 3

    User Score Acceptance Weighting

    Accepted weight 1.0; rejected excluded; unconfirmed counted at 0.4.

  4. 4

    Unit Ranking

    Within the same eval run (batch eval task or quick eval session), weighted quality scores convert to 0–1 percentile.

  5. 5

    Sample Size Discount

    When samples are insufficient, ranking contribution is discounted by min(1, n / 50) to avoid small-sample spikes.

  6. 6

    Cross-Unit Aggregation

    Weighted average across all public eval runs the model participated in yields the global primary score GlobalScore.

Data Sources

The public board counts only valid evaluations on the public prompt library; both entry types follow the same rules.

Batch Eval

In completed public batch eval tasks, each model response is included after judge scoring and user score acceptance.

Quick Eval

Quick comparison sessions on the public library count as one eval run when scoring is complete and at least 2 models are in the same session.

When the algorithm version upgrades, this page updates accordingly; the leaderboard uses the same rules as this article.

Chapter 02

Core Formula

Definitions for the global primary score and secondary metrics, consistent with actual leaderboard calculation.

1.Single Sample Quality Score

qi=1∣Di∣∑d∈Disi,d(s∈[1, 5])q_i = \frac{1}{|D_i|} \sum_{d \in D_i} s_{i,d} \qquad (s \in [1,\,5])

Dᵢ is the set of valid Judge dimensions for that sample.

2.User Score Acceptance Weighting

wi={1.0User accepted0User rejected (excluded)0.4Unconfirmedw_i = \begin{cases} 1.0 & \text{User accepted} \\ 0 & \text{User rejected (excluded)} \\ 0.4 & \text{Unconfirmed} \end{cases}

Rejecting a score treats that judge score as invalid; it is not counted as 0.

3.In-unit weighted quality mean

qˉm,u=∑iwi qi∑iwi\bar{q}_{m,u} = \frac{\sum_i w_i \, q_i}{\sum_i w_i}

4.Unit Ranking

Rm,u=rank(qˉm,u)−1Nu−1∈[0, 1]R_{m,u} = \frac{\mathrm{rank}(\bar{q}_{m,u}) - 1}{N_u - 1} \in [0,\,1]

When only 1 model is in a unit, R = 1.0; the public unit pool requires at least 2 models.

5.Sample Size Discount

Cm,u=min⁡ ⁣(1, nm,u50),R~m,u=Cm,u⋅Rm,uC_{m,u} = \min\!\left(1,\, \frac{n_{m,u}}{50}\right), \qquad \tilde{R}_{m,u} = C_{m,u} \cdot R_{m,u}

6.Global Quality Ranking

GlobalScorem=∑uwu R~m,u∑uwu\mathrm{GlobalScore}_m = \frac{\sum_u w_u \, \tilde{R}_{m,u}}{\sum_u w_u}

Averaged only over public eval runs the model actually participated in; no zero-fill for non-participation.

7.Private Prompt Index

PrivateIndexm=∑u∈Uprivatewu R~m,u∑u∈Uprivatewu\mathrm{PrivateIndex}_m = \frac{\sum_{u \in \mathcal{U}_{\mathrm{private}}} w_u \, \tilde{R}_{m,u}}{\sum_{u \in \mathcal{U}_{\mathrm{private}}} w_u}

Same calculation as GlobalScore, but the eval run pool is private prompts only; shown independently, excluded from main sort.

8.Secondary metrics

Score acceptance rate=NapprovedNapproved+NrejectedFailure rate=NerrorNdone+NerrorCost efficiency score=GlobalScoreCost index+ε\begin{aligned} \text{Score acceptance rate} &= \frac{N_{\mathrm{approved}}}{N_{\mathrm{approved}} + N_{\mathrm{rejected}}} \\[0.4em] \text{Failure rate} &= \frac{N_{\mathrm{error}}}{N_{\mathrm{done}} + N_{\mathrm{error}}} \\[0.4em] \text{Cost efficiency score} &= \frac{\mathrm{GlobalScore}}{\text{Cost index} + \varepsilon} \end{aligned}

Platform Parameters

Fixed weights and thresholds for the current version; version number increments on change.

Pending sample weight α0.4
Sample confidence threshold N₀50
Quick eval weight1
Batch eval weight1
Minimum valid sample size100
Minimum public eval run coverage50%
Minimum score acceptance rate60%
Maximum score rejection rate40%

Listing & Sorting

Models meeting all conditions receive official ranks; others are marked "Provisional" with metrics still public.

Official listing conditions

  • · Total valid samples ≥ 100
  • · Public eval run coverage ≥ 50%
  • · Score acceptance rate ≥ 60% (confirmed samples only)
  • · Score rejection rate ≤ 40%

Primary sort priority

  1. 1. Global primary score GlobalScore (descending)
  2. 2. Score acceptance rate (descending)
  3. 3. Failure rate (ascending)
  4. 4. Speed index (descending)
  5. 5. Cost index (descending)

Transparency Commitment

Evidence-based rankings, rules updated in sync with the board

We do not adjust rankings using undisclosed rules. Algorithm upgrades increment the version number and update this article; models below listing thresholds are marked "Provisional" with metrics still publicly visible.

  • The judge model is the only scoring source
  • Private prompts excluded from main board
  • Parameters and formulas fully public
View Public Leaderboard
Back to Blog

Algorithm version v0.0.1 · Consistent with public leaderboard rules