API keys and the command-line runner

Quickstart

Requires Python 3.9+ and a C++17 compiler. CMake is optional. Get your key from Settings.

curl -fsSL https://hpcbench.com/runner/install.sh | bash

hpcbench login                 # paste the key; stored 0600 in ~/.config/hpcbench/config.json
hpcbench doctor                # compiler, cores, memory, ISA, fingerprint, API reachability
hpcbench run p1-kernel --label "my laptop" --kind laptop

Authentication

Every endpoint takes Authorization: Bearer <key>. Keys are stored as SHA-256 hashes, so a value is shown exactly once at issue time; regenerating invalidates the old one. There are two kinds: personal (a participant) and cluster (a shared cluster machine, admin-owned).

export HPCBENCH_TOKEN=<your key>
export HPCBENCH_API=https://hpcbench.com/api/public

curl -fsS -H "Authorization: Bearer $HPCBENCH_TOKEN" $HPCBENCH_API/whoami

Endpoints

Errors are JSON with an error string: 401 unknown or revoked key, 403 a key that may not do this, 404 unknown or draft competition, 400 a malformed bundle or request.

What source a key can read

• Open competition, personal key: the baseline plus your own entries. You never receive a competitor’s solution.cpp.
• Open competition, cluster or admin key: all entries, so cluster machines can measure the whole field.
• Closed competition: every key may read every entry. Closing a competition is what makes submissions public.

The manifest always includes the baseline. Your score on a machine is baseline_median / your_median measured in the same session, so a bundle whose baseline is missing or failed its digest is rejected.

Which entries get measured

The manifest, not the client, decides. It selects the baseline, every entry never measured on this machine, every entry changed since it was last measured, and each owner’s three fastest entries. An unchanged entry outside its owner’s top three is skipped with a reason, so a long tail of old attempts does not consume a cluster allocation.

Ingest bundle

POST /ingest. The shape below is illustrative; the runner fills in real values. Per-sweep digests are checked against the reference here — the client’s own verdict is never trusted — and the device is upserted by fingerprint without changing its owner or whether it counts toward the board.

{
  "format": "hpcbench/device-run/1",
  "task": "<competition slug>",
  "owner": "<your handle>",
  "device": {
    "fingerprint_id": "<derived by the runner>",
    "label": "my laptop",
    "kind": "laptop",
    "cpu": "<cpu model>",
    "arch": "x86_64",
    "cores_physical": "<count>",
    "cores_logical": "<count>",
    "memory_gb": "<gb>",
    "os": "<os version>",
    "compiler": "<compiler version>",
    "isa": [
      "<isa flags>"
    ]
  },
  "session": {
    "tier": "device",
    "runs": "<count>",
    "warmup": "<count>",
    "bundle_id": "<uuid>"
  },
  "machine": {
    "caches": "<sysfs cache sizes>",
    "cpu_governor": "<governor>",
    "threads_requested": "<count>",
    "slurm_job_id": "<if any>",
    "perf_available": "<bool>"
  },
  "results": [
    {
      "submission": "baseline",
      "status": "ok",
      "per_sweep": [
        {
          "key": "<sweep value>",
          "median_ms": "<measured>",
          "digest": "<64 lowercase hex chars>",
          "samples_ms": [
            "<per repetition>"
          ]
        }
      ],
      "peak_rss_mb": "<measured>",
      "cpu_efficiency": "<measured>",
      "profile": {
        "build_cmd": "<command>",
        "user_s": "<s>",
        "sys_s": "<s>"
      }
    }
  ]
}

The response lists each stored run with its checked status and recomputed speedup, plus any digest mismatches with the sweep keys that were wrong or missing. Only the most recent successful run per owner, entry and machine counts, and an entry's score is the geometric mean of its speedup across every included machine. An admin can exclude a machine or a single measurement.

Fixed harness

Each competition owns a src/main.cpp harness returned in the task manifest under harness and staged into every entry as src/main.cpp. Participants can read it but never replace it: an entry contributes only its src/solution.cpp. The harness does setup, data loading and result writing outside the timed region and prints the timed call as HPCBENCH_TIME_MS <ms> on its own line. The runner uses that number when present and falls back to process wall clock only when it is absent, so loading and generating inputs never counts toward a score.

Input data

A competition declares where its input comes from: bundled with the repository, an external URL with an optional SHA-256 and a preparation command that converts it into the required format, or a private upload served as a short-lived signed link. The runner verifies the checksum when one is declared and refuses to measure against a file that does not match. Every measurement uses the official input exactly as declared.

CLI reference

hpcbench login                       # store a key for this machine
hpcbench whoami                      # who this key is and what it may read
hpcbench doctor                      # compiler, machine facts, API reachability
hpcbench pull <task> --out DIR        # write manifest + permitted source, measure nothing
hpcbench run <task> [flags]           # build, measure, profile, post

  --label TEXT --kind laptop|desktop|workstation|hpc|vm|browser
  --runs N --warmup N                 # fewer repetitions: setup only, not a result
  --threads N --cxx CC --cxxflags "…"
  --only NAME [NAME …]                # subset of entries (the baseline is always measured)
  --dry-run                           # write the bundle, post nothing
  --workdir DIR                       # where sources, builds and inputs are staged
  --keep-build --keep-input           # skip the automatic cleanup

run is self-contained: it fetches the config, harness and permitted source, downloads and checksum-verifies the input, builds and measures every entry against the baseline, digests each result file, posts the bundle, then deletes every file it created (including on failure). Nothing is left to clean up by hand.

Every sweep point runs under a timeout, and once the baseline is known, an entry that exceeds a small multiple of it is killed and recorded as too_slow instead of burning the allocation. Machine facts that this host does not expose are recorded as null — never estimated.