Gathering the first fleet snapshot…
{% else %} {# #168: the summary lives inside the polled fragment so its totals move with the table. Allocation (what running jobs declared) and measurement (fresh watchdog samples) are separate tiles and never added together. #} {% for item in coverage.unlisted %} {% endfor %} {% for item in coverage.duplicates %} {% endfor %} {% for item in coverage.usage_errors %} {% endfor %}
Totals cover the {{ filtered_count }} job{{ "" if filtered_count == 1 else "s" }} matching the filters.
Allocated is what running jobs declared; used is measured, from
{{ summary.measured_jobs }} of {{ summary.counts.running }} running job{{ "" if summary.counts.running == 1 else "s" }}.
{% if summary.partial %}* Partial:
{% for item in summary.unmeasured %}{{ item.count }} {{ item.label }} ({{ item.detail }}){% if not loop.last %}; {% endif %}{% endfor %}.
{% endif %}
GPUs allocated is what scheduler jobs requested with --gpus. A GPU job's own row adds its measured use, from nvidia-smi on its batch node: the newest sample while it runs, its summary once finished. It is device-level and is not added into these totals.
Trends cover the last {{ trend.points }} sweep{{ "" if trend.points == 1 else "s" }} of this console since {{ trend.since[:19] | replace("T", " ") }} UTC (kept in its memory; a restart starts afresh). Used is measured from fresh samples; a dot under a point marks a partial sweep (an unmeasured running job or an unlisted host), so a dip there is missing data, not idle work.
{% endif %} {% if host_load %}| host | running | cpus allocated / budget | cores used | rss MB / total (free) | ||||
|---|---|---|---|---|---|---|---|---|
| {{ h.host }} | {% if not h.listed %}not listed in this snapshot: unknown, not idle | {% else %}{{ h.running }} | {{ h.allocated_cpus }} / {{ h.max_cpus if h.max_cpus is not none else "?" }} | {% if h.lane %}not sampled {% elif h.measured_cores is not none %}{{ h.measured_cores }}{% if h.partial %}* ({{ h.measured_jobs }} of {{ h.running }}){% endif %} {% elif h.running %}no fresh sample{% else %}-{% endif %} | {% if h.measured_rss_mb is not none %}{{ "%.0f" | format(h.measured_rss_mb) }}{% elif h.lane %}not sampled{% else %}-{% endif %} {% if h.mem_total_mb %} / {{ h.mem_total_mb }}{% if h.mem_available_mb is not none %} ({{ h.mem_available_mb }} free){% endif %}{% endif %} | {% endif %}|||
| id{{ sort_label('id') }} | host{{ sort_label('host') }} | state{{ sort_label('state') }} | sched{{ sort_label('sched') }} | runtime{{ sort_label('runtime') }} | wait{{ sort_label('wait') }} | cpus{{ sort_label('cpus') }} | cpu used{{ sort_label('cpu_used') }} | mem MB{{ sort_label('mem') }} | rss MB{{ sort_label('rss') }} | peak MB{{ sort_label('rss_peak') }} | nodes{{ sort_label('nodes') }} | software{{ sort_label('software') }} | sample{{ sort_label('sample') }} | submitted (UTC){{ sort_label('submitted') }} | submitter{{ sort_label('submitter') }} | name / tags{{ sort_label('name') }} | workdir{{ sort_label('cwd') }} | diagnosis{{ sort_label('diagnosis') }} | command{{ sort_label('command') }} |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| {{ job.id }} | {{ job.host }} | {{ job.effective_state }} | {{ job.sched or "-" }} | {{ format_duration(rt.seconds) }}{% if rt.limit_seconds %} / {{ format_duration(rt.limit_seconds) }}{% endif %} | {% if rt.waiting %}waiting {% endif %}{{ format_duration(rt.wait_seconds) }} | {{ job.cpus if job.cpus else "-" }}{% if job.gpus %} +{{ job.gpus }} gpu{% set gpu = job.calc.gpu %}{% if gpu %} {% if gpu.status == 'unmeasured' %}gpu not measured{% else %}{{ "%g" | format(gpu.utilization_percent) ~ "%" if gpu.utilization_percent is not none else "?%" }} · {{ "%.1f" | format(gpu.memory_mb / 1024) if gpu.memory_mb is not none else "?" }}{% if gpu.memory_total_mb is not none %}/{{ "%.0f" | format(gpu.memory_total_mb / 1024) }}{% endif %} GiB{% if gpu.status == 'final' %} peak{% else %} · {{ gpu.label }}{% endif %}{% endif %}{% endif %}{% endif %} |
{{ "%.1f" | format(obs.cpu_cores) if obs.cpu_cores is not none else "-" }}{% if obs.status == 'final' and obs.cpu_cores is not none %} avg{% endif %} | {{ job.mem_mb if job.mem_mb else "-" }} | {{ "%.0f" | format(obs.rss_mb) if obs.rss_mb is not none else "-" }} | {{ "%.0f" | format(obs.rss_peak_mb) if obs.rss_peak_mb is not none else "-" }}{% if obs.rss_peak_scope == 'largest task' %} task{% elif obs.status == 'final' and obs.rss_peak_mb is not none %} process{% endif %} | {{ (loc.label | truncate(32, true)) if loc.label else "-" }} {% if loc.driver %}via {{ loc.driver }}{% endif %} | {{ job.calc.software or "-" }} | {{ obs.label or "-" }} | {{ job.submitted_at[:19] | replace("T", " ") if job.submitted_at else "-" }} | {{ job.submitter or "-" }} | {{ job.name or "-" }} {% if job.tags %}[{{ job.tags | join(", ") }}]{% endif %} | {# Truncated from the LEFT: the distinguishing part of a workdir is its tail (which checkout, which job id), not the shared prefix. #}{{ ("…" + job.cwd[-44:]) if job.cwd | length > 45 else (job.cwd or "-") }} |
{{ job.diagnosis or "-" }} | {{ job.command | truncate(80, true) }} |
No jobs match.
{% endif %} {# #168: an unlisted host's last successful listing. These rows are not current, so they are in no count or total above; each is shown as of the listing (runtime stops there), without measured usage. #} {% if last_known %}What each host showed at its last successful listing. These jobs may have finished, failed or been requeued since; none of them is counted above. Runtime is as of that listing.
| id | host | listed (UTC) | state then | sched then | runtime then | cpus | mem MB | nodes | software | submitted (UTC) | name / tags | command |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| {{ job.id }} | {{ job.host }} | {{ group.observed_at[:19] | replace("T", " ") }} ({{ group.age }} ago) | {{ job.effective_state }} | {{ job.sched or "-" }} | {{ format_duration(rt.seconds) }}{% if rt.limit_seconds %} / {{ format_duration(rt.limit_seconds) }}{% endif %} | {{ job.cpus if job.cpus else "-" }}{% if job.gpus %} +{{ job.gpus }} gpu{% set gpu = job.calc.gpu %}{% if gpu %} {% if gpu.status == 'unmeasured' %}gpu not measured{% else %}{{ "%g" | format(gpu.utilization_percent) ~ "%" if gpu.utilization_percent is not none else "?%" }} · {{ "%.1f" | format(gpu.memory_mb / 1024) if gpu.memory_mb is not none else "?" }}{% if gpu.memory_total_mb is not none %}/{{ "%.0f" | format(gpu.memory_total_mb / 1024) }}{% endif %} GiB{% if gpu.status == 'final' %} peak{% else %} · {{ gpu.label }}{% endif %}{% endif %}{% endif %}{% endif %} |
{{ job.mem_mb if job.mem_mb else "-" }} | {{ (loc.label | truncate(32, true)) if loc.label else "-" }}{% if loc.driver %} via {{ loc.driver }}{% endif %} | {{ job.calc.software or "-" }} | {{ job.submitted_at[:19] | replace("T", " ") if job.submitted_at else "-" }} | {{ job.name or "-" }}{% if job.tags %} [{{ job.tags | join(", ") }}]{% endif %} | {{ job.command | truncate(80, true) }} |