Worker

Power usage

The console shows how much power each worker's machine is drawing: a live figure on the worker's page, and a chart covering the last hour, day, week or month. Use it to report the energy your AI workloads use, to see when a GPU is running flat out, or to plan power and cooling for a new machine.

The live figure is on every plan. The chart, with the energy used, average and peak for each period, is part of the Pro and Enterprise plans. Enterprise customers who need energy reports, for example for sustainability reporting, can ask us for custom energy reporting.

Where to find it

  • Workers → your worker shows a Power tile next to the worker's other vitals. It's the whole machine's draw right now, refreshed every few seconds. On a machine with NVIDIA GPUs, the bar underneath shows how hard the GPUs are working against their power limits.
  • On Pro and Enterprise, the Power usage card further down the same page charts that draw over time. Pick 1h, 24h, 7d or 30d at the top of the card. On the Free plan, the card offers an upgrade instead. Your worker reports its readings on every plan, so after an upgrade the chart already covers the time before it.
  • The Workers list shows each worker's live draw, such as 512 W, beside its status.

What's measured

The worker reads every power sensor it can find on its machine. What that covers depends on the hardware:

PartHow it's read
NVIDIA GPUsEach GPU's power is read from the NVIDIA driver. Measured.
CPUOn Linux, read from the processor's own energy counter where the operating system allows it, or from a power sensor the machine publishes for its CPU. Measured. Linux only lets an administrator read that counter and hides it from containers, so a worker installed as a service or run in a container usually can't read it, and it is only used when the worker has the whole machine. On Windows, or where there is nothing to read, the CPU's share is estimated from how busy it is and how many cores the worker can use. In a container or on a rented pod, that is only the cores the worker is given, not the whole host's.
Mac with Apple siliconThe whole machine is read from the Mac's own power sensor. Measured.
Linux machines with their own power sensorsSome servers and single-board machines publish power readings through Linux's hardware-monitoring interface. Where one covers the whole machine, the worker reads it and the machine is Measured, with no estimate. Where a sensor covers the processor chip as a whole (a system-on-chip, or SoC) and not the CPU alone, the worker reports that total as SoC and doesn't split it. This needs no setup: if the sensor is there and its label says what it measures, it is used. The worker only uses these sensors when it has the whole machine (see below).
Everything else in a PC or serverThe motherboard, memory, fans, drives and power-supply losses usually have no sensor, so their share is estimated.

If a machine has none of these sensors, the worker doesn't report power at all rather than guess, and the console says Power isn't measured on this machine.

Measured or Estimated

Every power figure carries a label:

  • Measured: the whole machine's draw came straight from a sensor, as on a Mac or a Linux machine with its own power sensor.
  • Estimated: the GPUs (and the CPU, where it can be read) were measured, and the rest of the machine was estimated from its size and added on. Hover the label to see exactly which parts were measured.

On a virtual machine, such as a cloud GPU instance, the worker can see its GPUs but not the host around them, so the estimate adds a fixed allowance for each GPU instead. Treat estimated figures as a good guide to how much your machine uses and how that changes over time, not as a meter reading.

A worker that is given only part of a machine, such as a rented pod or a container limited to some of the machine's processors or GPUs, doesn't use the machine's own power sensors either: they would count everything else running on it. It reports its GPUs as measured and estimates the CPU and the rest of the system for the processors it was given.

Reading the chart

The chart stacks the machine's parts on top of each other, so the top edge is the whole machine:

  • one band for each GPU (GPU 0, GPU 1, …);
  • CPU (or SoC, on a machine that reports its processor chip as one total);
  • Rest of system (estimated): everything else, drawn with a dashed outline so it's easy to tell apart from what was measured.

A machine measured as a whole shows its measured parts with a solid Rest of system band above them. A Mac, which has no separate parts to show, is a single Machine band. Hover anywhere on the chart to see each part and the machine's total at that moment. A gap in the chart means there were no readings then, usually because the worker was offline.

Each point is an average over a short window: one minute on the 1h view, five minutes on 24h, and one hour on 7d and 30d. Minute-by-minute readings are kept for 14 days and hourly figures after that, so the 7-day and 30-day views stay available. A worker's history stays when it goes offline or is removed.

Under the chart:

FigureMeaning
EnergyTotal energy the machine used over the range, in kWh.
AverageAverage draw over the time the worker was reporting.
Peak (momentary)The highest reading in the range. The worker reads its sensors every few seconds (every second on most machines), so this catches short bursts that the averaged chart smooths out. That is why it is usually higher than the top of the chart.
CoverageHow much of the range has readings. Below 100% means the worker was offline for some of it, and the energy total only covers the time it was reporting.
SourcesWhere the readings came from, such as the NVIDIA driver or an estimate. A reading from one of the machine's own power sensors shows that sensor's name.

When there's no chart

The console saysWhat to do
Update this worker to see power.The worker is older than power reporting. Update it to the latest release.
Power reporting is turned off on this worker.Someone set PENDRA_DISABLE_POWER_TELEMETRY on the machine. See below to turn it back on.
Power isn't measured on this machine.The machine has no sensor the worker can read (see What's measured).
Collecting — the first reading appears within a minute.Nothing to do. The worker has just started reporting.

What leaves your machine

The worker sends Pendra one-minute averages for each part it reads (average, lowest and highest draw, plus GPU load and temperature), and a live reading every few seconds for the tile. Nothing else is collected for this: no per-request detail, and nothing about your prompts or responses.

Turn power reporting off

Power reporting is on by default. To turn it off on a worker, set PENDRA_DISABLE_POWER_TELEMETRY to 1 (or true / yes) where the worker runs, then restart it:

shell
# On the worker's machine, then restart the worker
export PENDRA_DISABLE_POWER_TELEMETRY=1
docker
docker run -e PENDRA_DISABLE_POWER_TELEMETRY=1 ...

The worker then stops reading its sensors and sends nothing. The console shows Power reporting is turned off on this worker, and the readings it sent before stay in its history. Remove the variable and restart to turn it back on. See Configuration for every worker setting.

Reading power from the API

The same history is available to your own code at GET /api/v1/workers/{worker_id}/power, with a key that has the Usage & workers permission, on the Pro and Enterprise plans. See Usage & workers.