Last updated: Fri Jan 03 2025

Powering down your idle Nvidia dGPU in Linux

Warning: this doesn't work as well as I initially thought it did. See the Known Issues

Laptop power drain is a complex topic, but in laptops with two GPUs (e.g. the typical gaming laptop), poor GPU power configuration can cause significant power drain.

In modern laptops with two GPUs, typically one GPU is fairly low performance and integrated into the CPU (iGPU), and the other is dedicated and high performance but also very power hungry (dGPU). It's theoretically possible and desirable to run the desktop and most applications on the iGPU and run only graphics intense applications (like games) on the dGPU. dGPUs will drain a laptop battery quickly even when they're not doing much, but when configured correctly, the dGPU will fully power down when not in use.

I've had trouble in the past getting this to work and resorted to fully disabling the dGPU, but I've had another go and as of December 2024 this is possible and easy to set up.

With this set up, I have access to the dGPU but it spends most of the time powered down. The system as a whole uses around 10-12W with a few programs open and the dGPU powered down, which equates to 5-6 hours of battery life, which is pretty much the same as it was when the dGPU was fully disabled. Sitting on an empty KDE desktop after boot the power usage is around 7-8W.

Hardware and software versions

My system is as follows:

  • Laptop: Lenovo Legion Slim 5 2023 16APH8
  • CPU/iGPU: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
  • dGPU: Nvidia Geforce 4070 dGPU
  • BIOS version: M3CN44WW (up to date as of December 2024)

Software:

  • Fedora 41
  • Kernel: 6.12.6-200
  • Nvidia driver version: 565.77 (from rpmfusion),

Back in April 2024 getting this set up was beyond my abilities. The changes since then are Fedora 40 -> 41, a newer Kernel, newer Nvidia drivers and a newer BIOS. I can't say for sure what made the difference, but I suspect there was a bug in the Nvidia drivers introduced from version 525 which has been resolved.

Known Issues

On my system, this set up works from a cold boot, but it rarely continues to work when the system is on for a while and then rebooted. In this case, the dGPU stays in D0 (powered) most of the time, occasionally briefly switching to D3cold (unpowered) and then immediately returning to D0. However, it does seem to work after shutting down and leaving the laptop powered off for a while. It also begins working if the laptop goes to sleep for a while then wakes up.

It's as if the GPU being warm at boot/wakeup will prevent the power management from initialising 🤷.

I have not found anything relevant in dmesg or the journalctl logs. I think it's a bug in the driver.

Config

Install the Nvidia drivers, then create the following files with contents:

# /etc/modprobe.d/nvidia.conf
# ===========================

options nvidia_drm modeset=1 fbdev=1
options nvidia NVreg_DynamicPowerManagementVideoMemoryThreshold=0
# /etc/udev/rules.d/nvidia.conf
# ==============================

# Remove NVIDIA USB xHCI Host Controller devices, if present
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c0330", ATTR{remove}="1"

# Remove NVIDIA USB Type-C UCSI devices, if present
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c8000", ATTR{remove}="1"

And reboot. This should be all you need for the dGPU to go into D3cold (powered down) when idle. Use the commands in the next section to confirm what the dGPU is doing.

Debugging and useful commands

To see current power draw:

$ awk '{print $1*10^-6 " W"}' /sys/class/power_supply/BAT*/power_now

View current dGPU power state:

$ cat /sys/class/drm/card0/device/power_state

(D0 is powered up, D3cold is fully powered down. D3hot also exists, which means the device is off but still has power connected)

You will have both /sys/class/drm/card0/ and /sys/class/drm/card1/. To check which one is which, use cat /sys/class/drm/card0/device/vendor. Nvidia: 0x10de AMD: 0x1002

View GPU power info:

$ cat /proc/driver/nvidia/gpus/0000:01:00.0/power
Runtime D3 status:          Enabled (fine-grained)
Video Memory:               Active

GPU Hardware Support:
 Video Memory Self Refresh: Supported
 Video Memory Off:          Supported

S0ix Power Management:
 Platform Support:          Supported
 Status:                    Disabled

If you find that it refuses to go into D3cold and everything looks like it should work (i.e. you've set the config and the previous command reports runtime D3 status is enabled), use nvidia-smi to see if there are any processes running on the gpu. Note that nvidia-smi will briefly put the GPU into D0.

Selecting the right GPU

So now that the dGPU is visible to the OS but powered down and not used by default, how do you actually launch a program on it?

Steam

Steam games will generally offload to the dGPU, as long as you launch Steam from the menu or its icon (steam.desktop). This is because the steam.desktop file contains some lines specifying its GPU preference:

$ cat ~/.local/share/applications/steam.desktop | grep gpu -i
PrefersNonDefaultGPU=true
X-KDE-RunOnDiscreteGpu=true

This does mean that the main Steam window renders on the dGPU, so I recommend minimising to the tray when you are not using it as otherwise it will keep the dGPU powered up.

Offload to the dGPU, with prime-offload

Adapted from RPM fusion

Create a prime-offload script, and place it in ~/.local/bin/prime-offload and give it executable permissions (chmod +x ~/local/bin/prime-offload)

#!/bin/bash

me=$(basename "$0")
if [ "$#" -eq 1 ]; then
    echo "Usage: $me COMMAND"
    echo
    echo "e.g. $me glxinfo | grep vendor"
fi

__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia "$@"

Then run applications with:

$ prime-offload glxgears
$ prime-offload firefox

Running glxinfo with an without this script should show different results:

$ glxinfo | grep "OpenGL vendor"
OpenGL vendor string: AMD
$ prime-offload glxinfo | grep "OpenGL vendor" 
OpenGL vendor string: NVIDIA Corporation

Forcing use of the iGPU, with prime-onload

It's worth noting that modern AMD iGPUs can be pretty capable. The AMD Radeon 780M in my laptop is roughly equivalent to a Steam Deck's GPU, or a GeForce 1050 Ti. It won't run Cyberpunk well, but it will handle many less intensive games just fine and more efficiently than the RTX 4070.

So, for the opposite to prime-offload, to force applications to run on the iGPU (in the case of some older games which will choose the dGPU even though they don't need it), I have also created a prime-onload script.

Use

$ MESA_VK_DEVICE_SELECT=list vulkaninfo
selectable devices:
  GPU 0: 10de:2860 "NVIDIA GeForce RTX 4070 Laptop GPU" discrete GPU 0000:01:00.0
  GPU 1: 1002:15bf "AMD Radeon 780M (RADV GFX1103_R1)" integrated GPU 0000:06:00.0
  GPU 2: 10005:0 "llvmpipe (LLVM 19.1.0, 256 bits)" CPU 0000:00:00.0

to view the names and IDs of your devices, and substitute them in below.

You can also add DXVK_HUD=1 to the parameters below to get an overlay showing which GPU is in use.

#!/bin/bash

me=$(basename "$0")
if [ "$#" -eq 0 ]; then
    echo "Usage: $me COMMAND"
    echo
    echo "e.g. $me glxinfo | grep vendor"
fi

# Use
# MESA_VK_DEVICE_SELECT=list vulkaninfo
# to get a list of devices, and their name and ID

__NV_PRIME_RENDER_OFFLOAD=0 DRI_PRIME=1002:15bf MESA_VK_DEVICE_SELECT=1002:15bf DXVK_FILTER_DEVICE_NAME="AMD Radeon 780M (RADV GFX1103_R1)" "$@"

In the case of Steam games, you can either close and launch Steam with $ prime-onload steam to launch the game, or set these options as the launch options of the game:

Steam launch options

(Civ IV, V and even VI being examples of games that do not benefit at all from a GeForce RTX)

It should read:

__NV_PRIME_RENDER_OFFLOAD=0 DRI_PRIME=1002:15bf MESA_VK_DEVICE_SELECT=1002:15bf DXVK_FILTER_DEVICE_NAME="AMD Radeon 780M (RADV GFX1103_R1)" %command%

with the IDs and device name substituted for your values from $ MESA_VK_DEVICE_SELECT=list vulkaninfo.