Proxmox Node Cannot Update Packages via apt-get

Problem

A virtualization host/node running Proxmox (whether clustered or not) will fail to update packages via apt-get and produce the below error messages in the task queue:

Environment

Proxmox Virtualization Environment (confirmed for versions 6.2 and newer) running the free/community editions without and enterprise subscription.

Cause

The cause of this issue results from not having a Proxmox Enterprise subscription/license. These errors are caused by apt-get attempting to procure updates from the subscription enterprise proxmox repository, but because the node or cluster does not have a valid enterprise subscription, the update fails and produces an error.

Solution

Access the shell for the node/virtualization host producing the error and run the following command:

cp /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.backup && echo -e "#$(cat /etc/apt/sources.list.d/pve-enterprise.list)" > /etc/apt/sources.list.d/pve-enterprise.list

This command copies the repository sources list, renames the original as a backup so the system does not use it, then comments out the line of the list for the enterprise repository.

Once this command runs successfully, you should be able to either (a) wait for the scheduled updates to try to run again or (b) manually search for and install updates using apt-get update/upgrade. Either way, no errors should be produced.


Additional Reading

https://dannyda.com/2020/06/19/how-to-fix-proxmox-pve6-1-26-1-7-update-failedfailed-to-fetch-401-unauthorized-task-error-command-apt-get-update-failed-exit-code-100/

How to Revert Node Certificates to Default Configuration in Proxmox VE

Purpose

The purpose of this document is to explain the steps necessary to reset and regenerate the state of the node/host certificates.

Assumptions

This document assumes you have a properly functioning Proxmox cluster with all nodes accessible on the network/via web browser. You’re replaced or modified any of the certificate or key files generated by PVE and now need to revert to the default state.

Step 1 – Cleanup existing certificate files

Delete or move the following files:

  • /etc/pve/pve-root-ca.pem
  • /etc/pve/priv/pve-root-ca.key
  • /etc/pve/nodes/<node>/pve-ssl.pem
  • /etc/pve/nodes/<node>/pve-ssl.key

The latter two need to be repeated for all nodes if you have a cluster.

If you do not repeat the latter two on each host, your issue will return.

Step 2 – Regenerate certificates

Afterwards, run the following command on each node of the cluster to re-generate the certificates and keys:

pvecm updatecerts -f


Additional Information

https://pve.proxmox.com/wiki/HTTPS_Certificate_Configuration_(Version_4.x,_5.0_and_5.1)#Revert_to_default_configuration

Fix “Cannot Start VM or Console Due to TASK ERROR: Can’t Lock File ‘/var/lock/qemu-server/lock-#.conf’ – Got Timeout”

Purpose

This document will delineate the steps necessary to correct an issue where you cannot start, stop, or open the console of a VM due to the following error:

TASK ERROR: can't lock file '/var/lock/qemu-server/lock-#.conf' - got timeout

Assumptions

This document assumes you have administrative access to the shell of the problem node, and that the problem node is available on the network.

Cause

This is often caused by a console session not closing out correctly or a hangup when moving between tabs of console sessions.

Solution

qm unlock ###

where ### is the unique VM ID of the virtual machine or container. If this does not work alone, safely reboot the host following use of this command.

Installing Proxmox Virtual Environment from a USB Flash Drive


Purpose

This document delineates the steps necessary to perform a clean installation of Proxmox Virtual Environment on a host device using a USB drive as the installation media.

Assumptions

This document assumes the device POSTs successfully with no errors and that the device meets the minimum requirements for Proxmox VE.

Step 1 – Creating Installation Media Using Etcher

To create the installation media, we will be assuming the USB drive will be flashed from a Microsoft Windows workstation or a Linux workstation with a distribution based on Debian/Ubuntu.

Download and install Balena Etcher from this link.

Download the desired version of the Proxmox VE installation ISO from this link.

Connect the USB drive to your workstation.

Open Etcher.

Select the USB drive from the drive list, then select the ISO to flash to the device.

Browse to and select the Proxmox VE installation ISO.

Click Flash! When done, you are ready to plug the USB drive into your intended host.

Step 2 – Booting to the USB Drive

Connect the USB drive to your intended host while it is powered off.

Immediately after power on and POST, we will need to boot from the USB drive. Every equipment manufacturer does this differnetly. For most Dell desktop systems, F12 will be the key to strike. For most HP desktop systems, F8 will be the key to strike. For most Lenovo desktop systems, Esc is the key to strike. Server hardware will vary from vendor to vendor as well.

Power on your intended host and use the appropriate keystroke to bring up the one-time boot menu, then select the USB drive.

Step 3 – Installation of Proxmox VE

Once the Proxmox GRUB installer loads, select Installation (Debug) and strike Enter on the keyboard. A black console screen will appear and the installer will check for attached installation media.

When the console gives the option to exit or strike Ctrl+D to continue installation, strike Ctrl+D.

Do note that you must use the debug install option first otherwise the installation will fail.

You may have to strike Ctrl+D a second time.

The screen will again display the Proxmox VE installer and move you through a series of prompts. Follow the prompts to assign timezone, select the hard disk device to which you’d like to install the system, configure host network settings, root user, etc.

When you reach the end of the installation process, you will be instructed to reboot the host and point another workstation’s web browser to the host URL, which will be the IPv4 address of the host on port 8006 using HTTPS. (Example: https://192.168.0.100:8006).

Step 4 – First Run

Once the host has rebooted, your web browser should be able to reach the web portal for your Proxmox host. Log in using the root credential you created during the installation process. The authentication type should be Linux PAM.

You are now able to configure your virtual environment with storage and start spinning up virtual machines and containers.


Additional Information

Additional information on installing the Proxmox Virtual Environment can be found here:

https://www.proxmox.com/en/

How to Fix “Cluster Not Ready – No Quorum?” in Proxmox

Purpose

This document will delineate the steps necessary to work around a situation where your Proxmox cluster will not allow you to interact with it due to lack of quorum between the hosts.

Assumptions

This document assumes that the nodes in the cluster are all online with proper certificates, network connectivity, and that the administrator intents to properly resolve the quorum issue following these steps.

This guide is not intended to be a permanent solution, but rather a work-around.

Cause

This occurs when the cluster quorum is not configured correctly –or– when a cluster member is offline. The intended function of the cluster is to not work if the quorum does not receive the adequate number of votes from the cluster to begin. You can manually set the number of expected quorum.

Step 1 – Override expected number of quorum votes

You can manually set the number of expected quorum votes by running this command:

pvecm expected #

where # is the number of votes you wish to make the quorum use to determine if the cluster is quorate.

Only manually override this if the node that is offline is about to be deleted from the cluster.

Following use of this command, the cluster should achieve quorum and allow you to interact with it. It is recommended that you remove the problem node from the cluster following the procedure.


Additional Information

https://pve.proxmox.com/wiki/Cluster_Manager

https://forum.proxmox.com/threads/cluster-and-quorum.7786/

https://www.jm.technology/post/proxmox_quorum_april_2019/