ProxmoxKVM WHMCS module Description Changelog Installation and configuration guide Basic concepts and requirements  Order now | Dowload | FAQ Requirements for the successfull operation of the module Minimal WHMCS version 8+ Minimal Proxmox server or cluster version 7+ Continuous and stable network availability between systems In the case of using static IP addresses, it is necessary to have the required number of free IP addresses that will be reserved for use in virtual machine services. If you are using a DHCP server in your virtual machine network, you will additional need to configure it properly. If you use VLAN in your network, then you need to correctly configure your internal network so that the VLAN is available on all nodes of the cluster. In order for the noVNC console to work correctly, it is necessary to configure the VNC Web proxy on a separate installation (for example, a virtual machine) that will have access to the Internet and to the secure network of the Proxmox cluster / server. You need to set up a DNS API proxy. To synchronize forward and reverse DNS records. In order to manage pools of IPv4 and IPv6 IP addresses, DNS zonesyou need to install the PUQ Customization addon module (free)https://doc.puq.info/books/puq-customization-whmcs-addonhttps://download.puqcloud.com/WHMCS/addons/PUQ-Customization/https://puqcloud.com/whmcs-addon-puq-customization.php If the module is set to use the DHCP server, the module does not manage the allocation of IP addresses and firewall rules, but only the throughput and setting the appropriate bridge and vlan settings on the network card In the case of using one Proxmox server, a Directory or NFS data store is required. In the case of using a Proxmox cluster, the cluster must have a shared network storage. On one network storage, there should be both templates of virtual machines and disks of virtual machines. A configuration is allowed when the virtual machine template has 2 disks that can be stored on different network storages. For example, the system disk of the virtual machine is stored on the fast storage, and an additional disk is stored on the slow storage, designed for backups inside the virtual machine. For ISO images, you can use another network storage For backups to work, network storage is also required. You can use Proxmox Backup and Your server must still be available to all nodes in the cluster. The configuration of datastores intended for backups must be configured in advance so that they do not rotate backup copies. Or the rotation has been configured so that it does not interfere with the number of spare copies in the client's package. If you want to use firewall rules that will protect against IP address spoofing in virtual machines, You need in advance to properly configure the firewall on your server/cluster. The policy for incoming and outgoing traffic must be "DENY". The module adds permissive firewall rules on a virtual machine with the IP address of this machine. The logic of the module. Creating a new Virtual Machine. After the client ordered and paid for the virtual machine services An available IP address is selected from the list in the server settings. !!! If the service is in the Terminated state, its assigned IP address will be taken into account when creating new machines and can be used for a new machine.!!! The available virtual machine ID is selected in such a way that it is unique in the WHMCS system and on the Proxmox server. The fields with the domain name and the name of the virtual machine are generated and filled. The procedure uses the data supplied in the virtual machine package. Starts the process of cloning a virtual machine from the desired virtual machine template. Notifications are sent to the client that the virtual machine is in the process of being created (Welcome email) Internal cron process monitors and cares about proper installation procedure. Each time cron is executed, the module tries to continue setting up the virtual machine if the cloning process is completed, virtual machine configuration steps in crontab job: Write the MAC address of the network card VM in the WHMCS database Set DNS Records Set CPU and RAM Set system disk size Set system disk bandwidth Created additional disk if needed Set additional disk size if needed Set additional disk bandwidth if needed Set network bridge and VLAN and bandwidth Set Firewall rules Configure cloud-nit (user, password, network configuration) Start VM Notify the client by e-mail that the virtual machine has been created, and the access parameters IP, user, password An example of output from crontab log upon successful creation of a virtual machine =========================================== VM_id: 2001 Service_id: 4785 User_id: 1 VMSetDedicatedIp: The local status should be creation VMClone: The local status should be set_ip VMSetCpuRam: success VMSetSystemDiskSize: success VMSetSystemDiskBandwidth: success VMSetCreatedAdditionalDisk: success VMSetAdditionalDiskSize: success VMSetAdditionalDiskBandwidth: success VMSetNetwork: success VMSetFirewall: success VMSetCloudinit: success VMStart: success Remote_status: running Local_status: ready ServiceSendEmailVMReady: OK Reinstalling the Virtual Machine. It is possible to reinstall VM from WHMCS panel. The reinstallation process removes the virtual machine and recreate it using the parameters and characteristics and the actual data of the package. After the reinstallation option has been chosen and run the virtual machine is removed. Backups are not deleted. After removal, the process of cloning the virtual machine from the selected virtual machine template is started. Each time cron is executed, the module tries to continue setting up the virtual machine if the cloning process is completed, virtual machine configuration steps in crontab job: Set CPU and RAM Delete DNS Records Set DNS Records Set system disk size Set system disk bandwidth Created additional disk if needed Set additional disk size if needed Set additional disk bandwidth if needed Set network Bridge and VLAN and bandwidth and MAC address Set Firewall rules Configure cloud-nit (user, password, network configuration) Start VM Notify the client by e-mail that the virtual machine has been ready, and provide access parameters IP, user, password Snapshots The client has the ability to create/delete/restore snapshots of his virtual machine. The number of snapshots is limited in the package configuration. Snapshot lifetime is configured in the package configuration (is it in the range of 1-10 days maximum) Snapshots are automatically deleted when their lifetime expires (using crontab jobs) Backups The client has direct ability to create/delete/restore backups of his virtual machine. The number of backups is limited in the package configuration. In the case of a schedule configuration for creating automatic backups, the client has the opportunity to select only the day on which the backup should be created. Backup times are generated automatically and randomly each time an automatic backup task is saved. How automatic backups work. during cron execution, it checks which virtual machine has a schedule for today, and if the execution time of the scheduled copy is younger than the moment at which the crona job is running checks if there is no today's backup then checks if there is a free slot for a backup and if not, it deletes the oldest backup creates a backup Backup recovery Before restoring a backup, the virtual machine must be in a powered off state. After a successful backup restore, the module does the following processes with the virtual machine. Set CPU and RAM if needed Set system disk size if needed Set system disk bandwidth if needed Created additional disk if needed Set additional disk size if needed Set additional disk bandwidth if needed Set network bridge and VLAN and bandwidth and MAC address if needed Start VM Notify the client by e-mail that the virtual machine restoring the  backup ready If for some reason it was not possible to restore the virtual machine from the backup, then the client is given the opportunity to try to restore the backup again or reinstall the virtual machine During the creation and restoration of a backup, all operations for managing the virtual machine are suspended Reset password The password reset procedure will work if the packages that are responsible for the operation of cloud-init have not been removed from the virtual machine Before reset password, the virtual machine must be in a powered off state. Processes that occur during the password change procedure a new random password is generated and recorded in the client's service settings Configure cloud-nit (user, password, network configuration) Start VM Mounting ISO image ISO images are configured and stored in standard Proxmox way. You need to prepare storage (shared storage in case of Proxmox cluster) and upload ISO images to it in advance. To organize and facilitate image selection, the module uses a display method that follows the file name formatting convention. Module displays ISO images names taken from Proxmox in WHMCS panel. One should pay attention to the correct naming of the image files. ISO images are cataloged and sorted by the symbol "-" in the name of the ISO image. For the name of the directory, the first occurrence is matched up to the symbol "-" If the name of the ISO image does not have a symbol "-" then such an image will be cataloged in the OTHER directory. 1. WHMCS setup(install/update)  Order now | Dowload | FAQ To install and update a module, you must perform one and the same action. 1. Download the latest version of the module. PHP 8.1 wget http://download.puqcloud.com/WHMCS/servers/PUQ_WHMCS-Proxmox-KVM/PUQ_WHMCS-Proxmox-KVM-latest.zip PHP 7.4 wget http://download.puqcloud.com/WHMCS/servers/PUQ_WHMCS-Proxmox-KVM/php74/PUQ_WHMCS-Proxmox-KVM-latest.zip All versions are available via link: http://download.puqcloud.com/WHMCS/servers/PUQ_WHMCS-Proxmox-KVM/ 2. Unzip the archive with the module. unzip PUQ_WHMCS-Proxmox-KVM-latest.zip 3. Copy and Replace "puqProxmoxKVM" from "PUQ_WHMCS-Proxmox-KVM" to "WHMCS_WEB_DIR/modules/servers/" 2. Virtual machine templates  Order now | Dowload | FAQ Preface Since the primary purpose of the module is to create and manage virtual KVM machines, the machine templates must be properly prepared and available in the Proxmox panel (standalone server or cluster). You can try to use available templates from the Internet (which support clud-init standard) and they should work fine, but we suggest initially using templates that are created according to our standards. We offer a basic set of images for download. In order for the automatic installation of the virtual machine to work correctly, You need to prepare virtual machine templates for all operating systems that you want to provide for your customers to choose from. The physical state of the virtual machine template All parameters of the virtual machine must be less than the smallest package you offer to clients. (cpu, RAM, disk size) If you provide a virtual machine with two disks and host these disks on different storages, you need to create these boards in the template of the virtual machine on the necessary data storage, otherwise the additional disk will be created automatically on the same storage as the main disk. You need to add a cloud-init disk in order to automatically configure the virtual machine after cloning, your client parameters (network settings, authorization data) Virtual machine template software Install the desired operating system on the template virtual machine During installation, partitions of the virtual machine disk must be divided so that the root partition is the last in the lists. This is necessary in order for the automatic change of the system partition to work at the moment when the disk of the virtual machine is enlarged. After installing the operating system, you need to install the cloud-init software suite ​ enable root ssh nano /etc/ssh/sshd_config PermitRootLogin yes /etc/init.d/ssh restart apt-get update apt-get install network-manager resolvconf -y apt-get install cloud-initramfs-growroot cloud-init cloud-utils -y systemctl enable cloud-init-local.service systemctl enable cloud-init.service systemctl enable cloud-config.service systemctl enable cloud-final.service dpkg -l | grep cloud-* Delete all from /etc/NetworkManager/dispatcher.d except cloud-init-hook-network-manager deluser --remove-home debian rm -R /home/debian/ cat /dev/null > ~/.bash_history && history -c && exit We have prepared templates for popular operating systems. Templates are in the form of a virtual machine backup.In order to use it, you need to download the archive. Then copy it to the Proxmox server and restore the backup. Operating system configuration: hdd 5 GB (virtio) disabled swap minimal server installation allow the root user to connect via ssh root password: puqcloud installed cloud-initramfs-growroot cloud-init cloud-utils localization and time zone Europe/Warsaw language English FAQ If you need any other operating system or certain parameters, you can discuss it on our faq https://faq.puqcloud.com/ Please note that we provide operating system templates only to demonstrate the functionality of our module.YOUR USE OF THESE OPERATING SYSTEMS IS AT YOUR OWN RISK. WE DO NOT GUARANTEE CORRECT OPERATION AND SAFETY. WE DO NOT RECOMMEND TO USE THEM AS OPERATING SYSTEM TEMPLATES FOR YOUR CLIENTS. Download prebuild Proxmox OS templates You can download the templates from the links below. https://files.puqcloud.com/ Debian Debian-10 Debian-11 Debian-12 Ubuntu Ubuntu-18 Ubuntu-20 Ubuntu-22 CentOS CentOS-7 CentOS-8 CentOS-9 Proxmox PBS-2.2 Official cloud images with root access. Debian Debian-10 Debian-11 Debian-12 Debian-13 Ubuntu Ubuntu-20.04 Ubuntu-22.04 Ubuntu-23.10 CentOS CentOS-7 CentOS-9 3. Install VNCproxy and noVNC  Order now | Dowload | FAQ Preface The module supports the ability to connect and use the console to manage a specific KVM virtual machine. To connect to the virtual machine console, we will use third-party software. noVNC - the open source VNC client - noVNC is both a VNC client JavaScript library as well as an application built on top of that library. noVNC runs well in any modern browser including mobile browsers (iOS and Android). Project site: https://novnc.com Project github: https://github.com/novnc/noVNC As we only use external project we dont take any responsibility for data leak, hacks etc. We use golang to build our products. We have used the following libraries https://github.com/evangwt/go-vncproxy (MIT License) https://github.com/gin-gonic/gin (MIT License) https://pkg.go.dev/golang.org/x/net/websocket (BSD License) If you have any difficulties, you can use our public vncproxy server. We strongly recommend setting up and using your own vncproxy server. You will retain control over server performance and security noVNC WEB proxy server: vncproxy.puqcloud.comnoVNC WEB proxy key:puqcloudWEB ports: 80/443VNC ports: 5900-5999 With vncproxy you make a proxy between the client and your PROXMOX server.vncproxy must have an unequal stable network with the proxmox server, ports 5900-5999 are enoughAlso, if you use a domain name in connecting the PROXMOX seraer to the WHMCS system, this domain name must be correctly resolved from the vncproxy server Let's start with installation. Installation process Domain definition First, define a domain name for the vncproxy server, in our case it will be vncproxy.puqcloud.com Further in the example, we will use the domain name vncproxy.puqcloud.com, but in all your configurations you must use your own domain name. NGINX installation and configuration Secondly, you need to install a server with your favorite operating system. In our case, this is the Debian 11 operating system. You also need to set up a DNS entry on your domain so that it returns the IP address of the server. At first, if you haven't updated the package database recently, update it: sudo apt update Install nginx WEB server and Certbot sudo apt install certbot nginx python3-certbot-nginx zip -y Download noVNC client cd /root/ wget https://github.com/novnc/noVNC/archive/refs/tags/v1.3.0.zip unzip v1.3.0.zip cp -R noVNC-1.3.0/* /var/www/html/ rm v1.3.0.zip rm -r noVNC-1.3.0/ Now, going to http://vncproxy.puqcloud.com/vnc.html will open the noVNC client. Generate SSL certificate and install it in WEB server using certbot certbot --nginx -d vncproxy.puqcloud.com In order for the certificate to be updated automatically, you must add to the crontab crontab -e 0 12 * * * /usr/bin/certbot renew --quiet NGINX virtual host configuration Make the necessary settings in your domain configuration file in the nginx server nano /etc/nginx/sites-available/default server { listen 80 default_server; listen [::]:80 default_server; root /var/www/html; index index.html index.htm index.nginx-debian.html; server_name _; location / { try_files $uri $uri/ =404; } } server { root /var/www/html; index index.html index.htm index.nginx-debian.html; server_name vncproxy.puqcloud.com; # managed by Certbot location / { try_files $uri $uri/ =404; } listen [::]:443 ssl ipv6only=on; # managed by Certbot listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/vncproxy.puqcloud.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/vncproxy.puqcloud.com/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot location /vncproxy { proxy_pass http://127.0.0.1:8080/vncproxy; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } server { if ($host = vncproxy.puqcloud.com) { return 301 https://$host$request_uri; } # managed by Certbot listen 80 ; listen [::]:80 ; server_name vncproxy.puqcloud.com; return 404; # managed by Certbot } service nginx restart Next step is to install vncproxy apt-get install screen -y cd /root/ wget http://download.puqcloud.com/WHMCS/servers/PUQ_WHMCS-Proxmox-KVM/vncproxy/vncwebproxy chmod +x vncwebproxy We will run the script in the screen.The first script parameter is a unique key. noVNC WEB proxy key - in WHMCS module parameter. screen ./vncwebproxy puqcloud After a successful launch, you can watch the execution log in the console. root@vncproxy:~# ./vncwebproxy puqcloud [./vncwebproxy puqcloud] proxmox-test.uuq.pl59002022/09/11 19:11:08 [vncproxy][debug] ServeWS 2022/09/11 19:11:08 [vncproxy][debug] request url: /vncproxy/proxmox-test.uuq.pl/5900/d91bac199c2ce79392d8e175076e3780 2022/09/11 19:11:13 [vncproxy][info] close peer [GIN] 2022/09/11 - 19:11:13 | 200 | 4.740249024s | 79.184.10.217 | GET "/vncproxy/proxmox-test.uuq.pl/5900/d91bac199c2ce79392d8e175076e3780" Security The security setting for the server should meet your standards. Do not forget that for the correct operation of the server, you must allow connections to 80/443 ports. And outgoing connections to the PROXMOX server. 4. Create new server for Proxmox in WHMCS  Order now | Dowload | FAQ Preface For the module to work properly, you must configure the server settings in Your main WHMCS panel. This is the place to set up a proxmox server or proxmox cluster, which will then be used to build KVM machines. Here we define the access data IP ranges and additional settings. Attention.If you have one server, or you do not use server groups. Van needs to make this server the default. By clicking on it with the mouse.Make this server the active default for new signups Server creation Login to Your WHMCS panel and create new server Proxmox in WHMCS (System Settings->Products/Services->Servers) System Settings->Servers->Add New Server Enter the correct Name and Hostname In field Assigned IP Addresses you must enter a list of IP addresses of virtual machines that will be reserved for this server. Starting with version 1.3, the module started supporting IPv4/IPv6 pools.In order to manage pools of IPv4 and IPv6 IP addresses, you need to install the PUQ Customization addon module (free)https://doc.puq.info/books/puq-customization-whmcs-addon/page/ip-poolshttps://doc.puq.info/books/puq-customization-whmcs-addon/chapter/module-puqproxmoxkvmhttps://doc.puq.info/books/puq-customization-whmcs-addonhttps://download.puqcloud.com/WHMCS/addons/PUQ-Customization/ Format to follow in the Assigned IP Addresses field. To define the available pool of IP addresses, for each available IP number you should enter another line where the data is separated by the "|" separator. Each line with an IP number definition has the following structure: |||||, - the bridge to which the machine is connected is virtual.  - vlan which will be installed on the map of the network machine. In case of not using vlan, you need to set 0 Example: vmbr0|10|192.168.10.2|24|192.168.10.1|8.8.8.8,1.1.1.1 vmbr0|10|192.168.10.3|24|192.168.10.1|8.8.8.8,1.1.1.1 vmbr0|10|192.168.10.4|24|192.168.10.1|8.8.8.8,1.1.1.1 vmbr0|30|192.168.20.2|24|192.168.20.1|8.8.8.8,1.1.1.1 vmbr0|30|192.168.20.3|24|192.168.20.1|8.8.8.8,1.1.1.1 vmbr0|30|192.168.20.4|24|192.168.20.1|8.8.8.8,1.1.1.1 vmbr1|333|172.16.5.2|24|172.16.5.1|8.8.8.8,1.1.1.1 vmbr1|333|172.16.5.3|24|172.16.5.1|8.8.8.8,1.1.1.1 vmbr1|333|172.16.5.4|24|172.16.5.1|8.8.8.8,1.1.1.1 vmbr3|0|10.0.25.2|24|10.0.25.1|10.0.10.10,10.0.10.20 vmbr3|0|10.0.25.3|24|10.0.25.1|10.0.10.10,10.0.10.20 vmbr3|0|10.0.25.4|24|10.0.25.1|10.0.10.10,10.0.10.20 vmbr3|0|10.0.25.5|24|10.0.25.1|10.0.10.10,10.0.10.20 vmbr3|0|10.0.25.6|24|10.0.25.1|10.0.10.10,10.0.10.20 Enter the correct data in the username and password field In the Server Details section, select the "PUQ Proxmox KVM" module and enter the correct username and password for the Proxmox web interface. To check, click the "Test connection" button Note that the username is entered with the Proxmox account type (@pam or @pve) During operation, the module will automatically fill in the Access Hash field. This field does not need to be completed. AttentionIf you have one server, or you do not use server groups. Van needs to make this server the default. By clicking on it with the mouse.Make this server the active default for new signups 5. Synchronization of DNS records  Order now | Dowload | FAQ Starting with version 1.4, the module started supporting DNS Synchronization (cloudflare.com, HestiaCP) In order to manage DNS zones to synchronization , you need to install the PUQ Customization addon module (free) https://doc.puq.info/books/puq-customization-whmcs-addon/page/dns-zoneshttps://doc.puq.info/books/puq-customization-whmcs-addon/chapter/module-puqproxmoxkvmhttps://doc.puq.info/books/puq-customization-whmcs-addonhttps://download.puqcloud.com/WHMCS/addons/PUQ-Customization/ For your convenience, we have prepared a mechanism that generates basic domain entries for newly created servers. Setting these entries is not necessary for the correct operation and use of the service. In this version of the module, we do not have integration of procedures and mechanisms for synchronizing DNS records with various DNS servers. The module returns all IP addresses and DNS records as json using an http request. You need to do the integration with your DNS server yourself. We want to introduce automatic integration of DNS records into popular services and DNS servers. But we don't know where to start. You can help us with this. Please visit our forum and post your needs. We will definitely try to implement them in future versions.https://faq.puqcloud.com/ How it works In order to get all IP addresses and DNS records, you need to send a GET request. https:///modules/servers/puqProxmoxKVM/lib/dns/dns.php Answer: [ { "forward" : "vlan-1-4779.vps.uuq.pl", "ip" : "192.168.0.2", "reverse" : "mail.uuq.pl" }, { "forward" : "vps-1-4780.vps.uuq.pl", "ip" : "192.168.0.3", "reverse" : "test.vps.uuq.pl" }, { "forward" : "vlan-1-4781.vps.uuq.pl", "ip" : "192.168.0.4", "reverse" : "blabla.vps.uuq.pl" } ] The script does not return entries that contain errors or are empty. With this information, you can import DNS records into your DNS server. Security For unauthorized access in the directory with the dns.php file, there is a .htaccess file in which you can allow access to specific IPs. .htaccess file example order deny,allow deny from all allow from 6a. Email Template (puqProxmoxVKM Welcome Email)  Order now | Dowload | FAQ Create an email template for customer notifications. System Settings->Email Templates->Create New Email Template Email Type: Product/service Unique Name: puqProxmoxVKM Welcome Email Subject: Virtual Machine Order Information Body: Dear {$client_name}, Your order has been accepted for implementation. Installing and pre-configuring the virtual machine will take some time. Please wait for an e-mail with information that the virtual machine is ready for use, also with access parameters. Product/Service: {$service_product_name} Payment Method: {$service_payment_method} Amount: {$service_recurring_amount} Billing Cycle: {$service_billing_cycle} Next Due Date: {$service_next_due_date} Important note - if you have also purchased the backup options, do not forget to configure the schedule in the service's subpage. Thank you for choosing us. {$signature} 6b. Email Template (puqProxmoxKVM VM is ready)  Order now | Dowload | FAQ Create an email template for customer notifications. System Settings->Email Templates->Create New Email Template Email Type: Product/service Unique Name: puqProxmoxKVM VM is ready Subject: Virtual Machine is ready Body: Dear {$client_name}, Your virtual machine is already ready. You can connect to it using data. Product/Service: {$service_product_name} Payment Method: {$service_payment_method} Amount: {$service_recurring_amount} Billing Cycle: {$service_billing_cycle} Next Due Date: {$service_next_due_date} IP address: {$service_dedicated_ip} or {$service_domain} Username: {$service_username} Password: {$service_password} Thank you for choosing us. {$signature} 6c. Email Template (puqProxmoxKVM Reset password)  Order now | Dowload | FAQ Create an email template for customer notifications. System Settings->Email Templates->Create New Email Template Email Type: Product/service Unique Name: puqProxmoxKVM Reset password Subject: Reset password is ready Body: Dear {$client_name}, Password reset successful. IP address: {$service_dedicated_ip} or {$service_domain} Username: {$service_username} Password: {$service_password} Thank you for choosing us. {$signature} 6d. Email Template (puqProxmoxKVM Backup restored)  Order now | Dowload | FAQ Create an email template for customer notifications. System Settings->Email Templates->Create New Email Template Email Type: Product/service Unique Name: puqProxmoxKVM Backup restored Subject: Backup restored successful Body: Dear {$client_name}, Backup restored successful. Product/Service: {$service_product_name} Payment Method: {$service_payment_method} Amount: {$service_recurring_amount} Billing Cycle: {$service_billing_cycle} Next Due Date: {$service_next_due_date} IP address: {$service_dedicated_ip} or {$service_domain} Thank you for choosing us. {$signature} 6e. Email Template (puqProxmoxKVM Upgrade Email)  Order now | Dowload | FAQ Create an email template for customer notifications. System Settings->Email Templates->Create New Email Template Email Type: Product/service Unique Name: puqProxmoxKVM Upgrade Email Subject: Virtual Machine upgrade is ready Body: Dear {$client_name}, Virtual Machine upgrade is successful. Product/Service: {$service_product_name} Payment Method: {$service_payment_method} Amount: {$service_recurring_amount} Billing Cycle: {$service_billing_cycle} Next Due Date: {$service_next_due_date} IP address: {$service_dedicated_ip} or {$service_domain} Thank you for choosing us. {$signature} 7. Troubleshooting  Order now | Dowload | FAQ CRON In the event that the module does not work properly and activities are not performed, please check the operation of the CRON service in the first place. Here you can read how the process of provisioning virtual machines works.https://doc.puq.info/books/proxmoxkvm-whmcs-module/page/basic-concepts-and-requirements If your problem is still not resolved, then please run the cron job manually and look at the output in the console. /usr/bin/php7.4 -q /WHMCS_DIR/crons/cron.php There must be something like that. =========================================== VM_id: 2001 Service_id: 4785 User_id: 1 VMSetDedicatedIp: The local status should be creation VMClone: The local status should be set_ip VMSetCpuRam: success VMSetSystemDiskSize: success VMSetSystemDiskBandwidth: success VMSetCreatedAdditionalDisk: success VMSetAdditionalDiskSize: success VMSetAdditionalDiskBandwidth: success VMSetNetwork: success VMSetFirewall: success VMSetCloudinit: success VMStart: success Remote_status: running Local_status: ready ServiceSendEmailVMReady: OK Client Area Home screen  Order now | Dowload | FAQ Basic information on client panel page The end customer, after logging in to his own customer panel, has access to the following information and options: Status and used resources of the virtual machine in real time (with a 5-second automatic data refresh). The keys with the following functions are available. Start and stop the virtual machine Connect to the virtual machine terminal using the noVNS WEB client. Charts - switch to real-time virtual machine resource usage statistics. Statistics are presented in the form of well-readable graphs. Reinstall the operating system. Snapshot - go to snapshot management page. Backup - go to backups management page. Reset the (root) password of the virtual machine's operating system. Configure the reverse DNS record is available. Mount an ISO image is available. There are times when not all control options are available. For example, when creating or restoring a backup, or when the options are disabled for a specific client. Unavailable options are deactivated but visible. In this case, their color will be darkened and will not be clickable. Panel view with inactive options An example of a homepage in case of not available options (disabled options: Charts, Reinstall, Reset passwords, revDNS configuration, ISO mount). Panel view while backup is in progress Navigation menu  Order now | Dowload | FAQ There is a navigation menu in the client service management area.With the help of this menu, the client can quickly switch between the settings of his service. If, for some reason, the configuration setting is not allowed, then after going to the page with this setting, the client will receive an error stating that access is denied. Resource usage charts  Order now | Dowload | FAQ The module gives you access to a subpage with charts. Here, you can conveniently select the resource usage history period. The following options are available for selection. For the last hour For the last day for the last week For the last month For the last year Data is transferred from the hypervisor server in real time when the desired option is selected. Virtual machine reinstallation procedure  Order now | Dowload | FAQ You can reinstall virtual machine at any time if you need to. On the operating system reinstallation option page the client is given the option to select a new operating system from a list of available ones.There is also a warning that all data will be deleted, and the machine will be reinstalled. In order to avoid erroneous reinstallation, we have provided an option that protects against this. The customer needs to manually write the word reinstall in capital letters "REINSTALL" The process begins by removing snapshots and data on the current machine and then starting the installation of a new machine with an already existing IP number, network card MAC address, vlan and ID. The process permanently deletes data from the machine leaving its backups intact. This gives you the option to restore your machine from a backup to the previous state it was in before reinstallation, even after reinstalling the new system. Please use this option carefully. Snapshots management  Order now | Dowload | FAQ On the page for managing snapshots Snapshots are a quick way to save the state of the system, which allow you to restore the state of the system as quickly as possible. Snapshots are not a method of system backups and are most often used during system administration work, hence their retention period is limited. The client gets the opportunity to create and remove snapshots of a virtual machine. It is also possible to restore the state of a virtual machine from the snapshot. It also shows information about the number and limit of snapshots. Depending on the configuration of the service package. The current lifetime of the snapshot will be indicated, after this time the snapshot will be automatically deleted. Backups management  Order now | Dowload | FAQ On the page with backup management The client gets the opportunity to manage backups of his virtual machine.There is a schedule option to automatically create spare copies. Clint is given the ability to create delete backups. It also provides the ability to restore a virtual machine from a backup. By default, the schedule is not configured and no copies are made. The client must configure the schedule itself. The time indicated in the schedule is generated automatically based on the analysis of the system load and is not subject to customer modifications. Reset password  Order now | Dowload | FAQ On the password reset page. The client is given the opportunity to reset the operating system password. If you use your own templates please note that the operating system must support cloud-init for this feature to work. All our templates support this mechanism. revDNS configuration  Order now | Dowload | FAQ On the configuration page of the reverse DNS zone, the client has the opportunity to set the domain name for the IP address of his virtual machine. Currently, the module does not support the automatic update of revDNS, but supports the creation of requests with a request to update this setting by a specialist responsible for it in the company. You can set this option 'Revdns ticket/RevDNS ticket department' during product configuration. ISO templates to mount  Order now | Dowload | FAQ We have prepared a set of templates in the form of ISO images that you can use on your own. All our images have been specially prepared for the proper operation of our module and have cloud-init support. To the iso mount page The client is given the opportunity to mount the ISO disk image on the CD-ROM of their virtual machine. Error pages in client area  Order now | Dowload | FAQ The error "Something went wrong" is returned if: there is no access from the WHMCS server to the Proxmox server, the virtual machine was not found on the Proxmox server. An "Access Denied" error is returned if the client tried to switch to an option that is not allowed for use in the package. Admin Area Product Home Screen  Order now | Dowload | FAQ The main screen of the product contains the following items. Module Command buttons of the module for managing the service and the virtual machine Section with the id of the virtual machine and a button for setting a new ID Connection API status to PROXMOX server Button to connect to the noVNC server console The section in which in real time with an update of 1 time for 5 seconds the parameters and stars of the virtual machine are shown Section with graphs with virtual machine statistics Module Command buttons  Order now | Dowload | FAQ Status concepts. Remote status The status of the virtual machine as returned by the PROXMOX server.May be: running/stopped VM remote lock May be: clone/backup Local status The status is internal, to control and select automation actions using the module. The local state can take the following values: creation - the first status issued at the time of service creation. Indicates that the virtual machine creation process should start reinstall -status if the reinstallation process is running and the virtual machine is in the reinstallation queue clone - the process of cloning a virtual machine from a template set_cpu_ram - the process of configuring the number of processor cores and RAM memory was successful set_system_disk_size  - the process of creating the size of the system disk has completed successfully set_system_disk_bandwidth  - the process of configuring system disk bandwidth limits was successful set_created_additional_disk - the process of creating an additional disk is successful. Also if the additional disk is not present in the package configuration set_additional_disk_size - the process of creating the size of theadditional disk has completed successfully. Also if the additional disk is not present in the package configuration set_additional_disk_bandwidth  - the process of configuringadditional disk bandwidth limits was successful. Also if the additional disk is not present in the package configuration set_network - network card configuration (vlan, bridge) completed successfully set_firewall - firewall rules added successfully set_cloudinit - cloud init configuration successful ready - the status characterizes that the virtual machine was created correctly and is ready to work set_dns_records - means that the next time the crontab is executed, DNS records will be synchronized change_package - means that the next time you run crontab, the settings of the virtual machine will be changed, such as (disk, ram, processor, IP) Standard Module Command buttons Create This function runs when a new product is provisioned. This can be initiated by WHMCS upon checkout or Client making payment for a new order. Also, it can be initiated by an admin user from the Products/Services tab in a client's profile of the admin area. General algorithm of action: An available IP address is selected from the list in the server settings. The available virtual machine ID is selected in such a way that it is unique in the WHMCS system and Proxmox server The fields with the domain name and the name of the virtual machine are generated and filled. The procedure uses the data supplied in the virtual machine package. Start the process of cloning a virtual machine from the desired virtual machine template. All other functions for creating a virtual machine will be implemented by a cron task Terminated service IP addresses are also taken into account and can be assigned to a new service. Suspend This function runs when a suspension is requested. It can be requested automatically by the WHMCS Cron when a product becomes overdue, or by admin user in the Client Profile. Makes the virtual machine stop if the virtual machine is powered on. Unsuspend This function runs when an unsuspension is requested. It can be requested upon payment of an overdue invoice for a product. Makes the virtual machine start if the virtual machine is powered off. Terminate This function runs when a termination is requested. It can be automatically requested by the WHMCS Cron for long overdue products when enabled (Automation Settings). It can be also requested by an admin user in the Client Profile. General algorithm of action: Deletes all backups and snapshots of the virtual machine. Makes the machine virtual stop if the virtual machine is powered on. Then it deletes the virtual machine. Adds information with the date and IP address to the admin notes on the client service. Returns the machine's used IP address back to the pool of free useable IP addresses. Cleans up data. Change Package This function runs during procedure of upgrading or downgrading products. This function runs when an upgrade or downgrade order, placed by the client, becomes paid. Admins can also invoke this from the product management pages. The same function runs for upgrades and downgrades of both products and configurable options. General algorithm of action: Makes the virtual machine stop if the virtual machine is powered on. Changes the parameters of the virtual machine to the parameters specified in the package. Makes the start of the virtual machine. Additional Module Command buttons All possible actions available to be performed in the admin panel have short symbols that will help you understand the consequences of the action.* -  It is possible to run a function with which will do actions on the virtual machine in the running state.** -  It is possible to run only if the virtual machine is in the stopped state->  - means that this function participates in the automatic creation of the virtual machine and points to the next function that must be run to correctly complete the creation of the virtual machine. Start Starts the virtual machine if the virtual machine is in the stopped state and the local state is ready Stop Stops the virtual machine if it is in the running state Reinstall Starts the process of reinstalling the virtual machine VMSetDedicatedIp An available IP address is selected from the list in the server settings. Terminated service IP addresses are also taken into account and can be assigned to a new service. VMClone Starts the process of cloning a virtual machine from the desired virtual machine template. **Set CPU RAM Forces configuration of the processor and memory settings of the virtual machine relative to the package configuration. *Set System Disk Size Forces the size of the system disk according to the parameters in the package *Set System Disk Bandwidth Forces system disk bandwidth settings according to packet settings. *Set Created Additional Disk Creates an additional virtual machine disk if configured accordingly in the package *Set Additional Disk Size Forces the size of the additional disk relative to the parameters in the package, if needed *Set Additional Disk Bandwidth Forces additional disk bandwidth settings relative to packet settings, if needed *Set Network Forces configuration of network card parameters such as bridge VLAN and bandwidth relative to the parameters in the packet *Set Firewall Adds rules to the firewall. DOES NOT REMOVE OLD. **SetCloudinit Forces cloud-init device settings to be configured relative to the settings in the package. **VMRemove Delete the virtual machine. Leaves backups and does not delete unused disks. Set DNS records DNS records will be synchronized immediately and before completion. May be busy for a long time. Product Configuration  Order now | Dowload | FAQ Add new product to WHMCS System Settings->Products/Services->Create a New Product In the Module settings section, select the "PUQ ProxmoxKVM" module License key A pre-purchased license key for the "PUQ ProxmoxKVM" module. For the module to work correctly, the key must be active Type of virtual machine creation How will the virtual machine will be cloned from the templateFull Clone - much more times for deployment.Linked clone VM - faster deployment, requires less disk space but cannot run without access to the base VM Template Target node The node in the cluster on which the virtual machine should be created. If you select the automatic option or the node was unavailable at the time the virtual machine was created, the module will actually select the node with the most real free RAM (not a percentage, but the most free RAM numerically).If a specific node is selected manually, the module does not check the available RAM in any way. Default OS template Sets the default operating system template. In cases where the template chosen by the client is not available, or if the options with the choice of the operating system are not configured, this template will be used. Backups storage Data store for backups of virtual machines If you are using a cluster, then you need to have this data store on a network protocol, such as NFS. ISO's storage Data store for ISO images If you are using a cluster, then you need to have this data store on a network protocol, such as NFS. VM configuration Basic configuration settings for the virtual machine CPU - number of CPU cores for a new machine RAM - amount of RAM for a new machine OS username - The user that will be configured in the virtual machine settings will be transferred to the client and will appear in the password change Depending on cloud-init settings in the template. If there is a root user in the template and you also enter root in the service configuration in WHMCS, a password for the root user will be generated when creating the VM. If you enter another user name in this window (but the template does not contain one), it will be created when the machine is created and the password will be sent to the client. VM name prefix: The VM name prefix will be added to the virtual machine name. VM name: -- First vm id: ID from which to start searching for a free id for a new virtual machine Snapshot lifetime: After how many days will snapshots be removed automatically System disk The system disk is determined by the one that is assigned to the first boot in the virtual machine or template. System disk options. The system drive is the drive on which the system is installed. Limitations are possible both in terms of bandwidth in megabytes per second, as well as restrictions on the number of I/O operations. Please note that the size of the system disk must be greater than or equal to the size of the system disk in the template. Additional disk The additional drive is the second drive in the virtual machine or template that is not assigned to the first boot option. If an additional disk is not created in the virtual machine template, it will be created automatically on the same storage as the system disk. If you need an additional disk on a different storage than the system disk, it must be then created in the virtual machine template on the required storage. Additional disk options.  Limitations are possible both in terms of bandwidth in megabytes per second, as well as restrictions on the number of I/O operations. Network configuration The section in which you must specify the parameters of the network virtual machine map. Network model - The network card model of the virtual machine Automatic selection bridge/VLAN -When creating a virtual machine, the IP address will be selected automatically from the list of available IP addresses from the PROXMOX server parameter or IP Pools added in the WHMCS system. Regarding the selected IP address, the network card, bridge and vlan will be configured. Network bridge, VLAN -  If you do not use the Automatic selection, bridge/VLAN. You need to select bridge and VLAN. The IP address will be selected from the list of IP addresses available in this bridge and VLAN in the PROXMOX server address list settings in the WHMCS system. DHCP IPv4 - Whether to use a DHCP server. The IP address will not be selected from the list in the PROXMOX server in the WHMCS system. In this case, firewall rules won't be applied as we don't know IP address. DHCP IPv6 - Whether to use a DHCP server. The IP address will not be selected from the list in the PROXMOX server in the WHMCS system. In this case, firewall rules won't be applied as we don't know IP address. Firewall - Enable firewall on the network card of the virtual machine. Only works when DHCP is not used. Anti spoofing rules - This option only works if you are not using a DHCP server. It adds allowing rules for incoming and outgoing traffic with the IP address of the virtual machine. For the correct operation of the firewall rules to avoid spoofing the IP address of the virtual machine, it is necessary to configure the firewall on the PROXMOX server, the firewall policy DENY/DENY. In order to manage pools of IPv4 and IPv6 IP addresses, you need to install the PUQ Customization addon module (free)https://doc.puq.info/books/puq-customization-whmcs-addonhttps://download.puqcloud.com/WHMCS/addons/PUQ-Customization/https://puqcloud.com/whmcs-addon-puq-customization.php Integrations configuration This section contains settings that are responsible for integrating the module with other systems. noVNC WEB proxy server, noVNC WEB proxy key - VNCPROXY server settings.VNCPROXY and noVNC are used to enable the client to connect to the virtual machine console (Keyboard, Video, Mouse). How to set up a VNCPROXY server is described in https://doc.puq.info/books/proxmoxkvm-whmcs-module/page/install-vncproxy-and-novnc For your comfort we provide our public VNCPROXY server with noVNC client. You can use it freely, but we strongly recommend creating your own VNCPROXY server with noVNC clientnoVNC WEB proxy server: vncproxy.puqcloud.comnoVNC WEB proxy key: puqcloud Main domain - The domain that you use to present the virtual machine. In the WHMCS system, the domain column on the client's product will be filled in the format --.
. Also, this parameter will be specified as "search" in the virtual machine in the /etc/resolv.conf file. Revdns ticket/RevDNS ticket department - In case you haven't created a reverse DNS zone record integration. The module can create a ticket when the client changes the reverse zone entry in the settings. You will have to manually process the request. How you can integrate DNS zone records with your DNS server is described here: https://doc.puq.info/books/proxmoxkvm-whmcs-module/page/synchronization-of-dns-records Email configuration In this section, you need to select email templates that will be used by the module to communicate with the client. VM is ready - This email will be sent each time a virtual machine is created. In case of creation and in case of reinstallation. In this email, the client will receive a login and password to access the virtual machine system. Reset password - This email will be sent to the client every time the client uses the password reset function, the client will receive a login and password to access the virtual machine system. Backup restored - This email will be sent to the client every time the client uses the restore virtual machine from backup function. Sample templates are available here:https://doc.puq.info/books/proxmoxkvm-whmcs-module/page/email-template-puqproxmoxkvm-vm-is-readyhttps://doc.puq.info/books/proxmoxkvm-whmcs-module/page/email-template-puqproxmoxkvm-reset-passwordhttps://doc.puq.info/books/proxmoxkvm-whmcs-module/page/email-template-puqproxmoxkvm-backup-restored Additionally: https://doc.puq.info/books/proxmoxkvm-whmcs-module/page/email-template-puqproxmoxvkm-welcome-emailhttps://doc.puq.info/books/proxmoxkvm-whmcs-module/page/email-template-puqproxmoxkvm-upgrade-email Client configuration In this section, you can configure the client's access rights to manage its virtual machine. Configurable Option (Operating system selection)  Order now | Dowload | FAQ In order for the client to have a choice of operating system in the configuration of the virtual machine. You need to configure the Configurable Option and connect them to the package. Create Group Add new Configurable Option to WHMCS System Settings->Configurable Option->Create a New Group Enter the group name, description, and select the products you need. Group Name: Operating system Add Configurable Option After saving, create a new Configurable Option Price definitions In this part of the settings, we define individual price lists for selected individual options. We use the WHMCS standard and add the sign "|" to describe the individual options. The convention presented in our guide must be followed on the left side of the sign. To the right of the sign "|" we give the name that will be displayed in the views for the user (admin, client) It is very important to enter the names of the options precisely, because they are also used to define the settings and any mistakes or typos will prevent the correct operation of the entire ordering process and starting the virtual machine. Option name Option Name: OS|Operating system You can replace the name "Operating system" with your own, ie: My offered system. It is important that the option name be started with the prefix "OS |" Options The Options format is this: template_id|name of system separated by "|" You can replace the name "name of system" with your own, ie: Debian, Nextcloud, my system x etc. template_id is the number of a specific template machine uploaded to proxmox and recreated as template. It is important that the options be started with the prefix "template_id |" Please look at this example - as options we have defined: 1010|Debian-10.12. We use that precise text string because we have deployed on Proxmox node/cluster template with VM_ID: 1010 which is based on Debian 10 release. Configurable Option (Backups/Snapshots/CPU/RAM/IPv4/IPv6)  Order now | Dowload | FAQ In order for the client to have a choice of Backups/Snapshotsin/CPU/RAM/IPv4/IPv6 the configuration of the virtual machine. You need to configure the Configurable Option and connect them to the package. List of available options It is important that the option name be started with the prefix "prefix|name" The prefix is required for the module to define the option. Name can be changed at will Option Name: B|Backup -> Options: |Name Option Name: S|Snapshot -> Options: |Name Option Name: CPU|Processor -> Options: |Name Option Name: RAM|Memory -> Options: |Name Option Name: ipv4|IPv4 -> Options: |Name Option Name: ipv6|IPv6 -> Options: |Name Option Name: OS|Operating system -> Options: |Name Option Group Add new Configurable Option to WHMCS System Settings->Configurable Option->Create a New Group Enter the group name, description, and select the products you need. Group Name: Backups/Snapshots After saving, create a new Configurable Option Price definitions In this part of the settings, we define individual price lists for selected individual options. We use the WHMCS standard and add the sign "|" to describe the individual options. The convention presented in our guide must be followed on the left side of the sign. To the right of the sign "|" we give the name that will be displayed in the views for the user (admin, client) It is very important to enter the names of the options precisely, because they are also used to define the settings and any mistakes or typos will prevent the correct operation of the entire ordering process and starting the virtual machine. Backup definitions Add an option with a choice of the number of backups Option name Option Name: B|Backup You can replace the name "Backup" with your own, ie: Sicherung, My backup offer etc.. It is important that the option name be started with the prefix "B|" Options The Options format is this: number_of_backups|name  separated by | You can replace the name "name" with your own. It is important that the options be started with the prefix "x |" where "x" represent number of allowed backup Snapshot definitions Add an option with a choice of the number of snapshots Option name Option Name: S|Snapshot You can replace the name "Snapshot" with your own, ie: Fast Snapshots, personal Snapshot etc. It is important that the option name be started with the prefix "S|" Options The Options format is this: number_of_snapshot|name    separated by | You can replace the name "name" with your own. It is important that the options be started with the prefix "x |" where "x" represent number of allowed snapshots IPv4/IPv6 Pools  Order now | Dowload | FAQ Starting with version 1.3, the module started supporting IPv4/IPv6 pools. In order to manage pools of IPv4 and IPv6 IP addresses, you need to install the PUQ Customization addon module (free) https://doc.puq.info/books/puq-customization-whmcs-addon/page/ip-poolshttps://doc.puq.info/books/puq-customization-whmcs-addon/chapter/module-puqproxmoxkvmhttps://doc.puq.info/books/puq-customization-whmcs-addonhttps://download.puqcloud.com/WHMCS/addons/PUQ-Customization/ DNS Zones Synchronization  Order now | Dowload | FAQ Starting with version 1.4, the module started supporting DNS Synchronization (cloudflare.com, HestiaCP) In order to manage DNS zones to synchronization , you need to install the PUQ Customization addon module (free) https://doc.puq.info/books/puq-customization-whmcs-addon/page/dns-zoneshttps://doc.puq.info/books/puq-customization-whmcs-addon/chapter/module-puqproxmoxkvmhttps://doc.puq.info/books/puq-customization-whmcs-addonhttps://download.puqcloud.com/WHMCS/addons/PUQ-Customization/