jt-pve-storage-qnap

Which Proxmox VE operations work支援的 Proxmox VE 操作

Implemented and unit-tested, not yet driven on hardware, for virtual machines and for containers. The rows that are not a plain yes say why. 已實作並通過單元測試，但尚未在實機上執行過，虛擬機與容器都一樣。不是單純「支援」的那幾列，都寫出原因。

A container's disk is the same object as a virtual machine's: one thin LUN, one multipath device. The difference is what sits on top. Proxmox VE puts a filesystem on a container's LUN and mounts it on the host, where a VM's disk is handed to the guest whole. That is why two rows below differ, and only two. 容器的磁碟和虛擬機的磁碟是同一種東西：一個 thin LUN、一個 multipath 裝置。差別在上層。Proxmox VE 會在容器的 LUN 上建立檔案系統並掛接在主機上，而虛擬機的磁碟是整顆交給 guest。所以下表只有兩列不同。

Which QNAP operating systems are supported支援的 QNAP 作業系統

Deliberately specific about the difference between supported, expected to work and not supported, because two of those are promises and one is not. 這裡把「支援」、「應可運作」與「不支援」分開說明，因為前後兩個是確定的答案，中間那個不是。

The hard requirement: Storage Manager V2硬性需求：Storage Manager V2

QNAP has two generations of Storage Manager and they are different sets of CGI calls, not two versions of one API. This plugin implements V2 only, and the firmware says which it has, so a firmware that is too old is refused at pvesm add, with its version in the message, instead of adding cleanly and then listing nothing. QNAP 有兩代 Storage Manager，兩者是兩套不同的 CGI 呼叫，不是同一套 API 的兩個版本。這個 plugin 只實作 V2。韌體會回報自己是哪一種，所以太舊的韌體會在 pvesm add 時被拒絕，訊息裡會寫出版本，不會先新增成功，之後才列不出任何東西。

The matrix支援對照表

"Expected to work" means nothing in what this plugin uses is known to differ. It does not mean anyone has run it. Treat it as you would any untested combination. 「應可運作」的意思是：這個 plugin 用到的部分沒有已知的差異。這不代表有人實際執行過，請當成任何未經測試的組合來看待。

Model requirements, which are not the same as firmware機型需求，這和韌體版本是兩回事

• The iSCSI target service. Every NAS in scope has it, but it ships off. The plugin refuses to add a storage while it is off and says so.iSCSI target 服務：範圍內的 NAS 都有，但出廠時是停用的。服務停用時，plugin 會拒絕新增 storage，並說明原因。
• A storage pool. This plugin creates block-based LUNs, which live in a pool. A NAS whose disks are configured as static volumes has no pool for them.儲存集區：這個 plugin 建立的是區塊型 LUN，必須放在儲存集區裡。磁碟設定成靜態磁碟區的 NAS，沒有可用的儲存集區。
• Snapshots. QNAP's entry-level models do not support snapshots of iSCSI LUNs at all, and this plugin has no way to check that in advance. It warns when the storage is added and asks you to take one snapshot of a test disk before relying on the feature.快照：QNAP 的入門機型完全不支援 iSCSI LUN 快照，而且 plugin 無法事先檢查。新增 storage 時 plugin 會提出警告，請先對一顆測試磁碟建立一次快照，再開始使用這個功能。

The real ceiling is the LUN count真正的上限是 LUN 數量

One VM disk is one LUN, and a NAS has a maximum number of them. QNAP publishes 128 for QTS and 256 for QuTS hero on its product pages, and 255 for LUNs and targets combined in its user guides. The figure is the same on a two-bay model as on a twelve-bay one. A VM with a system disk and a data disk spends two, so plan on roughly 64 such VMs per NAS on QTS. The plugin reads the ceiling from the NAS rather than assuming, warns as it approaches, and refuses an allocation at the limit with a message that says free space will not help. 一顆虛擬機磁碟就是一個 LUN，而每台 NAS 能有的 LUN 數量有上限。QNAP 在產品頁公布的是 QTS 128、QuTS hero 256，在使用手冊公布的是 LUN 與 target 合計 255。兩顆硬碟的機型和十二顆硬碟的機型，數字都一樣。一台有系統磁碟與資料磁碟的虛擬機會用掉兩個，所以在 QTS 上，一台 NAS 大約可以規劃 64 台這樣的虛擬機。plugin 會向 NAS 讀取上限，不自行假設。接近上限時提出警告，到達上限時拒絕配置，並且說明增加空間沒有幫助。

storage 'qnap1': the NAS already holds 256 LUNs, which is this model's
maximum (256). Free space is not the problem and adding capacity will not
help. Delete LUNs, or use a second NAS. The count includes LUNs this
storage does not own, such as Virtual Machine Manager disks.

Requirements系統需求

The NAS account, and where the password livesNAS 帳號，以及密碼放在哪裡

It has to be an administrator必須是管理員帳號

The CGIs this plugin calls are administrator-only in QTS. A standard user, or one with delegated permissions on a shared folder, gets a session that logs in perfectly well and then has every one of those calls refused. Use a second administrator account created for this purpose rather than admin: it can be disabled without locking anyone out, and the NAS's own logs then show which actions were Proxmox VE's. 這個 plugin 呼叫的 CGI 在 QTS 裡只有管理員可以使用。一般使用者，或只有共用資料夾授權的使用者，可以正常登入，但每一個呼叫都會被拒絕。請另外建立一個專用的管理員帳號，不要使用 admin：需要時可以停用它而不影響其他人，而且從 NAS 自己的記錄就能分辨哪些動作是 Proxmox VE 執行的。

Not in storage.cfg密碼不會寫進 storage.cfg

/etc/pve/storage.cfg is root:www-data 0640, and PVE returns any property it does not know is a secret from GET /storage/<id> to any user holding Datastore.Audit. A read-only auditor would have been handed an administrator credential for your NAS. The plugin declares its three secrets as sensitive, so PVE strips them from the configuration, and they are written to /etc/pve/priv/storage/<storeid>.qnap: root only, and replicated to every node. /etc/pve/storage.cfg 的權限是 root:www-data 0640，而且 PVE 不知道是機密的欄位，會透過 GET /storage/<id> 回傳給任何具有 Datastore.Audit 權限的使用者。只有唯讀稽核權限的人，就會拿到 NAS 的管理員憑證。這個 plugin 把三個欄位宣告為機密，PVE 就會把它們從設定檔移除，改寫入 /etc/pve/priv/storage/<storeid>.qnap：只有 root 能讀取，並且會同步到每個節點。

A wrong password locks the node out, once密碼錯誤只會鎖一次

QTS blocks a source address after a few failed logins. Proxmox VE polls every storage every ten seconds on every node, so a wrong password would reach that threshold in well under a minute. The symptom afterwards is a refused connection, which looks like a dead NAS rather than a bad credential. The plugin therefore latches a refused credential: one failed attempt, recorded under /run/, and no retry until the storage configuration changes. Any pvesm set clears it. QTS 會在數次登入失敗之後封鎖來源位址。Proxmox VE 的每個節點每十秒都會輪詢每個 storage，所以密碼錯誤時，不到一分鐘就會達到門檻。之後的症狀是連線被拒絕，看起來像 NAS 故障，而不像密碼錯誤。所以 plugin 會記住被拒絕的憑證：只嘗試一次，記錄在 /run/ 底下，在 storage 的設定改變之前不再重試。執行任何一次 pvesm set 就會解除。

Installing安裝

On every node of the cluster: 在叢集的每一個節點上執行：

# on each node: the two packages PVE does not install for you
apt update
apt install -y open-iscsi multipath-tools

cd /tmp
# no version in the filename: this URL always gives you the newest release
wget -O jt-pve-storage-qnap_all.deb \
  https://github.com/jasoncheng7115/jt-pve-storage-qnap/releases/latest/download/jt-pve-storage-qnap_all.deb
apt install -y ./jt-pve-storage-qnap_all.deb
systemctl restart pvedaemon pveproxy pvestatd

dpkg -l jt-pve-storage-qnap | awk '/^ii/{print $3}'    # check what you got

Keep the -O. Without it, wget does not overwrite a file that is already there and saves the download under another name. apt then installs the old file left in /tmp from last time. 請保留 -O。少了它，wget 不會覆寫已經存在的檔案，而是存成另一個檔名。接著 apt 安裝的就是上次留在 /tmp 的舊檔。

Every node in the cluster, including the one you browse from, and keep them on the same version. A storage operation runs on the node that owns the guest, and a node without the plugin makes the storage invisible in the web interface rather than reporting an error. 叢集中的每個節點都要安裝，包含用來開啟網頁介面的那一台，而且版本要一致。storage 的操作是在擁有該 guest 的節點上執行的。沒有安裝 plugin 的節點不會回報錯誤，而是讓這個 storage 從網頁介面上消失。

Adding the storage新增 Storage

Every precondition is checked here (the firmware generation, the iSCSI service, the pool, the account), so a configuration that cannot work is refused now rather than at the first allocation. 所有前置條件都在這一步檢查（韌體版本、iSCSI 服務、儲存集區、帳號），所以無法運作的設定會在這時被拒絕，不會等到第一次配置磁碟才發現。

pvesm add qnapsan qnap1 \
    --qnap-portal 192.0.2.10 \
    --qnap-username pve \
    --qnap-password '<password>' \
    --qnap-pool 1 \
    --qnap-chap-username pve \
    --qnap-chap-password '<secret>' \
    --qnap-ssl-verify 0 \
    --content images

Then use it like any other storage: qm create --scsi0 qnap1:32, snapshots, rollback, templates, linked clones, live migration. 之後的用法和其他 storage 相同：qm create --scsi0 qnap1:32、快照、倒回、範本、連結複製、線上遷移。

The discovery tool探索工具

Run this before you add anything. It is read only (every call it makes is a query or the login itself), so it is safe against a production NAS, and it is the first thing to run when a storage will not add or a device will not appear. 請在新增任何東西之前先執行它。它是唯讀的（每個呼叫不是查詢就是登入），所以對正式環境的 NAS 是安全的。storage 無法新增，或裝置沒有出現時，也請先執行它。

pve-qnap-api-probe --host 192.0.2.10 --user admin --insecure --node

It prints the model, the firmware, whether it is QTS or QuTS hero, the LUN and target ceilings, the storage pools with their free space, and, with --node, what this node has installed and which sessions it holds. The password is read from the terminal rather than taken as an argument, so it stays out of the shell history and out of ps. 它會印出機型、韌體、是 QTS 還是 QuTS hero、LUN 與 target 的數量上限、各儲存集區與剩餘空間。加上 --node 還會印出這個節點安裝了哪些工具、有哪些工作階段。密碼是從終端機讀取，不是當成參數傳入，所以不會留在 shell 的歷史記錄裡，也不會出現在 ps 中。

The cleanup tool清理工具

Proxmox VE never tells the source node that a shared volume is no longer needed there. So a node a VM was migrated away from keeps a multipath map for a LUN it no longer uses, and if the VM was later destroyed elsewhere, for a LUN that no longer exists. 共用的磁碟在某個節點上不再需要時，Proxmox VE 不會通知該節點。所以虛擬機遷離之後，來源節點會留著一個已經用不到的 multipath map。如果那台虛擬機之後在別的節點被刪除，留下的就是一個指向已不存在 LUN 的 map。

pve-qnap-reap --all             # report only
pve-qnap-reap --all --remove    # act

It never touches a device that is in use, and it refuses rather than guessing when it cannot tell. 它不會動到正在使用中的裝置，無法確認狀態時會拒絕動作，不會猜測。

When something goes wrong故障處理

The storage will not addstorage 無法新增

• Run the probe first. It separates "the NAS refused the account" from "the firmware is too old" from "the iSCSI service is off", and those three look identical from pvesm add.先跑探索工具。它能分辨「NAS 拒絕帳號」、「韌體太舊」與「iSCSI 服務沒開」，而這三種從 pvesm add 看起來一模一樣。
• If the probe succeeded and pvesm add reports a login failure, suspect the POST question in TESTING.md.如果探索工具成功、pvesm add 卻回報登入失敗，請先懷疑 TESTING.md 裡關於 POST 的那一項。

A second node cannot log in to the target第二個節點無法登入 target

QTS's clustered-access flag is what admits more than one initiator, and targetInfo does not return it, so there is nothing for the plugin to compare against on the activation path. It is written when the target is created and re-applied on every pvesm set. If a node is refused, run: QTS 的叢集存取設定決定了能不能讓多個 initiator 同時連線，而 targetInfo 不會回傳它，所以 plugin 在啟用磁碟的過程中沒有東西可以比對。plugin 會在建立 target 時寫入，並在每次 pvesm set 時重新套用。如果有節點被拒絕，請執行：

pvesm set qnap1

A device does not appear on a node節點上出不來裝置

• Check find_multipaths in /etc/multipath.conf. With the default, strict, no device gets a map until its WWID is listed in /etc/multipath/wwids, however many paths it has. With yes, a single-path device gets no map at all, and a QNAP with one data address is the common case. The plugin handles both with multipath -a, but the message tells you where to look if that did not work.檢查 /etc/multipath.conf 裡的 find_multipaths。使用預設值 strict 時，WWID 沒有列在 /etc/multipath/wwids 的裝置，不論有幾條路徑都不會產生 map。設成 yes 時，單一路徑的裝置不會產生 map，而只有一個資料位址的 QNAP 很常見。這兩種情況 plugin 都會用 multipath -a 處理。如果沒有成功，訊息會說明該檢查哪裡。
• Check cat /sys/block/sdX/device/vendor. If it is not QNAP, the multipath drop-in this plugin writes does not apply to your model. Please report it.檢查 cat /sys/block/sdX/device/vendor。如果不是 QNAP，plugin 寫入的 multipath 設定就不會套用到這個機型，請回報。

Options設定選項

Related projects相關專案

Proxmox VE storage plugins for other arrays, sharing the host-side layer and the operational rules this one inherits. 其他儲存設備的 Proxmox VE storage plugin，與本專案共用主機端的架構與維運規則。