v0.2.14 MIT License
jt-pve-storage-netapp
NetApp ONTAP SAN/iSCSI/FC Storage Plugin for Proxmox VE NetApp ONTAP SAN/iSCSI/FC Proxmox VE 儲存外掛程式
Enterprise SAN storage integration for Proxmox VE using NetApp ONTAP REST API. Supports iSCSI and Fibre Channel with multipath, snapshots, FlexClone, live migration, and automatic device lifecycle management. 透過 NetApp ONTAP REST API 將企業級 SAN 儲存整合至 Proxmox VE。支援 iSCSI 及 Fibre Channel，具備 multipath、快照、FlexClone、即時遷移，以及自動裝置生命週期管理。
Disclaimer 免責聲明
WARNING: This is a newly developed project. Use at your own risk.
警告：此為新開發的專案，使用風險自負。
- This plugin is provided "AS IS" without warranty of any kind. 本 plugin 以「現況」提供，不附帶任何形式的保證。
- iSCSI protocol has been tested but not extensively in production environments. iSCSI 協定已經過實際客戶環境測試，但尚未大規模驗證。
- FC (Fibre Channel) protocol has NOT been fully verified. FC (Fibre Channel) 協定尚未完全驗證。
- Always test thoroughly in a non-production environment before deployment. 部署前請務必在非正式環境中徹底測試。
- Back up your data regularly and have a recovery plan in place. 請定期備份資料，並準備好復原計畫。
Features 功能特性
Complete SAN storage integration leveraging NetApp ONTAP enterprise capabilities. 完整的 SAN 儲存整合，充分運用 NetApp ONTAP 企業級功能。
1 VM Disk = 1 LUN = 1 FlexVol
1 VM Disk = 1 LUN = 1 FlexVol
Clean snapshot semantics matching the Proxmox VE model. Each disk is an independent ONTAP volume.
乾淨的快照語意，完全匹配 Proxmox VE 模型。每個磁碟都是獨立的 ONTAP volume。
Snapshot Create/Delete/Rollback
快照建立/刪除/倒回
Via ONTAP Volume Snapshots. Instant, space-efficient, with host-side buffer flush for consistency.
透過 ONTAP Volume Snapshots 實現。即時、空間效率佳，並以 host 端 buffer flush 確保一致性。
Template & Linked Clone
範本與 Linked Clone
Instant clones via NetApp FlexClone. Space-efficient -- no data copy required.
透過 NetApp FlexClone 即時複製。節省空間 -- 不需要複製資料。
Full Clone from Snapshot
從快照完整複製
Clone any VM from a specific snapshot to a fully independent VM.
從特定快照複製任何 VM 為完全獨立的 VM。
Real-time Capacity Reporting
即時容量報告
Storage usage reported directly from ONTAP REST API. Aggregate or volume-based.
儲存使用量直接從 ONTAP REST API 回報。可基於 aggregate 或 volume。
Multipath I/O for HA
Multipath I/O 高可用性
Automatic multipath configuration with device discovery and failover.
自動 multipath 設定，含裝置探索與容錯移轉。
Cluster-aware Live Migration
叢集感知即時遷移
LUNs mapped to all node igroups for seamless live migration between Proxmox VE nodes.
LUN 對應至所有節點 igroup，實現 Proxmox VE 節點間的無縫即時遷移。
Thin Provisioning
精簡配置 (Thin Provisioning)
Efficient storage utilization. Aggregate overcommit warnings at 85% usage.
高效利用儲存空間。Aggregate 使用率超過 85% 時發出 overcommit 警告。
iSCSI and FC SAN Support
iSCSI 及 FC SAN 支援
Choose transport protocol per storage. Both share the same multipath and WWID identification.
可依儲存選擇傳輸協定。兩者共用相同的 multipath 與 WWID 識別。
LXC Container Support
LXC 容器支援
Container rootfs on NetApp LUN. Configure with
--content images,rootdir.
容器 rootfs 可放在 NetApp LUN 上。以 --content images,rootdir 設定。
EFI, Cloud-init, TPM, vmstate
EFI、Cloud-init、TPM、vmstate
Full support for EFI disk, Cloud-init ISO, TPM 2.0 state, and VM RAM snapshot (vmstate).
完整支援 EFI 磁碟、Cloud-init ISO、TPM 2.0 state、以及 VM 記憶體快照 (vmstate)。
Requirements 系統需求
Proxmox VE
| PVE Version | Storage API | Compatibility 相容性 |
|---|---|---|
| PVE 9.1+ | 13 | Supported 支援 |
NetApp ONTAP
- ONTAP 9.8 or later (REST API required) ONTAP 9.8 或更新版本（需 REST API）
- iSCSI or FC license enabled 已啟用 iSCSI 或 FC 授權
- SVM with iSCSI/FC service enabled and at least one data LIF configured SVM 已啟用 iSCSI/FC 服務，且至少設定一個 data LIF
- Aggregate with available space Aggregate 有可用空間
- User account with REST API permissions for volumes, LUNs, igroups, snapshots, aggregates, and network interfaces 使用者帳號需有 REST API 權限，可管理 volumes、LUNs、igroups、snapshots、aggregates 及 network interfaces
Proxmox VE Node Dependencies Proxmox VE 節點相依套件
| Package 套件 | Purpose 用途 | Required 必要 |
|---|---|---|
open-iscsi | iSCSI initiator (iscsiadm) iSCSI initiator (iscsiadm) | Yes (for iSCSI) 是（iSCSI 模式） |
multipath-tools | Multipath I/O daemon (multipathd) Multipath I/O 背景服務 (multipathd) | Yes 是 |
sg3-utils | SCSI utilities (sg_inq) SCSI 工具 (sg_inq) | Yes 是 |
psmisc | Process utilities (fuser) for device-in-use detection 行程工具 (fuser)，偵測裝置使用中 | Yes 是 |
libwww-perl | HTTP client for REST API HTTP 客戶端，供 REST API 使用 | Yes 是 |
libjson-perl | JSON encoding/decoding JSON 編碼/解碼 | Yes 是 |
liburi-perl | URI handling URI 處理 | Yes 是 |
lsscsi | List SCSI devices (troubleshooting) 列出 SCSI 裝置（疑難排解用） | Recommended 建議 |
Installation 安裝
First-Time Installation 首次安裝
Important: Install dependencies BEFORE installing the plugin package to avoid dependency resolution issues.
重要：請在安裝 plugin 套件前先安裝相依套件，以避免相依性解析問題。
1
Update apt cache
更新 apt 快取
apt update
2
Install ALL dependencies
安裝所有相依套件
apt install -y open-iscsi multipath-tools sg3-utils psmisc \ libwww-perl libjson-perl liburi-perl lsscsi
3
Enable required services
啟用必要服務
systemctl enable --now iscsid systemctl enable --now multipathd
4
Install the plugin package
安裝 plugin 套件
dpkg -i jt-pve-storage-netapp_0.2.14-1_all.deb
The plugin automatically configures multipath and reloads Proxmox VE services (pvedaemon, pvestatd, pveproxy). Plugin 會自動設定 multipath 並重新載入 Proxmox VE 服務（pvedaemon、pvestatd、pveproxy）。
Fix Broken State 修復安裝失敗狀態
If you ran dpkg -i before installing dependencies and got errors:
如果在安裝相依套件前就執行 dpkg -i 而出現錯誤：
apt update apt --fix-broken install -y dpkg -l | grep jt-pve-storage-netapp
Cluster Installation 叢集安裝
CRITICAL: In a Proxmox VE cluster, this plugin MUST be installed on ALL nodes. Nodes without the plugin will show
Parameter verification failed. (400).
重要：在 Proxmox VE 叢集中，此 plugin 必須在所有節點上安裝。未安裝的節點會顯示 Parameter verification failed. (400)。
Repeat the 4 steps above on EACH node. Install the plugin on ALL nodes first, then add the storage configuration (only once, on any node). 在每個節點上重複上述 4 個步驟。先在所有節點安裝 plugin，再新增儲存設定（只需在任一節點操作一次）。
Upgrade SOP 升級標準作業程序
When upgrading, follow these steps on every cluster node in sequence (one node at a time): 升級時，請依序在每個叢集節點上執行以下步驟（一次一個節點）：
1
Pre-upgrade backup
升級前備份
# Backup multipath.conf cp /etc/multipath.conf /etc/multipath.conf.bak.$(date +%Y%m%d-%H%M%S) # Note current version dpkg -l jt-pve-storage-netapp | tail -1
2
Stop or migrate VMs (recommended)
停止或遷移 VM（建議）
For safest upgrade, migrate or stop VMs that have disks on this storage. Live VMs will continue to work during upgrade, but a clean state simplifies recovery if anything goes wrong. 最安全的做法是先將使用此儲存的 VM 遷移或停止。升級期間執行中的 VM 仍可繼續運作，但乾淨的狀態有助於出問題時的復原。
3
Install new package
安裝新套件
dpkg -i jt-pve-storage-netapp_0.2.14-1_all.deb
4
Review postinst warnings and verify
檢視 postinst 警告並驗證
# If warned about dangerous multipath settings, edit: nano /etc/multipath.conf # Apply: no_path_retry queue -> 30, remove queue_if_no_path, dev_loss_tmo infinity -> 60 systemctl restart multipathd # restart, NOT reload # Verify dpkg -l jt-pve-storage-netapp | grep ii pvesm status | grep netapp multipath -ll
5
Repeat on next node
在下一個節點重複
Move to the next cluster node and repeat from Step 1. Do not upgrade multiple nodes simultaneously. 移至下一個叢集節點，從步驟 1 重複。請勿同時升級多個節點。
Quick Start 快速入門
Add Storage (iSCSI) 新增儲存（iSCSI）
pvesm add netappontap netapp1 \ --ontap-portal 192.168.1.100 \ --ontap-svm svm0 \ --ontap-aggregate aggr1 \ --ontap-username pveadmin \ --ontap-password 'YourSecurePassword' \ --content images,rootdir \ --shared 1
Add Storage (FC / Fibre Channel) 新增儲存（FC / Fibre Channel）
pvesm add netappontap netapp-fc \ --ontap-portal 192.168.1.100 \ --ontap-svm svm0 \ --ontap-aggregate aggr1 \ --ontap-username pveadmin \ --ontap-password 'YourSecurePassword' \ --ontap-protocol fc \ --content images \ --shared 1
Verify 驗證
pvesm status # Name Type Status Total Used Available # netapp1 netappontap active ... ... ...
Note: This is a custom/third-party storage plugin. It is NOT listed in the Web UI "Add Storage" dropdown -- storage must be added via CLI (
pvesm add). After adding, the storage will appear in the Web UI storage list and support all VM operations normally.
注意：這是第三方自訂儲存 plugin，不會出現在 Web UI 的「新增儲存」下拉選單中 -- 必須透過 CLI (pvesm add) 新增。新增後，儲存會出現在 Web UI 的儲存清單中，並正常支援所有 VM 操作。
Configuration Reference 設定選項參考
Required Options 必要選項
| Option 選項 | Description 說明 | Example 範例 |
|---|---|---|
ontap-portal | ONTAP management IP or hostname ONTAP 管理 IP 或主機名稱 | 192.168.1.100 |
ontap-svm | Storage Virtual Machine (SVM/Vserver) name Storage Virtual Machine (SVM/Vserver) 名稱 | svm0 |
ontap-aggregate | Aggregate for volume creation 建立 volume 用的 aggregate | aggr1 |
ontap-username | ONTAP API username ONTAP API 使用者名稱 | pveadmin |
ontap-password | ONTAP API password ONTAP API 密碼 | YourSecurePassword |
Optional Options 可選選項
| Option 選項 | Default 預設值 | Description 說明 |
|---|---|---|
ontap-protocol | iscsi |
SAN protocol: iscsi or fc (Fibre Channel)
SAN 協定：iscsi 或 fc (Fibre Channel)
|
ontap-ssl-verify | 1 | Verify SSL certificates (0 = disable for self-signed) 驗證 SSL 憑證（0 = 停用，適用自簽憑證） |
ontap-thin | 1 | Use thin provisioning (0 = thick provisioning) 使用精簡配置（0 = 完整配置） |
ontap-igroup-mode | per-node |
igroup mode: per-node (recommended) or shared
igroup 模式：per-node（建議）或 shared
|
ontap-cluster-name | pve | Cluster name for igroup naming. Use different values for multiple storages on the same SVM. igroup 命名用的叢集名稱。同一 SVM 上有多個儲存時請使用不同值。 |
ontap-device-timeout | 60 | Device discovery timeout in seconds 裝置探索逾時秒數 |
Proxmox VE Standard Storage Options Proxmox VE 標準儲存選項
These are standard Proxmox VE storage options that apply to all storage types, including this plugin. 這些是 Proxmox VE 標準儲存選項，適用於所有儲存類型，包含本外掛程式。
| Option 選項 | Default 預設值 | Description 說明 |
|---|---|---|
content | images |
Content types this storage can hold. Use images for VM disks only, or images,rootdir to also support LXC containers. Other valid values: iso, vztmpl, backup (not supported by this plugin).
此儲存可存放的內容類型。使用 images 僅支援 VM 磁碟，或使用 images,rootdir 同時支援 LXC 容器。其他值：iso、vztmpl、backup（本外掛不支援）。
|
shared | 0 |
Mark storage as shared across cluster nodes. Must set to 1 for this plugin -- required for live migration. Without this, Proxmox VE will try to copy disk data during migration instead of using the shared SAN path.
標記儲存為叢集節點間共享。本外掛必須設為 1 -- 即時遷移所需。若未設定，Proxmox VE 會在遷移時嘗試複製磁碟資料，而非使用共享 SAN 路徑。
|
nodes | (all) （全部） |
Restrict storage to specific nodes. Comma-separated list of node names. Example: --nodes pve1,pve2,pve3. Omit to allow all nodes.
限制儲存僅供特定節點使用。以逗號分隔的節點名稱清單。範例：--nodes pve1,pve2,pve3。省略表示允許所有節點。
|
disable | 0 |
Disable this storage. Set to 1 to temporarily deactivate without removing the configuration.
停用此儲存。設為 1 可暫時停用而不刪除設定。
|
Tip: For LXC container support, you must include
rootdir in the content type: --content images,rootdir. Without it, LXC creation on this storage will fail.
提示：若要支援 LXC 容器，必須在 content 類型中加入 rootdir：--content images,rootdir。未設定時，在此儲存上建立 LXC 會失敗。
Example storage.cfg (iSCSI) 範例 storage.cfg (iSCSI)
netappontap: netapp1
ontap-portal 192.168.1.100
ontap-svm svm0
ontap-aggregate aggr1
ontap-username pveadmin
ontap-password YourSecurePassword
ontap-protocol iscsi
ontap-thin 1
ontap-igroup-mode per-node
content images,rootdir
shared 1
Example storage.cfg (FC SAN) 範例 storage.cfg (FC SAN)
netappontap: netapp-fc
ontap-portal 192.168.1.100
ontap-svm svm0
ontap-aggregate aggr1
ontap-username pveadmin
ontap-password YourSecurePassword
ontap-protocol fc
ontap-thin 1
ontap-igroup-mode per-node
content images,rootdir
shared 1
Note: For FC,
ontap-portal is still required for ONTAP REST API access. The FC data path uses the FC fabric, not the management IP.
注意：FC 模式下仍需 ontap-portal 用於 ONTAP REST API 存取。FC 資料路徑使用 FC fabric，不是管理 IP。
Architecture 系統架構
1:1:1 Architecture Model 1:1:1 架構模型
Each VM disk maps to exactly one ONTAP FlexVol containing exactly one LUN, providing clean snapshot semantics. 每個 VM 磁碟對應到一個 ONTAP FlexVol，內含一個 LUN，提供乾淨的快照語意。
Object Mapping 物件對應
| Proxmox VE Object Proxmox VE 物件 | ONTAP Object ONTAP 物件 | Naming Pattern 命名規則 |
|---|---|---|
| Storage | -- |
User defined (e.g., netapp1)
使用者定義（例如 netapp1）
|
| VM Disk | FlexVol | pve_{storage}_{vmid}_disk{id} |
| VM Disk | LUN | /vol/{flexvol}/lun0 |
| Snapshot | Volume Snapshot | pve_snap_{snapname} |
| Proxmox VE Node | igroup | pve_{cluster}_{node} |
| Cloud-init | FlexVol | pve_{storage}_{vmid}_cloudinit |
| VM State | FlexVol | pve_{storage}_{vmid}_state_{snap} |
Module Architecture 模組架構
NetAppONTAPPlugin.pm
Main plugin -- Proxmox VE Storage Plugin API implementation. Volumes, snapshots, clones.
主 plugin -- Proxmox VE Storage Plugin API 實作。Volumes、快照、複製。
API.pm
ONTAP REST API client. Volumes, LUNs, igroups, snapshots, capacity.
ONTAP REST API 客戶端。Volumes、LUNs、igroups、快照、容量。
ISCSI.pm
iSCSI management. Target discovery, login, session handling, device wait.
iSCSI 管理。Target 探索、登入、session 處理、裝置等待。
Multipath.pm
Multipath and SCSI device management. Device discovery, cleanup, anti-hang protection.
Multipath 與 SCSI 裝置管理。裝置探索、清理、防卡死保護。
FC.pm
Fibre Channel HBA management. WWPN discovery, LIP rescan.
Fibre Channel HBA 管理。WWPN 探索、LIP rescan。
Naming.pm
Proxmox VE to ONTAP naming conventions. Volume names, LUN paths, igroup names.
Proxmox VE 至 ONTAP 命名規則。Volume 名稱、LUN 路徑、igroup 名稱。
Multipath Safety Rules Multipath 安全規則
READ THIS BEFORE INSTALLING. These rules prevent Proxmox VE node hangs and accidental disconnection of other storage.
安裝前請務必閱讀。這些規則防止 PVE 節點當機及意外斷開其他儲存。
Rule 1: NEVER use multipath -F (capital F) 規則 1：絕對不要使用 multipath -F（大寫 F）
multipath -F flushes ALL unused multipath maps system-wide. If you have other storage (manual iSCSI LVM, other vendors, etc.) and there is no active I/O on it at the moment, it will be disconnected. Always use targeted flushing:
multipath -F 會清除系統上所有未使用的 multipath map。如果您有其他儲存（手動 iSCSI LVM、其他廠商等）且當時沒有 active I/O，該儲存會被斷開。請使用指定目標的清除：
# Identify stale WWIDs (look for "failed faulty" in all paths) multipath -ll # Flush ONE specific stale WWID (lowercase f) multipath -f 3600a09807770457a795d5a7653705853
Rule 2: After editing multipath.conf, use restart not reload 規則 2：編輯 multipath.conf 後，使用 restart 而非 reload
# CORRECT - applies new settings AND flushes stale state systemctl restart multipathd # WRONG - only re-reads config, leaves stale maps in place systemctl reload multipathd
Rule 3: Check your multipath.conf settings 規則 3：檢查您的 multipath.conf 設定
If your config contains any of these, the entire PVE node can hang when a LUN is deleted or becomes unavailable: 如果您的設定包含以下任何一項，當 LUN 被刪除或變得不可用時，整個 PVE 節點可能會當機：
| Setting 設定 | Risk 風險 | Fix 修正 |
|---|---|---|
no_path_retry queue |
I/O queues forever I/O 永久排隊 |
Change to no_path_retry 30
改為 no_path_retry 30
|
queue_if_no_path |
Same as above 同上 |
Remove from features line
從 features 行移除
|
dev_loss_tmo infinity |
Stale devices never removed 殘留裝置永遠不會被移除 |
Change to dev_loss_tmo 60
改為 dev_loss_tmo 60
|
Rule 4: v0.2.2+ handles cleanup automatically 規則 4：v0.2.2+ 自動處理殘留清理
You do not need to manually clean stale devices after upgrading to v0.2.2+. The plugin automatically detects and removes its own orphan devices in the background during normal storage status polling. It only touches WWIDs it created and never affects other storage. 升級至 v0.2.2+ 後不需要手動清理殘留裝置。Plugin 會在正常儲存狀態輪詢時，於背景自動偵測並移除自己產生的殘留裝置。只會處理自己建立的 WWID，不會影響其他儲存。
Supported Features Matrix 功能支援表
| Feature 功能 | Status 狀態 | Notes 備註 |
|---|---|---|
| Disk create/delete 磁碟建立/刪除 | Supported 支援 | FlexVol + LUN |
| Disk resize 磁碟調整大小 | Supported 支援 | Online resize supported 支援線上調整 |
| Snapshots 快照 | Supported 支援 | ONTAP Volume Snapshots |
| Snapshot rollback 快照倒回 | Supported 支援 | VM must be stopped VM 必須先停止 |
| Live migration 即時遷移 | Supported 支援 | Via shared iSCSI/FC access 透過共享 iSCSI/FC 存取 |
| Thin provisioning 精簡配置 | Supported 支援 | Default enabled 預設啟用 |
| Multipath I/O | Supported 支援 | Automatic configuration 自動設定 |
| Template | Supported 支援 | Convert VM to template 將 VM 轉為範本 |
| Linked Clone | Supported 支援 | Via NetApp FlexClone (instant, space-efficient) 透過 NetApp FlexClone（即時、節省空間） |
| Full Clone | Supported 支援 | Via qemu-img copy 透過 qemu-img 複製 |
| Full Clone from Snapshot 從快照完整複製 | Supported 支援 | Via temporary FlexClone + qemu-img 透過暫時 FlexClone + qemu-img |
| Backup (vzdump) | Supported 支援 | Via snapshot 透過快照 |
| RAM Snapshot (vmstate) | Supported 支援 | v0.1.7+ |
| LXC Container (rootdir) | Supported 支援 | v0.2.0+ |
| EFI Disk | Supported 支援 | v0.2.0+ |
| Cloud-init Disk | Supported 支援 | v0.2.0+ |
| TPM State | Supported 支援 | v0.2.0+ |
Troubleshooting 疑難排解
Common issues and their solutions. For a comprehensive guide, see docs/TROUBLESHOOTING.md in the repository. 常見問題及解決方案。完整指南請參閱 repository 中的 docs/TROUBLESHOOTING_zh-TW.md。
Quick Diagnostic Commands 快速診斷指令
# Check storage status pvesm status # Check PVE daemon logs journalctl -xeu pvedaemon --since "10 minutes ago" # Check iSCSI sessions iscsiadm -m session # Check multipath devices multipathd show maps # Check ONTAP API connectivity curl -k -u username:password https://ONTAP_IP/api/cluster
Storage Not Active 儲存未啟用
Symptoms: Storage shows "inactive" in pvesm status. Cannot create VMs on storage.
症狀：在 pvesm status 中顯示 "inactive"。無法在該儲存上建立 VM。
Common Causes: 常見原因：
- Invalid credentials -- test with:
curl -k -u pveadmin:password https://192.168.1.100/api/cluster認證無效 -- 測試指令：curl -k -u pveadmin:password https://192.168.1.100/api/cluster - Network connectivity -- check:
ping <ontap-portal>andnc -zv <ontap-portal> 443網路連線問題 -- 檢查：ping <ontap-portal>及nc -zv <ontap-portal> 443 - SSL certificate issues -- temporarily disable:
pvesm set <storage-id> --ontap-ssl-verify 0SSL 憑證問題 -- 暫時停用：pvesm set <storage-id> --ontap-ssl-verify 0 - SVM not accessible or iSCSI service not enabled on the SVMSVM 無法存取或 SVM 上的 iSCSI 服務未啟用
Device Not Found After Create 建立後找不到裝置
Symptoms: Disk created successfully but device not appearing in /dev/.
症狀：磁碟建立成功但裝置未出現在 /dev/ 中。
Solutions: 解決方案：
# Rescan iSCSI sessions iscsiadm -m session --rescan # Reload multipath multipathd reconfigure multipath -v2
igroup Issues igroup 問題
Symptoms: LUN mapped but node cannot see the device. iscsiadm -m session shows active sessions but lsscsi shows no NETAPP devices.
症狀：LUN 已 map 但節點看不到裝置。iscsiadm -m session 顯示 active session 但 lsscsi 沒有 NETAPP 裝置。
Verify the node's iSCSI initiator IQN (from /etc/iscsi/initiatorname.iscsi) is listed in the correct igroup on ONTAP. Check with ONTAP CLI: igroup show -vserver <svm>.
確認節點的 iSCSI initiator IQN（來自 /etc/iscsi/initiatorname.iscsi）已列在 ONTAP 上正確的 igroup 中。ONTAP CLI 檢查：igroup show -vserver <svm>。
Hung Kernel Tasks (vgs blocked, D-state processes) Kernel 卡住（vgs 阻塞、D-state 行程）
Symptoms: vgs or other commands hang. ps aux shows processes in D state. PVE operations time out.
症狀：vgs 或其他指令卡住。ps aux 顯示 D state 的行程。PVE 操作逾時。
Root cause: Usually a stale multipath device with queue_if_no_path or no_path_retry queue. Any process that touches the device enters uninterruptible sleep (D-state).
根本原因：通常是設定了 queue_if_no_path 或 no_path_retry queue 的殘留 multipath 裝置。任何碰觸該裝置的行程都會進入不可中斷睡眠（D-state）。
Manual cleanup for stale devices with queue_if_no_path: 有 queue_if_no_path 的殘留裝置手動清理：
# 1. Disable queueing multipathd disablequeueing map <wwid> dmsetup message <wwid> 0 fail_if_no_path # 2. Flush the specific device multipath -f <wwid> # 3. If step 2 fails, force remove dmsetup remove --force --retry <wwid>
Cannot Delete Volume (LVM holders / "device is still in use") 無法刪除 Volume（LVM holders / 「裝置仍在使用中」）
Symptoms: Cannot delete volume: device is still in use (has holders). Common on PVE nodes upgraded from 7 to 8 to 9.
症狀：Cannot delete volume: device is still in use (has holders)。常見於從 PVE 7 升級至 8 再到 9 的節點。
Root cause: The host's LVM scanner auto-activated VGs found INSIDE VM disks (guest OS LVM). This happens when /etc/lvm/lvm.conf has no global_filter to exclude plugin multipath devices.
根本原因：主機的 LVM 掃描器自動 activate 了 VM 磁碟裡面的 VG（客體 OS 的 LVM）。這在 /etc/lvm/lvm.conf 沒有 global_filter 排除 plugin multipath 裝置時會發生。
Fix: 修復：
# Deactivate the guest VG on the host vgchange -an <guest-vg-name> # Long-term fix: add global_filter to /etc/lvm/lvm.conf # global_filter = [ "r|/dev/mapper/3.*|" ]
Common Error Messages 常見錯誤訊息
| Error 錯誤 | Cause / Fix 原因 / 修復 |
|---|---|
unknown storage type 'netappontap' | Plugin not loaded. Reinstall and restart pvedaemon. Plugin 未載入。重新安裝並重啟 pvedaemon。 |
Failed to map LUN | ASA eventual consistency (v0.2.9 fixes with retry). Or igroup permissions issue. ASA 最終一致性問題（v0.2.9 已加入重試修復）。或 igroup 權限問題。 |
Cannot grow device files | Kernel did not see new LUN size. Fixed in v0.2.3+ (per-device rescan). Kernel 未偵測到新的 LUN 大小。v0.2.3+ 已修復（per-device rescan）。 |
trying to acquire lock... got timeout | D-state child blocking kernel lock. See "Hung Kernel Tasks" above. Upgrade to v0.2.5+. D-state child 佔住 kernel lock。見上方「Kernel 卡住」。請升級至 v0.2.5+。 |
sysfs write ... timed out | Writing to non-iSCSI SCSI host. Fixed in v0.2.5+. 對非 iSCSI SCSI host 寫入。v0.2.5+ 已修復。 |
Changelog 變更紀錄
Version history. For full details see CHANGELOG.md. 版本紀錄。完整內容請參閱 CHANGELOG_zh-TW.md。
v0.2.14 Fix
2026-05-14
Temp Clone Host-side Cleanup (production regression in v0.2.13, found day 1). v0.2.13 fixed snapshot delete on ONTAP side but left host's dm-multipath + sd* paths orphaned.
multipathd spammed "tur checker reports path is down" indefinitely after every backup. New shared helper _remove_temp_clone() mirrors free_image()'s 7-step pattern (capture slaves -> unmap -> cleanup_lun_devices -> remove sd* -> multipath_reload -> split -> wait -> delete). Both volume_snapshot_delete and _cleanup_temp_clones route through it. Section 24 hardened with mandatory host-side device residual assertions.
Temp Clone Host 端清理修正(v0.2.13 deploy 後一天客戶現場發現的 regression)。v0.2.13 修了 ONTAP 端的 snapshot 刪除,但 host 端 dm-multipath + sd* 路徑沒清。multipathd 在每次備份後持續洗版「tur checker reports path is down」。新增共用 helper _remove_temp_clone(),流程對齊 free_image() 的 7 步模式(抓 slave → unmap → cleanup_lun_devices → 移除 sd* → multipath_reload → split → wait → delete)。volume_snapshot_delete 和 _cleanup_temp_clones 兩個 call site 都統一走 helper。Section 24 強化加上必須的 host 端 device 殘留斷言。
v0.2.13 Fix
2026-05-13
Snapshot Delete Cleanup Fix (production incident). vzdump CT snapshot-mode backups previously failed at the cleanup step with "Snapshot ... has not expired or is locked" because the temp FlexClone created for snapshot read-access was still holding the parent snapshot.
volume_snapshot_delete() now detaches the dependent temp clone via volume_clone_split + wait + delete BEFORE deleting the snapshot, ensuring the owner reference is released on every ONTAP platform (real FAS and simulator behaved differently). Local in-use safety check via is_device_in_use prevents tearing down a temp clone that's still being read.
Snapshot 刪除清理修正(正式環境事件)。vzdump 對 CT 做 snapshot-mode 備份備份本身成功,但清理 snapshot 必失敗訊息「has not expired or is locked」— 因為讓 PVE 讀 snapshot 而建立的暫時 FlexClone 還在持有 parent snapshot。volume_snapshot_delete() 現在會在刪 snapshot 之前先透過 volume_clone_split + wait + delete 解除暫時 clone,確保 owner reference 在所有 ONTAP 平台都釋放(實機 FAS 與 simulator 行為不同)。透過 is_device_in_use 做本機 in-use 安全檢查,避免拆掉正在被讀取的 temp clone。
v0.2.12 Fix
2026-05-05
iSCSI Portal TCP Pre-check.
activate_storage() now TCP-probes every iSCSI LIF before invoking iscsiadm. Pre-fix behaviour stalled 30s (discovery) + 60s (login) per unreachable LIF, cascading via pvestatd polls into web UI hangs. New probe_portal() helper uses bounded IO::Socket::INET connect. New option ontap-portal-probe-timeout (0..30, default 2). Sibling-pattern audit from jt-pve-storage-purestorage v1.1.9.
iSCSI Portal TCP 預先檢查。activate_storage() 現在會在呼叫 iscsiadm 之前先用 TCP 確認 LIF 是否可達。修正前每個不通的 LIF 會吃 30 秒（discovery）+ 60 秒（login），透過 pvestatd 輪詢連鎖造成 web UI 凍結。新增 probe_portal() helper 用帶上限的 IO::Socket::INET 連線。新選項 ontap-portal-probe-timeout（0..30,預設 2）。修正來源:姊妹專案 jt-pve-storage-purestorage v1.1.9 的同類型稽核。
v0.2.11 Fix
2026-04-30
SAN LIF Redundancy Detection Fix. After NetApp clarification: SAN (iSCSI/FC) LIFs do NOT auto-migrate during takeover -- only NAS LIFs do. Path failover relies on host MPIO + ALUA. Plugin now detects "all LIFs share same home_node" misconfiguration (single controller failure risk). New
iscsi_get_lifs_with_home_node() API returns LIF metadata for HA validation. Documentation corrected.
SAN LIF 冗餘偵測修正。NetApp 原廠確認後：SAN (iSCSI/FC) LIF 在 takeover 時不會自動遷移，只有 NAS LIF 會。路徑切換靠 host MPIO + ALUA。外掛現在會偵測「所有 LIF 都在同一 home_node」的設定錯誤（單一 controller 故障即全斷風險）。新增 iscsi_get_lifs_with_home_node() API 回傳 LIF metadata 供 HA 驗證。文件已修正。
v0.2.10 Feature
2026-04-30
Disaster Prevention & Monitoring. New syslog alerting: storage outage ERROR after 30s+ failure (with 60s re-alert cooldown), recovery INFO message, aggregate capacity WARNING at 90%/CRITICAL at 95%, LIF redundancy WARNING when SVM has <2 iSCSI LIFs. postinst now detects in-flight storage operations and warns with 5-second grace. Documentation: 4 new SOPs (disconnect recovery, power loss recovery, password change, ONTAP HA best practices) and "Will upgrading affect running VMs?" Q&A.
災難預防與監控。新增 syslog 警示：儲存中斷超過 30 秒觸發 ERROR（60 秒冷卻避免洪水）、恢復連線發送 INFO、Aggregate 容量 90% WARNING / 95% CRITICAL、SVM iSCSI LIF 少於 2 個發送 WARNING。postinst 偵測進行中儲存操作並警告 5 秒緩衝。文件新增 4 個 SOP（斷線恢復、斷電恢復、密碼變更、ONTAP HA 最佳實踐）以及「升級會影響執行中 VM 嗎？」Q&A。
v0.2.9 Fix
2026-04-25
ASA Eventual Consistency Fix. Fixed
lun_map() failing with "LUN not found" on NetApp ASA systems due to ONTAP internal propagation delay after lun_create(). Now retries UUID lookup up to 5 times with 1-second intervals.
ASA 最終一致性修復。修復 lun_map() 在 NetApp ASA 系統上因 lun_create() 後 ONTAP 內部傳播延遲而回報 "LUN not found"。現在會重試 UUID 查詢最多 5 次，每次間隔 1 秒。
v0.2.8 Fix
2026-04-11
Code Review Fix Release. Fixed orphan cleanup unconditionally untracking WWIDs. Fixed
alloc_image() TOCTOU race retry (now bounded loop, max 5). Removed all multipath -F recommendations. Fixed bare glob() without alarm timeout.
程式碼審查修復 Release。修復殘留清理無條件 untrack WWID。修復 alloc_image() TOCTOU race retry（改為有界迴圈，最多 5 次）。移除所有 multipath -F 建議。修復 bare glob() 缺少 alarm timeout。
v0.2.7 Critical
2026-04-11
kpartx Partition Holder Fix. Fixed
is_device_in_use() blocking ALL volume deletions on systems with kpartx partition scanning. Added kpartx -d cleanup step.
kpartx Partition Holder 修復。修復 is_device_in_use() 在有 kpartx partition 掃描的系統上擋住所有 volume 刪除。新增 kpartx -d 清理步驟。
v0.2.6 Fix
2026-04-10
Postinst Service Reload + Operator UX. Added pvestatd to service reload list. Changed to
systemctl reload (SIGHUP) to avoid D-state stop-phase hang. Added lvm.conf global_filter detection. Detailed is_device_in_use error messages with LVM VG names and fix commands.
Postinst 服務 Reload + Operator UX。新增 pvestatd 到服務 reload 清單。改用 systemctl reload (SIGHUP) 避免 D-state stop 階段卡住。新增 lvm.conf global_filter 偵測。詳細的 is_device_in_use 錯誤訊息，含 LVM VG 名稱與修復指令。
v0.2.5 Critical
2026-04-10
Non-iSCSI SCSI Host Scan Fix (production incident on HPE ProLiant). Fixed
rescan_scsi_hosts() writing to non-iSCSI hosts (smartpqi, USB, etc.), causing D-state cascades and node hangs. Now sources host list from /sys/class/iscsi_host/.
非 iSCSI SCSI Host 掃描修復（HPE ProLiant 正式環境事件）。修復 rescan_scsi_hosts() 對非 iSCSI host（smartpqi、USB 等）寫入，導致 D-state 連鎖與節點當機。改從 /sys/class/iscsi_host/ 取得 host 清單。
Older versions (v0.2.4 -- v0.1.0) 更早版本（v0.2.4 -- v0.1.0）
v0.2.4 Fix
2026-04-09
Cleanup Path Hardening + Concurrency + Operator UX. Fixed
clone_image() TOCTOU race and missing lun_unmap_all() in cleanup. Added _translate_limit_error() for operator-friendly ONTAP error messages. Removed dead code get_multipath_wwid(). Pre-snapshot buffer flush.
Cleanup 路徑強化 + 並行 + Operator UX。修復 clone_image() TOCTOU race 及 cleanup 缺少 lun_unmap_all()。新增 _translate_limit_error() 提供 operator 友善的 ONTAP 錯誤訊息。移除無用程式碼 get_multipath_wwid()。Snapshot 前 buffer flush。
v0.2.3 Critical
2026-04-09
Pre-Upgrade Stale Device Handling. Auto-imports ONTAP WWIDs on every status() poll. Multipath hang prevention (disablequeueing, dmsetup fallback). Symlink resolution fix (DATA LOSS prevention). Per-device rescan for resize and rollback. Extended 60s timeout for volume_delete.
升級前殘留裝置處理。每次 status() 輪詢自動匯入 ONTAP WWID。Multipath 卡住預防（disablequeueing、dmsetup fallback）。Symlink 解析修復（防止資料遺失）。Resize 及 rollback 改用 per-device rescan。volume_delete 延長至 60 秒 timeout。
v0.2.2 Feature
2026-04-08
Cluster Orphan Device Cleanup. Automatic WWID tracking and orphan device cleanup. Concurrency fixes (flock, atomic write, double-fork). 63 tests pass.
叢集殘留裝置清理。自動 WWID 追蹤與殘留裝置清理。並行修復（flock、atomic write、double-fork）。63 項測試通過。
v0.2.1 Fix
2026-04-08
Production Hardening. TOCTOU race fixes, multipath safety (no_path_retry 30, dev_loss_tmo 60), stale device prevention, migration safety, API resilience (401 retry).
正式環境強化。TOCTOU race 修復、multipath 安全（no_path_retry 30、dev_loss_tmo 60）、殘留裝置防護、遷移安全、API 韌性（401 重試）。
v0.2.0 Feature
2026-04-07
Anti-Hang Protection. All sysfs writes/reads with timeout. Fixed iSCSI multipath single-session bug. LXC container, EFI disk, Cloud-init, TPM support.
防卡死保護。所有 sysfs 寫入/讀取加上 timeout。修復 iSCSI multipath 只建立單一 session。LXC 容器、EFI disk、Cloud-init、TPM 支援。
v0.1.9 Fix
2026-02-27
Safety Audit. Fixed command injection vulnerability. Fixed IPC::Open3 deadlock and zombie processes. Snapshot rollback buffer flush. Online resize support.
安全稽核。修復命令注入漏洞。修復 IPC::Open3 deadlock 與 zombie process。Snapshot rollback buffer flush。線上 resize 支援。
v0.1.8 Fix
2026-02-12
FC SAN bug fixes. Fixed is_fc_available(), added lun_unmap_all(), fixed deactivate_storage parameters.
FC SAN bug 修復。修復 is_fc_available()，新增 lun_unmap_all()，修復 deactivate_storage 參數。
v0.1.7 Feature
2026-01-25
RAM Snapshot (vmstate) support. Automatic multipath configuration on install. License changed to MIT.
RAM 快照（vmstate）支援。安裝時自動設定 multipath。授權變更為 MIT。
v0.1.6 Feature
2026-01-24
Full Clone support. Full Clone from VM Snapshot via temporary FlexClone. Automatic temp FlexClone cleanup.
Full Clone 支援。透過暫時 FlexClone 從 VM 快照完整複製。暫時 FlexClone 自動清理。
v0.1.5 Feature
2026-01-03
Template support. Create base, rename volume. Graceful handling of missing LUNs.
範本支援。Create base、rename volume。優雅處理遺失的 LUN。
v0.1.4 Feature
2026-01-03
FC SAN protocol support. New FC.pm module. Batch LUN query. Configurable device timeout.
FC SAN 協定支援。新增 FC.pm 模組。批次 LUN 查詢。可設定裝置 timeout。
v0.1.3 Feature
2026-01-03
FlexClone support (Linked Clone). Template deletion protection. Volume autogrow enabled.
FlexClone 支援（Linked Clone）。範本刪除保護。啟用 volume autogrow。
v0.1.2
2026-01-02
Bug fix and dependency release. Volume autogrow, psmisc dependency.
Bug 修復與相依套件 release。Volume autogrow、psmisc 相依套件。
v0.1.1
2026-01-02
Safety improvements. Shrink protection, collision detection, API cache TTL, taint mode fix.
安全性改進。縮小保護、碰撞偵測、API 快取 TTL、taint mode 修復。
v0.1.0 Feature
2026-01-02
Initial Release. FlexVol/LUN creation, igroup management, iSCSI discovery/login, multipath, snapshots, real-time storage status.
初始 Release。FlexVol/LUN 建立、igroup 管理、iSCSI 探索/登入、multipath、快照、即時儲存狀態。
Acknowledgments 致謝
Special thanks to NetApp for generously providing the development and testing environment that made this project possible. 特別感謝 NetApp 慷慨提供開發測試環境，使本專案得以順利完成。