jt-pve-storage-dellemc

Disclaimer & Risk 免責聲明與風險

1.0.0 is the on-hardware test pass, not more code. What has and has not been verified is tracked in docs/TESTING.md. 1.0.0 的門檻是實機測試通過，而不是再寫更多程式。哪些已驗證、哪些還沒，記錄在 docs/TESTING_zh-TW.md。

Product Families 產品系列

Dell EMC's product lines differ too much to share one PVE storage type, so each family gets its own. They share the host-side layer, so adding a family is a plugin file and an API client rather than a restructuring. Dell EMC 各產品線的差異太大，無法共用同一個 PVE storage type，因此每個系列各自對應一個。它們共用主機端底層，所以新增一個系列只需要一個 plugin 檔加一個 API 客戶端，不必重構。

Why not one plugin with a type option: PVE asks a plugin for its content types and disk formats before any configuration is parsed, so one type cannot describe families that differ in what they can hold. The type string is also a permanent contract — changing it later invalidates every existing storage.cfg. 為什麼不做成單一 plugin 加型號參數：PVE 會在解析任何設定之前就向 plugin 詢問支援的 content type 與磁碟格式，因此單一 type 無法描述「能放的東西不同」的系列。而且 type 字串一旦公開就不能再改 —— 日後修改會讓所有既有的 storage.cfg 失效。

Features 功能特性

Which Proxmox VE operations work 支援的 Proxmox VE 操作

The right-hand column says where each row has actually been driven. “Code” means it is implemented and covered by tests, and no array has run it — which is most of this table, and the reason this is beta software. 最右欄寫的是每一列實際跑過的地方。「僅程式」表示已實作、有測試涵蓋，但沒有任何儲存伺服器跑過 — 本表大多數列都是如此，這也是本軟體仍為 beta 的原因。

Why this plugin has to be verified on hardware 為什麼這個外掛需要實機驗證

Dell publishes documentation for all four families, and it is not enough. The documentation site refuses the pages that carry the collection-query syntax; a printed column heading is not a property name; and two firmware generations of the same array document the same command with its arguments in a different order. So every array-facing fact here was cross-checked against Dell’s own client code — python-powerstore, python-powerflex, gounity, the ansible collections — and where those disagreed with the prose, the code won. Dell 四個系列都有公開文件，而那並不足夠。文件網站會拒絕提供帶有集合查詢語法的頁面；指令印出的欄位標題不等於屬性名稱；同一台儲存伺服器的兩個韌體世代，還會把同一個指令的參數順序記載成不同的樣子。所以這裡每一項面向儲存伺服器的認知，都與 Dell 自己的客戶端程式碼交叉比對過 — python-powerstore、python-powerflex、gounity、ansible collections — 兩者不一致時，以程式碼為準。

It still is not enough. The first run on a real ME4024 found three defects in a row, each hidden by the one before it, and none of them reachable by reading. The first run on a real PowerStore found four more, starting with a WWPN written without colons. That is what this table is for: a row that says “Code” means nobody has been able to be wrong about it yet. 而這樣仍然不夠。在真正的 ME4024 上第一次執行，一口氣找出三個缺陷，每一個都被前一個遮住，而且沒有一個是用讀的能找到的。在真正的 PowerStore 上第一次執行，又找出四個，第一個是沒有冒號的 WWPN。上面那張表就是為此存在的：標示「僅程式」的那一列，代表還沒有人有機會在它上面犯錯。

Where nothing could be established, this plugin refuses the operation rather than guessing at it. A refusal that names what it could not establish is recoverable; a guess that deletes the wrong volume is not. 凡是無法確立的地方，這個外掛會拒絕該操作，而不是用猜的。一個說得出「什麼無法確立」的拒絕是可以挽回的；一個刪錯磁碟區的猜測不是。

How to help this project 如何協助這個專案

What this project needs is not code. It is time on other people’s arrays: one PowerVault ME4024 over Fibre Channel has run it, and one PowerStore is running it now. PowerFlex and Unity XT have never been connected to anything. 這個專案缺的不是程式，是在別人儲存伺服器上的時間：目前有一台走 Fibre Channel 的 PowerVault ME4024 跑過，另有一台 PowerStore 正在跑。PowerFlex 與 Unity XT 從來沒有連上過任何東西。

• Run it against a non-production array. docs/FIRST_RUN.md is nine steps and stops before anything is created if the array does not answer as expected. 拿一台非正式環境的儲存伺服器跑跑看。docs/FIRST_RUN.md 有九個步驟，而且只要儲存伺服器的回應不如預期，它會在建立任何東西之前停下來。
• Report what broke. The useful details are the model and firmware, the protocol, what you were doing, and the error exactly as it appeared — including the array’s own message, which is usually the part that names the real cause. 回報壞在哪裡。有用的細節是機型與韌體、使用的協定、當時在做什麼，以及錯誤訊息原本的樣子 — 包含儲存伺服器自己那一段，通常真正的原因就寫在那裡。
• Three things only an array can answer, and they take minutes: sg_inq /dev/sdX on a mapped volume, whether the WWID this plugin reports matches multipath -ll, and whether a qm rollback leaves the expected snapshot on the array. 有三件事只有儲存伺服器答得出來，而且幾分鐘就好：對已對應的磁碟區跑 sg_inq /dev/sdX、外掛回報的 WWID 是否與 multipath -ll 一致，以及 qm rollback 之後儲存伺服器上是否留下預期的快照。

Issues and reports: github.com/jasoncheng7115/jt-pve-storage-dellemc/issues 問題回報：github.com/jasoncheng7115/jt-pve-storage-dellemc/issues

Requirements 系統需求

Proxmox VE

Arrays 儲存伺服器

Node packages 節點套件

Installation 安裝

# always the newest build; this URL never has to be edited
curl -LO https://github.com/jasoncheng7115/jt-pve-storage-dellemc/releases/latest/download/jt-pve-storage-dellemc_all.deb
curl -LO https://github.com/jasoncheng7115/jt-pve-storage-dellemc/releases/latest/download/SHA256SUMS
sha256sum -c SHA256SUMS --ignore-missing # must say OK before installing

# which version did that fetch?
dpkg-deb -f jt-pve-storage-dellemc_all.deb Version

apt install ./jt-pve-storage-dellemc_all.deb

systemctl restart pvestatd

Building from source 從原始碼建置

Only needed to work on the plugin, or to run the test suite against your own PVE version. For installing it, use the release package above — that is the build the release was tested with. 只有在要修改這個外掛、或想針對你自己的 PVE 版本跑測試套件時才需要。單純安裝請用上面的發行版套件 — 那才是該版本測試時所用的建置。

git clone https://github.com/jasoncheng7115/jt-pve-storage-dellemc.git
cd jt-pve-storage-dellemc
make test    # syntax checks, unit tests, safety guard
make deb

Upgrading 升級

Volumes created by an older version keep working, and nothing needs doing. Every name the plugin creates begins with a prefix, which is pve unless a storage sets dell-name-prefix — and that is the literal the plugin used before the option existed. A storage that has never set it produces byte-identical names after an upgrade, so every volume already on the array stays recognised. 由舊版建立的磁碟區會繼續正常運作，而且什麼都不用做。外掛建立的每一個名稱都以一個前置字串開頭，除非該 storage 設定了 dell-name-prefix，否則就是 pve — 而那正是這個選項存在之前外掛寫死的那個字串。從未設定過它的 storage，升級後會產生逐位元組相同的名稱，因此儲存伺服器上既有的每一個磁碟區都仍然被認得。

Nothing acts on old volumes unattended. The orphan reaper touches only the node’s own devices and never deletes an array volume; the temporary-clone reaper works from a state file this plugin wrote, which old volumes are not in; and per-VM volume groups are off by default and never add existing volumes retroactively. 沒有任何路徑會在無人值守的情況下動到舊磁碟區。孤兒回收器只處理節點自己的裝置，從不刪除儲存伺服器上的磁碟區；暫時複製回收器依據的是外掛自己寫下的狀態檔，而舊磁碟區不在其中；每 VM 一個 volume group 預設關閉，也不會把既有磁碟區追溯加入群組。

Two things that do change, and both are fixes有兩件事確實會改變，而兩者都是修正

• A multipath map may be built on first use. Older versions could accept a single /dev/sdX and leave the guest with no failover. From 0.8.28 the WWID is claimed and the map is waited for, so multipath -a appearing in the journal for an existing volume is expected. 第一次使用時可能會建立 multipath map。舊版可能接受單一的 /dev/sdX，讓客體沒有任何路徑備援。從 0.8.28 起會認領 WWID 並等待 map 出現，因此在日誌中看到對既有磁碟區執行 multipath -a，是預期的。
• Adding an existing storage warns that volumes already exist under its prefix. That is the cross-cluster check; for a storage you are re-adding it is expected, and the message says so. 加入既有 storage 時會警告該前綴下已經有磁碟區。那是跨叢集檢查；對於你自己重新加入的 storage，這是預期的，訊息裡也寫明了這種讀法。

Quick Start 快速入門

PowerStore

pvesm add dellpowerstore ps1 \
    --dell-portal 192.168.1.50 \
    --dell-username pveadmin \
    --dell-password 'SecurePassword' \
    --dell-protocol iscsi \
    --content images,rootdir \
    --shared 1

PowerVault ME4 / ME5

pvesm add dellpowervault me5 \
    --dell-portal 192.168.1.60,192.168.1.61 \
    --dell-username manage \
    --dell-password 'SecurePassword' \
    --pvault-pool A \
    --content images,rootdir \
    --shared 1

PowerFlex

pvesm add dellpowerflex pflex1 \
    --dell-portal 192.168.1.70 \
    --dell-username admin \
    --dell-password 'SecurePassword' \
    --dell-protocol nvme \
    --pflex-storage-pool pool1 \
    --content images,rootdir \
    --shared 1

Unity XT

pvesm add dellunity u480 \
    --dell-portal 192.168.1.80 \
    --dell-username admin \
    --dell-password 'SecurePassword' \
    --dell-protocol fc \
    --unity-pool pool_1 \
    --content images,rootdir \
    --shared 1

Verify 驗證

pvesm status
journalctl -t pvestatd | grep dellpowerstore

Configuration 設定選項

Options shared by every family use the dell- prefix and are declared once; family options use their own prefix. PVE registers storage properties in one shared schema, so a name may only have one definition across all plugins. 所有系列共用的選項使用 dell- 前置字串且只宣告一次；各系列專屬選項使用自己的前置字串。PVE 的 storage property 註冊在同一份共用 schema，同一個名稱在所有外掛之間只能有一種定義。

Common options 共通選項

Family options 系列專屬選項

Full reference: docs/CONFIGURATION.md. 完整說明：docs/CONFIGURATION_zh-TW.md。

Host objects the array already has 儲存伺服器上原本就有的 host 物件

An array usually has a host object for each node before this plugin ever runs — built by whoever zoned the fabric — holding that node’s WWPNs or IQN under a name of its own. An initiator belongs to one host object, so the plugin cannot register the same ports a second time under its own name. 在這個外掛跑起來之前，儲存伺服器上通常每台節點都已經有一個 host 物件 — 由當初做 fabric 分區的人建立 — 以它自己的命名持有該節點的 WWPN 或 IQN。一個 initiator 只能屬於一個 host 物件，所以外掛沒辦法用自己的名稱把同樣的埠再註冊一次。

On PowerStore it does not try. When there is no host under pve-<cluster>-<node>, it asks the array which host holds this node’s initiators and uses that one, recording the name in /var/lib/pve-storage-dellemc/<storeid>-host. Nothing is renamed, nothing is removed, and no initiator is moved. 在 PowerStore 上它不會去試。當 pve-<叢集>-<節點> 底下沒有 host 時，它會問儲存伺服器「本節點的 initiator 在哪個 host 上」，改用那一個，並把名稱記在 /var/lib/pve-storage-dellemc/<storeid>-host。不改名、不刪除，也不搬動任何 initiator。

It adopts only a host whose initiators are a subset of this node’s. One that also carries another host’s ports is refused, by name: a volume mapped there would be visible to whatever those belong to. Ports split across two host objects are refused as well — a node is one host object, and merging them is the operator’s call. 它只會採用 initiator 是本節點子集的 host。若那個 host 同時持有別台的埠，就會被拒絕並指名是哪一個：對應到它的磁碟區會被那些東西看見。本節點的埠散在兩個 host 物件上也一樣拒絕 — 一個節點就是一個 host 物件，要合併是操作者的決定。

System Architecture 系統架構

DellPowerStorePlugin / DellPowerVaultPlugin      # SAN families
        |
        v
DellEMC::Common::BlockBase           # activation, allocation, device
        |                            # discovery, snapshots, clones,
        |                            # multipath drop-in, orphan reaper
        +-- Common::REST             # retries, timeouts, sessions
        +-- Common::ISCSI            # initiator, portals, rescan
        +-- Common::FC               # HBA discovery, WWN handling
        +-- Common::Multipath        # SCSI lifecycle, dm-multipath
        +-- Common::Naming           # PVE names <-> array objects
        +-- Common::WwidState        # WWID tracking, reap guards
        +-- Common::Health           # outage and capacity reporting
        +-- Common::Schema           # the shared dell-* options

DellPowerFlexPlugin                  # no SCSI LUN, no multipath
        +-- PowerFlex::Host          # SDC or NVMe/TCP device access

Why the plugin is careful about the host 為什麼外掛在主機端如此謹慎

PowerFlex: SDC vs NVMe/TCP PowerFlex：SDC 與 NVMe/TCP

PowerFlex volumes do not arrive as SCSI LUNs. There are two ways a node can see them, and the choice outlives this plugin. PowerFlex 的 volume 不是以 SCSI LUN 出現。節點有兩種方式看到它們，而這個選擇的影響會比這個外掛活得更久。

Official Dell references Dell 官方資料

Bookmark these; they are the authority on what is supported, and they change. 請加入書籤；支援範圍以它們為準，而且內容會變動。

Dell notes that the E-Lab matrix does not always list every kernel a distribution has released, and gives a prefix rule: a kernel is supported if its version prefix matches a listed one — 4.18.0-553 covers 4.18.0-553.51.1.el8_10.x86_64. Details and the commands to check a node are in docs/POWERFLEX_SDC.md. Dell 也說明 E-Lab 矩陣不一定涵蓋各發行版釋出的每一個 kernel，並給了一條前置字串規則：版本前置字串相符即視為支援 —— 4.18.0-553 涵蓋 4.18.0-553.51.1.el8_10.x86_64。細節與節點檢查指令請見 docs/POWERFLEX_SDC_zh-TW.md。

Multipath Safety Multipath 安全規則

Supported Features 功能支援表

Verification Status 驗證狀態

3,411 unit tests run without an array or a device. What they cannot tell you is whether the endpoints exist, whether the field names are right, or whether a device ever appears — only an array can, and exactly one has. 3,411 個單元測試不需要儲存伺服器或實體裝置即可執行。它們無法告訴你的是：端點是否存在、欄位名稱是否正確、裝置到底會不會出現 — 那隻有儲存伺服器能回答，而目前只有一台回答過。

What has run on hardware 哪些在實機上跑過

The ME4024 run covered the whole of the first-run test: capacity agreeing with the array's own GUI, several allocations with LUNs in sequence, dm-multipath with two paths, dd read and write verified by checksum, snapshot, rollback, snapshot delete, template, a linked clone in seconds, the array correctly refusing to delete a template with a live clone, and unmap, delete and local device cleanup. It also found three defects that reading Dell's documentation had not — each hidden behind the one before it. 那次 ME4024 的測試涵蓋了整套首次執行測試：容量與儲存伺服器自己的 GUI 一致、連續數次配置且 LUN 依序遞增、dm-multipath 兩條路徑、以雜湊驗證的 dd 讀寫、快照、倒回、刪除快照、範本、秒級完成的連結複製、儲存伺服器正確拒絕刪除仍有存活複製的範本，以及解除對應、刪除與本機裝置清理。它同時找出三個讀 Dell 文件找不出來的缺陷 — 而且一個藏在另一個後面。

Where the rest came from 其餘的依據是什麼

The full matrix, including what came from official documentation and what did not, is in docs/TESTING.md. 完整矩陣（包含哪些來自官方文件、哪些不是）請見 docs/TESTING_zh-TW.md。

Before the first run against an array, work through docs/FIRST_RUN.md: the order to do it in, what to check after each step, and what each failure most likely means. 第一次接上儲存伺服器之前，請照著 docs/FIRST_RUN_zh-TW.md 走一遍：該用什麼順序做、每一步之後要看什麼，以及每種失敗最可能代表什麼。

Troubleshooting 疑難排解

Every message carries a [type:storeid] prefix, so one storage can be grepped out of a busy journal. 每一則訊息都帶有 [type:storeid] 前置字串，可以從繁忙的 journal 中過濾出單一儲存。

journalctl -t pvestatd -t pvedaemon | grep dellpowerstore

Full guide: docs/TROUBLESHOOTING.md. 完整指南：docs/TROUBLESHOOTING_zh-TW.md。

Changelog 變更紀錄

Full changelog: CHANGELOG.md. 完整變更紀錄：CHANGELOG_zh-TW.md。

Acknowledgments 致謝

The host-side layer — device discovery, multipath handling, the anti-hang rules and the orphan reaper — is carried over from two related projects, where those lessons were paid for in production: 主機端底層 —— 裝置探索、multipath 處理、防止卡死的規則與 orphan 清理 —— 承接自兩個相關專案，那些教訓是在正式環境付出代價換來的：

In the absence of hardware, this plugin was written against Dell's published documentation: the PowerStore REST guide, the PowerVault ME5 CLI Reference Guide, and the PowerFlex REST and SDC material. 在沒有實機的情況下，本外掛是依據 Dell 公開的文件撰寫的：PowerStore REST 指南、PowerVault ME5 CLI Reference Guide，以及 PowerFlex 的 REST 與 SDC 資料。

Related projects 相關專案

Proxmox VE storage plugins for other storage servers, sharing the host-side layer and the operational rules this one inherits. Several of the defects fixed here were found by reading their incident records. 其他儲存伺服器的 Proxmox VE 儲存外掛，與本專案共用主機端那一層，以及本專案承接的維運規則。這裡修掉的缺陷有好幾個，是讀它們的事故紀錄找到的。