First, let’s understand some terminology:

Fig.1: SSD key components (source)

A storage device has many parts, here we focus on the controller and the physical connector used to communicate with the host.

1. Physical Interface: It is the physical layer which includes the connectors, cable, motherboard slots. This determines how fast bits can physically move and what shape the plug is. Examples: SATA, SAS, PCIe.

2. Protocol Interface: It includes the language and rules used to send commands over that physical connection. This determines how the host CPU and storage device communicate - what commands exist, how many queues, how errors are reported. Examples: ATA, SCSI, NVMe.

A storage solution is often a combination of both a physical interface and a protocol. For example, a “SATA storage device” uses ATA commands over a serial connection (hence, Serial ATA), while a “SAS storage device” uses SCSI commands over a serial connection (hence, Serial Attached SCSI).

TL;DR: The storage device has a controller chip which implements the protocol interface. On the host, the operating system has driver software to communicate using the same protocol. And the physical interface is the physical connector between the host and storage device.

A brief history of storage interfaces

Early storage systems were designed around mechanical hard drives.

On a mechanical hard drives, when working with multiple files in different locations, the drive head moved at high speed (across various parts of the drive) to read and write data on disk. They could effectively only seek to one location/file at a time.

Consumer storage used ATA (also called IDE or Parallel ATA) and enterprise systems used SCSI. These interfaces defined both the command set and the controller architecture.

In the early 2000s, these two evolved to use serial links:

• ATA evolved into SATA (Serial ATA)
• SCSI evolved into SAS (Serial Attached SCSI)

But both new interfaces continued to use their predecessor’s command set. SATA continued using ATA commands, while SAS continued using SCSI commands, but transmitted them over modern serial connections.

When SATA was introduced, the industry wanted a standard way for operating systems to talk to SATA controllers. Intel developed AHCI (Advanced Host Controller Interface), which has become a de facto standard.

Since hard drives couldn’t support multiple command queues, AHCI was built with 1 command queue, with a maximum of 32 commands per queue. This was sufficient for hard drives, but not for newer flash storage.

Why was NVMe needed?

With flash storage used in SSDs, the physical limitation no longer existed (as they don’t have a drive head and used technology such as NAND) and can service many requests in parallel.

By late 2010, mainstream consumer SSDs were being bottlenecked by the comparatively limited throughput granted by SATA (~6Gb/s - speed limitation inherent to SATA). In the enterprise market, this limitation was overcome by creating SSDs that connected via PCI Express.

Now, these SSDs were being bottlenecked by AHCI’s random I/O performance. Aternatively, SSDs came with custom interface specification which varied a lot between vendors (and even within models of same vendors). This variation made it difficult to support vendor-specific software drivers.

So NVM Express was introduced as an industry standard interface for PCIe-connected SSDs to use, with a common driver specification (eliminating the need for vendor-specific drivers).

NVMe specification has a flash-first philosophy and offered 65k command queues with up-to 65k commands per queue - leading to higher performance.

The NVM Express Base Specification defines a protocol for host software to communicate with non-volatile memory subsystems over a variety of memory-based transports (PCIe) and message-based transports (RDMA, TCP). It is the industry standard for solid state drives (SSDs) in all form factors (U.2, M.2, AIC, EDSFF).

Source: https://nvmexpress.org/

Why is NVMe faster?

1. Multiple independent I/O queues

   What is a queue? When your CPU wants to read or write data, it doesn’t talk to the drive directly - it puts a command in a queue and the drive’s controller picks it up and executes it.

   AHCI provides 1 command queues with 32 commands per queue. So all CPU cores share that one queue, which means there’s “locking” - one core has to wait for another core to finish writing its command before it can add its own to the queue. It was fine with hard disks because the disk could only physically seek to one location at a time.

   NVMe supports 64 thousand queues and each queue can handle 64k commands. So, each CPU core can have it’s own IO queue who can all go simultaneously and independently, and they don’t have to wait for each other.

   We will learn more about how queues work in a later section.

2. Direct communication over PCIe: With SATA and SAS, an HBA (Host Bus Adaptor) sits between the CPU and storage - it translates PCIe signals into SATA/SAS signals, and runs the AHCI/SCSI command protocol. But with NVMe, the drive sits directly on the PCIe bus and speaks PCIe natively - the HBA is gone, and so is the protocol translation layer.

3. Lower command overhead: AHCI fetches each command in two RAM reads - a pointer to the command, then the actual command. NVMe puts the full command directly in the submission queue, so the controller fetches it in one.

Form Factors

Form factor is a hardware design that defines the size, shape, connector type of a storage device - independent of the protocol it speaks internally.

Traditional hard drives have generally come in 3.5″ and 2.5″ form factors, and SSDs have typically been available as 2.5″ or mSATA cards. NVMe drives have a variety of form factors– the most common is the M.2 format, which is 22mm wide with lengths of 30, 42, 80, or less often 110 millimeters. (source)

Picture: Form Factors (source)

NVMe architecture

Picture: NVMe Architecture

Linux natively supports NVMe Driver through the nvme kernel module, it exposes devices:

• /dev/nvmeX -> the physical NVMe hardware controllers
• /dev/nvmeXnY -> namespaces, which are logical storage volumes

NVMe Driver (Software)

The NVMe driver (nvme kernel module) sits between the OS and the NVMe hardware. When an NVMe SSD is plugged in, the driver detects it as a PCIe endpoint and sets up the communication channel: the submission and completion queues in host memory, and the doorbell registers it uses to signal the controller.

NVMe Driver supports two kind of commands - admin commands and I/O commands.

NVMe Controller (Hardware)

The NVMe controller is a dedicated processor built directly onto the SSD, responsible for managing communication between the host system and the storage media.

It receives commands from the driver, executes them against the flash storage, and sends results back. Under the hood it also handles the messier parts of managing flash - wear leveling, garbage collection, and error correction - so the host never has to think about them.

In Linux, the controller is exposed as /dev/nvmeX. Each physical SSD has one controller, and namespaces live under it - which is why the device naming is nvme0n1, nvme0n2: controller 0, namespace 1 and 2. More about this in Namespaces section.

Queues and its workflow

NVMe’s performance benefits stem from its queue-based architecture.

Picture: NVMe admin and multiple I/O queue pair (source)

Queue Pairs (Submission Queue + Completion Queue) & Doorbell registers

Every queue in NVMe is a queue pair - a Submission Queue (SQ) and a Completion Queue (CQ):

• SQ: The NVMe driver submits commands on the submission queue for controller to execute.
• CQ: The controllers posts the results of executed commands on the completion queue.

These queues are fixed-size circular buffers in host memory. The driver writes commands at the tail of the SQ, and the controller reads from the head. The controller writes results at the tail of the CQ, and the driver reads from the head. When either reaches the end of the buffer, it wraps back to position 0.

There’s one more part: doorbell registers - tiny registers on the NVMe controller hardware, used as a notification mechanism to let the controller or the driver know that the SQ/CQ are updated.

How they work together:

1. Once the driver places a command in the submission queue, the host updates the “submission queue tail doorbell register” to notify the controller.
2. The controller fetches the command from submission queue and executes it.
3. The controller then writes result into the completion queue. Then the controller updates the “completion queue head doorbell register” to inform the host that new completion entries are available.

Picture: NVMe queue pair workflow (source)

The admin queue and I/O queues: There are two kinds of queue pair to handle two kinds of commands:

1. The Admin queue

It is the queue pair (SQ + CQ) in host RAM that handle admin commands. These include commands for identifying devices, managing namespaces, and fetching healthcheck information.

Every queue pair has a unique identifier (QID): QID 0 is always the admin queue, QID 1 onwards are IO queues.

The admin queue is separate from IO queues so management commands always have a clear path, even when the drive is under heavy IO load.

2. I/O queues

They are queue pairs (SQ + CQ) in host RAM that handle I/O commands. These commands enable the data operations on the device, including read/write operations.

NVMe theoretically supports up to 64K queues, with each queue capable of holding up to 64K commands. However, most real-world NVMe devices support significantly fewer queues and commands.

Namespaces

Namespaces provide logical segmentation within an NVMe device. NVMe namespaces are logically isolated set of Logical Block Addresses (LBAs). The host sees a contiguous address space - block 0 to block N (regardless of how the data is physically laid out on the flash). Read here to understand more about LBAs.

In Linux, each namespace is exposed as a block device (e.g., /dev/nvme0n1), where nvme0 refers to the controller and n1 refers to the namespace. Each namespace looks like a distinct storage target to the host.

Example, namespaces across multiple controllers/drives:

$ nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            012510B00426         MTFDKCC1T9TGP-1BK1JABYY                  0x1        141.53  GB /   1.92  TB    512   B +  0 B   E3MQ000
/dev/nvme1n1          /dev/ng1n1            012510B00440         MTFDKCC1T9TGP-1BK1JABYY                  0x1        139.12  GB /   1.92  TB    512   B +  0 B   E3MQ000
/dev/nvme2n1          /dev/ng2n1            25174FCD463C         Micron_7450_MTFDKBA960TFR                0x1         18.29  GB / 960.20  GB    512   B +  0 B   E2MU200 

In this example,

• /dev/nvme0n1 is the block device - the one that gets mounted, formatted with a filesystem, and used for IO.
• /dev/ng0n1 is a character device - a direct unbuffered channel to the controller used for admin commands like health checks or firmware updates.
• “Format” is the logical block size (512 bytes here).

Namespaces are accessible by the host through a unique Namespace Identifier (NSID). NSID are named local to the controllers. So in above example, all three namespaces have NSID 1 on their respective controllers.

A single controller can expose multiple namespaces - each appearing as an independent block device to the host:

One controller, multiple namespaces:
/dev/nvme0n1   ← controller 0, namespace 1
/dev/nvme0n2   ← controller 0, namespace 2
/dev/nvme0n3   ← controller 0, namespace 3

This helps to isolate workloads on the same drive. So, each namespace can have its own encryption key (so two users cannot read each other’s data on same hardware), and its own write protection, etc.

Subsystem

An NVMe subsystem is a top-level abstraction that encapsulates:

• one or more controllers
• one or more namespaces within the controllers
• one or more paths (i.e. physical or logical access points - PCIe ports for local NVMe or network ports for NVMe-oF)

Fig: An example of a subsystem as shown in NVMe Base Spec (source)

Each NVMe subsystem is exposed under /sys/class/nvme and initialilzed by the nvme-core and nvme driver.

Example, 3 local NVMe SSDs, each as its own subsystem with one PCIe path:

$ nvme list-subsys --output-format=json
[
  {
    "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:b8b85908-723a-45a9-922d-a912639f1461",
    "HostID":"b8b85908-723a-45a9-922d-a912639f1461",
    "Subsystems":[
      {
        "Name":"nvme-subsys0",
        "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B00426",
        "Paths":[
          {
            "Name":"nvme0",
            "Transport":"pcie",
            "Address":"0000:03:00.0",
            "State":"live"
          }
        ]
      },
      {
        "Name":"nvme-subsys1",
        "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B00440",
        "Paths":[
          {
            "Name":"nvme1",
            "Transport":"pcie",
            "Address":"0000:04:00.0",
            "State":"live"
          }
        ]
      },
      {
        "Name":"nvme-subsys2",
        "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-25174FCD463C",
        "Paths":[
          {
            "Name":"nvme2",
            "Transport":"pcie",
            "Address":"0000:02:00.0",
            "State":"live"
          }
        ]
      }
    ]
  }
]

Here, "Transport": "pcie" means local NVMe device. In NVMe-oF, the same structure appears but with "Transport": "tcp" and a network address.

NQN (NVMe Qualified Name)

NQN (NVMe Qualified Name) is a unique identifier for a subsystem or a host. Every subsystem has one, every host has one.

1. Host NQN is used to uniquely identify the host/client system.
2. Subsystem NQN is used to uniquely identify the storage device.

From the above nvme list-subsys output, we can see a host NQN and subsystem NQN:

[
  {
    // host NQN 
    "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:b8b85908-723a-45a9-922d-a912639f1461",
    "HostID":"b8b85908-723a-45a9-922d-a912639f1461",
    "Subsystems":[
      {
        "Name":"nvme-subsys0",
        // subsystem NQN
        "NQN":"nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-012510B00426",
        "Paths":[
           ... 
          ]
      },  
    ]
  }
]

Host NQN is stored at /etc/nvme/hostnqn:

$ cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:6def1fb3-a3a2-447b-9938-59856fc4805f

Now, let’s observe the format of these NQN:

Format:  nqn.{YYYY-MM}.{reverse.domain}:{unique-name}

The reverse domain (com.micron, org.nvmexpress) is the naming authority. It ensures two different vendors can’t accidentally create the same NQN. The unique name after the colon is whatever makes it distinct - a serial number for drives, a UUID for hosts.

NVMe-oF

Storage can be accessed through 3 approaches: Direct-attached storage (DAS), NAS (network-attached storage), and SAN (storage area networks).

1. DAS systems have the storage devices directly attached to the client systems.

The other two are remote storage systems (accessed over a network):

2. NAS systems serves files over the network using protocols such as NFS (network file system) and SMB (server message block) to access the file system. The storage server manages the filesystem and clients access files and directories rather than raw storage devices. It is often used within small teams or household file servers.

3. SAN systems are used to access block devices over the network. The storage server provides raw blocks, while the client is responsible for creating and managing filesystems.

Since SAN clients operate directly on block devices, they need a protocol to issue storage admin/io commands to a remote storage.

A brief history of remote block storage protocols

Before NVMe, the industry standardized on the SCSI command set for many enterprise storage. Originally designed for locally attached disks, SCSI became the standard way for hosts to communicate with storage devices and was later reused in technologies such as SAS. Then the industry extended SCSI beyond local connections.

Instead of creating a new protocol, SCSI commands were encapsulated and transported across a network. This became known as iSCSI. This allowed SCSI commands to be exchanged between initiator (iSCSI client) and target (iSCSI server with storage).

NVMe over Fabric

Similar to how iSCSI made it possible to send SCSI commands over a network, NVMe-oF (part of NVMe Base specifications) allows encapsulation of NVMe commands over a network. Also like iSCSI, NVMeoF also requires an initiator (client) and a target (server) to establish a connection and exchange commands.

The key advantage of NVMe-oF is that it keeps the performance benefits of NVMe:

1. Parallel I/O operations: In NVMe-oF, the queues allow multiple I/O requests to be processed concurrently between the host and storage target.
2. Reduced CPU overhead: Because NVMe allows different CPU cores to use separate queues, they can process I/O requests independently without frequently coordinating access to a shared queue. This reduces CPU overhead on both the host and storage target.
3. Fewer protocol interactions: Since NVMe requires fewer protocol interactions (one round trip and two interactions) to complete an I/O request than traditional SCSI-based protocols (two round trips and four interactions), it results in NVMeoF providing lower latency.

Transport types

NVMeoF standard is not fabric-specific, instead there is a separate Transport Binding spec for each transport layer. From host’s perspective, NVMe commands are issued identically regardless of which transport layer carries them underneath.

Three transport bindings are currently standardized:

1. NVMe/TCP
2. NVMe/RDMA
3. NVMe/FC

NVMe-oF Architecture

NVMeoF has the client/server model: The initiator / host is the client system that needs to access remote storage, and runs the NVMe-oF initiator driver (nvme-fabrics kernel module on Linux). The target is the storage server that exposes subsystems with namespaces to the network.

Picture: NVMe-oF Architecture (diagram inspired by smartx blog)

The initiator connected to a remote subsystem (on target hosts) and once connected, the remote namespace appears exactly like a local NVMe namespace - eg, /dev/nvme0n1 - on the initiator system.

NVMe-oF Subsystems

NVMe-oF subsystem is similar to local NVMe - it’s still the top-level container with controllers and namespaces, identified with an NQN. The difference is it now exists on the target machine and is reachable over the network.

1. NVMeoF Controllers

The spec defines two controller models: the static controller model and the dynamic controller model.

In local NVMe, the controller is a static controller model because is a hardware chip and always present. In NVMe-oF, the spec defines a dynamic controller - it is instantiated by the target when an initiator successfully connects, and destroyed when the initiator disconnects. And this exclusive relationship between a controller and a host is called an association.

First, the initiator requests connection to a remote subsystem using the connect command:

$ nvme connect \
   --transport tcp \
   --traddr 10.243.64.86 \
   --trsvcid 4420 \
   --nqn nqn.2016-06.io.spdk:cnode1
connecting to device: nvme0

Then, the target instantiates a dynamic controller for that host. This controller owns the admin queue and I/O queue pairs for that host connection, just like a local NVMe controller. Only difference is that these commands are carried over TCP (here) instead of PCIe.

After connecting, we can see NVMeoF subsystem from the initiator like this:

$ nvme list-subsys --output-format=json
[
  {
    "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:6def1fb3-a3a2-447b-9938-59856fc4805f",
    "HostID":"6def1fb3-a3a2-447b-9938-59856fc4805f",
    "Subsystems":[
      {
        "Name":"nvme-subsys0",
        "NQN":"nqn.2016-06.io.spdk:cnode1.mygroup1",  // subsystem NQN
        "Paths":[
          {
            "Name":"nvme0",
            "Transport":"tcp",
            "Address":"traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88",
            "State":"live"
          }
        ]
      }
    ]
  }
]

Here, the output shows that the initiator (“HostNQN”) is connected to one remote subsystem (nqn.2016-06.io.spdk:cnode1.mygroup1) through a single TCP path. The path connects the initiator (10.243.64.88) to the target (10.243.64.86:4420), and the dynamic controller created for this connection appears on the initiator as nvme0. “State": "live” indicates that the connection is active and able to do I/O.

When the host disconnects, the association ends and the target destroys the dynamic controller.

2. NVMeoF Namespace

Here, Namespaces are same as local NVMe: they are LBA exposed as block devices and identified by NSID. In fact, remote namespaces looks just like local NVMe namespaces to the initiator. Example: /dev/nvme0n1 (controller nvme0, namespace 1).

Example of a NVMeoF target with a subsystem and 3 namespaces:

$ nvme show-topology
nvme-subsys0 - NQN=nqn.2016-06.io.spdk:cnode1.mygroup1
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6def1fb3-a3a2-447b-9938-59856fc4805f
               iopolicy=numa
\
 +- ns 1
 \
  +- nvme0 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
 +- ns 3
 \
  +- nvme0 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
 +- ns 5
 \
  +- nvme0 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized

This output shows a single dynamic controller (nvme0) connected to a subsystem that exposes three namespaces (NSIDs 1, 3, and 5), and all are accessible through the same port. The “live optimized” state is the ANA state, indicating this is the optimal path for each namespace. We’ll cover ANA in a later section.

Multiple initiator hosts: Since each host connection creates a dynamic controller, multiple hosts can connect to same namespaces through their own controllers.

Bonus: NVMe spec also defines “shared namespaces” where a namespace can be attached to two or more controllers concurrently.

3. Allowed Host List / Access control

In local NVMe, physical access to the device is the only access control. Over a network, you need explicit control over which hosts are allowed to connect.

Each NVMe-oF subsystem maintains an Allowed Host List which is a list of Host NQNs that are allowed access to connect to that subsystem. When a host sends a connect command, it presents its host NQN. The subsystem checks it against this list before allowing the session. If the NQN isn’t on the list, the connection is rejected.

NVMe-oF Discovery

Before a host can connect to a subsystem, it needs to know what subsystems exist and where to reach them. This is what the Discovery Service is for.

Discovery Service is a subsystem that hosts only Discovery controllers. It allows hosts to automatically find available NVMeoF targets and subsystems on the network.

Discovery controller is a special controller type that exposes no namespaces and serves no I/O - its sole purpose is to return a Discovery Log Page: a list of all subsystems accessible to this host, along with the transport address needed to reach each one.

How it works?

1. The host queries a target gateway at the well-known discovery address (port 8009 for “centralized discovery” - i.e. a discovery controller that maintains the full target list).

   Note: this port is separate from 4420, which is used for actual I/O subsystems.

2. It sends a “Get Log Page” command to the Discovery Controller, which responds with a Discovery Log Page (see nvme discover example below).

# Discover all NVMe-oF targets at a given address
nvme discover -t tcp -a 10.243.64.86 -s 8009

# Connect to a discovered subsystem by NQN
nvme connect -t tcp -a 10.243.64.86 -s 4420 -n nqn.2016-06.io.spdk:cnode1.mygroup1

# Connect to all discovered subsystems at once
nvme connect-all -t tcp -a 10.243.64.86

So the discovery only works after a host already knows the address of at least one Discovery controller. nvme connect-all does same discovery internally by sending the same “Get Log Page” request to a Discovery controller, then automatically issues a connect command for every entry returned.

Here is an example of discovery of two NVMeoF target gateways (for multipath):

# nvme discover -t tcp -a 10.243.64.86 -s 8009

Discovery Log Number of Records 8, Generation counter 6
=====Discovery Log Entry 0======
trtype:  tcp                                  // transport type
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  0
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode1.mygroup1    // subsystem NQN
traddr:  10.243.64.86
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  1
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode1.mygroup1
traddr:  10.243.64.87
eflags:  none
sectype: none
....
....
.... (more entries)

The Discovery Service also helps to discover multiple paths to a subsystem.

Multipath:

In above nvme discover output, both of the entries have the same “subnqn” but different “traddr” values - these are two paths to the same subsystem (i.e. 2 subsystem ports on two different target gateways). This is how the host discovers that multipath is available: multiple Discovery Log entries for the same NQN, each pointing to a different address.

ANA (Asymmetric Namespace Access)

A NVMe-oF subsystem can often be reached through multiple paths. For example, a distributed storage system can expose the same subsystem through multiple target gateways. This gives the host more than one way to reach the same namespaces.

If every path were treated equally, the host would have no way of knowing which path provides the best access to a namespace. Some paths may provide direct access to the storage media, while some other path may require requests to be forwarded internally to another controller. Even though all paths can access the same data, one path might offer lower latency or higher throughput than the other.

Core Idea - the why and how

Each namespace belongs to an ANA Group. Instead of tracking the state of every namespace individually, ANA state is tracked at the group level, so all namespaces in the same ANA Group transition state together. This makes state changes much more efficient. Rather than updating thousands of namespaces individually during a controller failover, the target only updates the state of the affected ANA Group.

$ nvme id-ns /dev/nvme1n1
NVME Identify Namespace 1:
..
anagrpid: 1
..

$ nvme id-ns /dev/nvme1n2
NVME Identify Namespace 2:
...
anagrpid: 2
...

$ nvme id-ns /dev/nvme1n3
NVME Identify Namespace 3:
..
anagrpid: 1
..

The host gets the ANA states by reading the “ANA Log Page”, which maps each ANA Group to its current ANA state. Since each namespace belongs to an ANA Group, the host immediately knows which paths should be used for every namespace.

ANA states

ANA group can have one of these 5 states:

1. Optimized: The preferred path for accessing the namespace.
2. Non-Optimized: The namespace is accessible through this controller, but another controller provides better performance.
3. Inaccessible: The namespace cannot currently be accessed through this controller.
4. Persistent Loss: The controller has permanently lost access to the namespace.
5. Change: The ANA state is changing. The host should retrieve an updated ANA Log Page.

Example:

After connecting to a 2 NVMeoF gateway setup, a subsystem with 6 namespaces looks like this from the host:

$ nvme show-topology
nvme-subsys1 - NQN=nqn.2016-06.io.spdk:cnode1.mygroup1
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6def1fb3-a3a2-447b-9938-59856fc4805f
               iopolicy=numa
\
 +- ns 1
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
 +- ns 2
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live optimized
 +- ns 3
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
 +- ns 4
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live optimized
 +- ns 5
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
 +- ns 6
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live optimized

Here, namespaces 1, 3, 5 are in one ANA Group - optimized through controller nvme1 (gateway 10.243.64.86), inaccessible through controller nvme2 (gateway 10.243.64.87). Namespaces 2, 4, 6 are in a second ANA Group - optimized through nvme2, inaccessible through nvme1.

Subsystem (nqn.2016-06.io.spdk:cnode1.mygroup1)
├── Controller 1 (nvme1)
│   ├── ANA Group 1 → Optimized
│   └── ANA Group 2 → Inaccessible
│
├── Controller 2 (nvme2)
│   ├── ANA Group 1 → Inaccessible
│   └── ANA Group 2 → Optimized
│
└── ANA Groups
    ├── ANA Group 1
    │   ├── Namespace 1
    │   ├── Namespace 3
    │   └── Namespace 5
    │
    └── ANA Group 2
        ├── Namespace 2
        ├── Namespace 4
        └── Namespace 6

What Happens During Failover?

If second gateway (10.243.64.87) goes down, nvme2 becomes unreachable (path becomes from “live” to “connecting”). The first surviving gateway sends an “Asymmetric Namespace Access Change Notice (AEN)” to the host controller, indicating that the ANA state has changed. That triggers the host to issue “Get Log Page” command to get updated ANA Log Page.

With the updated ANA information, the surviving controller (nvme1) becomes the optimized path for both ANA Groups:

// after second gateway (10.243.64.87) goes down:

$ nvme show-topology
nvme-subsys1 - NQN=nqn.2016-06.io.spdk:cnode1.mygroup1
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6def1fb3-a3a2-447b-9938-59856fc4805f
               iopolicy=numa
\
 +- ns 1
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting inaccessible
 +- ns 2
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting optimized
 +- ns 3
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting inaccessible
 +- ns 4
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting optimized
 +- ns 5
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting inaccessible
 +- ns 6
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting optimized

If we focus on namespace 2, the nvme1 path changes from “inaccessible”:

// 2 gateways up:
 +- ns 2
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live inaccessible
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420,src_addr=10.243.64.88 live optimized

to “optimized”:

// then after 1 gateway goes down:
 +- ns 2
 \
  +- nvme1 tcp traddr=10.243.64.86,trsvcid=4420,src_addr=10.243.64.88 live optimized
  +- nvme2 tcp traddr=10.243.64.87,trsvcid=4420 connecting optimized

References

• https://sniansfblog.org/a-beginners-guide-to-nvme/
• https://blogs.oracle.com/linux/overview-of-nvme-architecture

pcie:

• https://www.synopsys.com/glossary/what-is-pci-express.html
• https://computer.howstuffworks.com/pci-express.htm

nvme:

• Official NVMe Base spec: https://nvmexpress.org/wp-content/uploads/NVM-Express-Base-Specification-Revision-2.3-2025.08.01-Ratified.pdf
• https://www.techrepublic.com/article/nvme-a-cheat-sheet/
• https://www.techtarget.com/searchstorage/feature/NVMe-market-and-technology-rapidly-evolve
• https://www.everpuredata.com/knowledge/what-is-nvme.html

nvmeof:

• https://codilime.com/blog/understanding-nvme-over-fabrics-nvmeof/
• https://techbuzzonline.com/nvme-over-fabrics-nvme-of-networking/
• https://nvme-storage.com/nvme-of/
• https://nvme-storage.com/glossary/nvme-of-discovery/
• https://simplyblock.io/blog/nvme-over-tcp-vs-iscsi/