iSER Draft Status

iSER Draft Status draft-ietf-ips-iser-00 Mike Ko November 8, 2004

iSER Flow Control for Control-Type PDU • As currently defined, the iSER protocol does not provide additional flow control beyond that provided by the iSCSI layer on control-type PDUs • From section 8.1: “The iSER Layer SHOULD provision enough Untagged buffers for handling incoming RDMAP Send Message Types to prevent a buffer underrun condition” • iWARP Verbs mechanisms such as the Shared Receive Queue mechanism can be used to effectively address the Send Message flow control question • Most iSCSI PDUs are paced by the CmdSN window mechanism in iSCSI • Question: Should some form of send side flow control be established for iSCSI control-type PDUs not regulated by the CmdSN window? M. Ko

Flow Control Alternatives for Unexpected PDUs • Require the receiver to replenish buffers faster than the arrival rate of incoming PDUs including the unexpected ones • Imposes a constraint on the implementation that may not be achievable in all circumstances • Require the RNIC to do graceful handling on lack of receiver buffer situations • Mandates the implementation of an optional feature • Declare a limit on the number of unexpected PDUs that can be sent • If architected correctly, can also accommodate options 1, 2, and 5 M. Ko

Flow Control Alternatives for Unexpected PDUs • Define an explicit credit based flow control mechanism • Basically a receive side flow control • Extra overhead even when options 1, 2, and 5 are used • Require the RNIC to use Shared RQ • Mandates the implementation of an optional feature • Drop the link if there is buffer overrun • Default solution if no other flow control alternatives • Drastic solution compared with other alternatives M. Ko

Declared Limit on Sender Side on the Number of Unexpected PDUs • First discussed on the IPS reflector in July/August of 2003 between Mike Ko, Caitlin Bestler, David Black, etc. • Idea revived and refined in October 2004 • A new key, MaxOutstandingUnexpectedPDU, can be used by both the initiator and the target to declare the maximum number of outstanding “unexpected” control-type PDUs it can receive • Key has session wide scope • Default is none • “None” works for options 1, 2, and 5 • If not “none”, is there a good default which is not implementation dependent? M. Ko

iSCSI Control-Type PDUs from the Initiator Regulated by CmdSN Window M. Ko

iSCSI Control-Type PDUs from Initiator with Known Buffering Requirement at Target • SCSI Data-out PDU • For unsolicited data, the iSER layer at the target can use FirstBurstLength to determine the amount of buffering required • Solicited data is handled using RDMA Read Response • Data cannot be “solicited” since an R2T is always transformed into an RDMA Read Request and is never sent as is by the target • Section 7.3.4 to be updated to remove references on sending “solicited” SCSI Data-out PDUs • NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff (ping echo) which is sent as a response to a NOP-in PDU from the target M. Ko

iSCSI Control-Type PDUs from the Initiator Subject to Declared Limit • Any PDU with the immediate flag set • For the PDUs listed on slide 6, the PDU is outstanding until the target responds with the corresponding response PDU • For NOP-out PDU with ITT = TTT = 0xffffffff, the PDU is outstanding until the target responds with a control-type PDU with ExpCmdSN set to at least x + 1 where x is the CmdSN of the PDU with the immediate flag set • NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff (ping echo) is not subject to the declared limit since it is sent as a response to a NOP-in PDU from the target • SNACK PDU • Not needed in iSER • If sent, PDU is outstanding until the target responds with a SCSI Response PDU for the referenced command M. Ko

Potential Problem with Overuse of Unidirectional NOP-out • When the target sends a PDU with ExpCmdSN set to x+1, up to MaxOutstandingUnexpectedPDU unidirectional NOP-out PDUs with ITT = TTT = 0xffffffff with CmdSN = x could be in flight from the initiator • Upon receiving the PDU from the target, the initiator can send MaxOutstandingUnexpectedPDU unidirectional NOP-out PDUs with CmdSN = x+1 • To guard against this pathetic case, the target needs to provision the number of buffers equal to twice MaxOutstandingUnexpectedPDU in this situation for the unexpected PDUs • The number of buffers for unexpected PDUs can be reduced to MaxOutstandingUnexpectedPDU when the initiator sends a non-immediate PDU with CmdSN = x+1 • Recommend that we add a cautionary note instead against the overuse of the unidirectional NOP-out M. Ko

iSCSI Control-Type PDUs from the Target Subject to Declared Limit • The following PDUs are outstanding until the initiator sends a control-type PDU with ExpStatSN set to at least x + 1 where x is the StatSN of the “unexpected” PDU • Asynchronous Message PDU • Reject PDU sent after the task is active • NOP-in PDU initiated by the target with ITT = TTT = 0xffffffff • A NOP-in PDU sent as a ping request by the target with ITT = 0xffffffff and TTT /= 0xffffffff is outstanding until the initiator responds with a NOP-out PDU (ping echo) M. Ko

Potential Problem with Overuse of Unidirectional NOP-in • For a NOP-in PDU from the target with ITT = TTT = 0xffffffff, StatSN is not advanced • When the initiator sends a PDU with ExpStatSN set to x+1 where x is the StatSN of this NOP-in PDU, up to MaxOutstandingUnexpectedPDU unidirectional NOP-in PDUs with ITT = TTT = 0xffffffff with StatSN = x could be in flight from the target • Upon receiving the PDU from the initiator, the target can send MaxOutstandingUnexpectedPDU unidirectional NOP-in PDUs with CmdSN = x+1 • To guard against this pathetic case, the initiator needs to provision the number of buffers equal to twice MaxOutstandingUnexpectedPDU in this situation for the unexpected PDUs • The number of buffers for unexpected PDUs can be reduced to MaxOutstandingUnexpectedPDU when the target sends a PDU with StatSN = x+1 • Recommend that we add a cautionary note instead against the overuse of the unidirectional NOP-in M. Ko

iSER Draft Status