[pnfs] [nfsv4] layout stateid handling/processing updates

[Noveck, Dave]

> I think I missed a bad stateid case
>
>       - operation layout stateid sequence number is greater than the
server stored value.

Fun with wraparound.  I assume this only applies if the seqid has nto
wrapped around.

I think you could enforce this, but it seems odd to just pick this one
constraint to enforce.  If the server does not use the value, and I
don't see where it does, why put this constarint on it, jsut so the
server can, in this one case, say "gotcha".

-----Original Message-----
From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu] 
Sent: Friday, March 21, 2008 12:57 PM
To: Noveck, Dave
Cc: pnfs at linux-nfs.org
Subject: Re: [nfsv4] layout stateid handling/processing updates





On Fri, Mar 21, 2008 at 12:44 PM, William A. (Andy) Adamson
<andros at citi.umich.edu> wrote:

here is what I come up with:

NFS4ERR_STALE_STATEID should apply to the layout stateid (?)
NFS4ERR_BAD_STATEID:
     - initial open/lock/delegation stateid is not valid
     - layout stateid sequence number is 0
     - opaque portion of layout stateid changed (layout stateid can't be
found....)

I think I missed a bad stateid case

        - operation layout stateid sequence number is greater than the
server stored value.


-->Andy
        



NFS4ERR_OLD_STATEID does not apply to layout stateid.



On Fri, Mar 21, 2008 at 11:59 AM, Noveck, Dave <Dave.Noveck at netapp.com>
wrote:

I don't get it.

Are you saying that the processing rules that you sent out on 3/7 are
not enforced by the server and I can send any random seqid I choose
(except zero perhaps).

The client can't use any old seqid:


---- from Spencer ----

  Once the client receives a layout stateid, it MUST use the correct
   "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations.  The
   correct "seqid" is defined as the highest "seqid" value from fully
   processed responses to LAYOUTGET or LAYOUTRETURN operations or fully
   processed arguments of a CB_LAYOUTRECALL operation.

---------

But, the server can't check that the above MUST is followed by the
client.



 But if any other value is OK, why isn't zero OK.

I guess the server can use ordering info as well?


---- from Spencer ----

   The fundemental
   requirement in client processing is that the "seqid" is used to
   provide the ordering of processing.  

---------

-->Andy




If that is so, then you would need to say so explicitly and that
OLD_STATEID (or BAD_STATEID because of a seid other than zero) is not
returned.

The section you mention just lists cases in which using the current
stateid is permissible and doesn't seem to say anything about things
which are impermissable.


-----Original Message-----
From: Spencer Shepler [mailto:Spencer.Shepler at Sun.COM]
Sent: Friday, March 21, 2008 11:31 AM
To: NFSv4
Subject: Re: [nfsv4] layout stateid handling/processing updates



On Mar 21, 2008, at 9:58 AM, Noveck, Dave wrote:

> Caveat: I haven't got a chance to read Spencer's email on this subject
> in detail;
>
> My expectation though was that OLD_STATEID would follow the rules
> Spencer set out in his mail of 3/7:
>
>> The processing rules would be something like this:
>> If the client is sending layoutgets, the stateid.seqid for
>> the requests must be greater than or equal to the last value
>> returned for a layoutreturn.  If the client is sending
>> layoutreturns, the stateid.seqid must be greater than or equal
>> to the last received layoutget or the value provided in the
>> cb_layoutrecall.
>
> My expectation was that if the seqid you sent did not follow the
> appropriate "equal or greater" clause, you would get OLD_STATEID while
> if it greater than the last one currently given out, you would get
> BAD_STATEID.  Chapter 8 would say that the rules for seqid on layouts
> are slightly different from those for IO, opens, byte-range locks,
> etc.
> and have an xref to chapter 12 where this was discussed.
>
> Appropriate wordsmithing of the above to deal with the fact that
> seqids
> are subject to wraparound is left as an exercize for anybody who
> doesn't
> pass the buck quickly enough :-)


Having reviewed all of the layout stateid material in the I-D I came
to the conclusion that seqid checking/validation the server needed
to do was limited to what is stated in:
       12.5.5.2.1.  Layout Recall and Return Sequencing
that I sent out (mainly unchanged).  Otherwise, the server just
processes the requests as they arrive and ensures that the seqid
is updated for each resposne.  It is then the client's responsibility
to sort out the responses to deal with overlapping layoutget/
layoutreturn
processing.

My apologies for not making this clear.

I would suggest this wording be added to the second paragraph of
"12.5.3.  Layout Stateid"

  Note the seqid processing for layout stateid differs from the stateid
  types.  To support the parallelism desired, seqid processing rules
  are limited to those specified in the pNFS chapters (mainly this
  section and Section 12.5.5.2).


Spencer


> -----Original Message-----
> From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu]
> Sent: Friday, March 21, 2008 10:05 AM
> To: Spencer Shepler
> Cc: NFSv4
> Subject: Re: [nfsv4] layout stateid handling/processing updates
>
>
> What about NFS4ERR_OLD_STATEID? If the client sends parallel
> LAYOUTGET/RETURN operations, they could all have the same stateid. The
> server will process the first one, and all remaining will have old
> stateids. Should the server ignore this error for layout stateids?
>
> --->Andy
>
>
> On Wed, Mar 19, 2008 at 3:06 PM, Spencer Shepler
> <Spencer.Shepler at sun.com> wrote:
>
> Included is the updated/proposed text to clarify layout stateid
> handling.
> The diffs were a little difficult to read so I have included the
> sections
> that have been updated.
>
> Spencer
>
> ----
>
> 12.5.3.  Layout Stateid
>
>    As with all other stateids, the layout stateid consists of a
> "seqid"
>    and "other" field.  Once a layout stateid is changed, the "other"
>    field will stay constant unless the stateid is revoked, or the
> client
>    returns all layouts on the file and the server disposes of the
>    stateid.  The "seqid" field is initially set to one, and is never
>    zero on any NFSv4.1 operation that uses layout stateids, whether it
>    is fore channel or backchannel operation.  After the layout stateid
>    is established, the server increments by one the value of the
> "seqid"
>    in each subsequent LAYOUTGET and LAYOUTRETURN response, and in each
>    CB_LAYOUTRECALL request.
>
>    Given the design goal of pNFS to provide parallelism, the layout
>    stateid differs from other stateid types in that the client is
>    expected to send LAYOUTGET and LAYTOUTRETURN operations in
> parallel.
>    Since the server is incrementing the "seqid" value on each layout
>    operation, the client may determine the order of operation
> processing
>    by inspecting the "seqid" value.  In the case of overlapping ranges
>    in the client's requests, the ordering information will provide the
>    client the knowledge of which layout ranges are held.  Additional
>    layout stateid sequencing requirements are provided in
>    Section 12.5.5.2.
>
>    Once the client receives a layout stateid, it MUST use the correct
>    "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations.  The
>    correct "seqid" is defined as the highest "seqid" value from fully
>    processed responses to LAYOUTGET or LAYOUTRETURN operations or
> fully
>    processed arguments of a CB_LAYOUTRECALL operation.
>
>    The client's receipt of a "seqid" is not sufficient for subsequent
>    use.  The client must fully process the operations before the
> "seqid"
>    can be used.  For LAYOUTGET results, if the client is not using the
>    forgetful model (Section 12.5.5.1), it MUST first update its record
>    of what ranges of the file's layout it has before using the seqid.
>    For LAYOUTRETURN results, the client MUST delete the range from its
>    record of what ranges of the file's layout it had before using the
>    seqid.  For CB_LAYOUTRECALL arguments, the client MUST send a
>    response to the recall before using the seqid.  The fundemental
>    requirement in client processing is that the "seqid" is used to
>    provide the ordering of processing.  LAYOUTGET results may be
>    processed in parallel.  LAYOUTRETURN results may be processed in
>    parallel.  LAYOUTGET and LAYOUTRETURN responses may be processed in
>    parallel as long as the ranges do not overlap.  CB_LAYOUTRECALL
>    request processing MUST be processed in "seqid" order at all times.
>
>    Once a client has no more layouts on a file, the layout stateid
> is no
>    longer valid, and MUST NOT be used.  Any attempt to use such a
> layout
>    stateid will result in NFS4ERR_BAD_STATEID.
>
>
> ------
>
> 12.5.5.2.1.  Layout Recall and Return Sequencing
>
>    One critical issue with regard to layout operations sequencing
>    concerns callbacks.  The protocol must defend against races between
>    the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
>    CB_LAYOUTRECALL.  A client MUST NOT process a CB_LAYOUTRECALL that
>    implies one or more outstanding LAYOUTGET or LAYOUTRETURN
> operations
>    to which the client has not yet received a reply.  The client
> detects
>    such a CB_LAYOUTRECALL by examining the "seqid" field of the
> recall's
>    layout stateid.  If the "seqid" is not one higher than what the
>    client currently has recorded, and the client has at least one
>    LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client
> knows
>    the server sent the CB_LAYOUTRECALL after sending a response to an
>    outstanding LAYOUTGET or LAYOUTRETURN.  The client MUST wait before
>    processing such a CB_LAYOUTRECALL until it processes all replies
> for
>    outstanding LAYOUTGET and LAYOUTRETURN operations for the
>    corresponding file with seqid less than the seqid given by
>    CB_LAYOUTRECALL (lor_stateid, see Section 20.3.)
>
>    In addition to the seqid-based mechanism, Section 2.10.5.3
> describes
>    the sessions mechanism for allowing the client to detect callback
>    race conditions and delay processing such a CB_LAYOUTRECALL.  The
>    server MAY reference conflicting operations in the CB_SEQUENCE that
>    precedes the CB_LAYOUTRECALL.  Because the server has already sent
>    replies for these operations before issuing the callback, the
> replies
>    may race with the CB_LAYOUTRECALL.  The client MUST wait for all
> the
>    referenced calls to complete and update its view of the layout
> state
>    before processing the CB_LAYOUTRECALL.
>
> 12.5.5.2.1.1.  Get/Return Sequencing
>
>    The protocol allows the client to send concurrent LAYOUTGET and
>    LAYOUTRETURN operations to the server.  The protocol does not
> provide
>    any means for the server to process the requests in the same
> order in
>    which they were created.  However, through the use of the "seqid"
>    field in the layout stateid, the client can determine the order in
>    which parallel outstanding operations were processed by the server.
>    Thus, when a layout retrieved by an outstanding LAYOUTGET operation
>    intersects with a layout returned by an outstanding LAYOUTRETURN on
>    the same file, the order in which the two conflicting operations
> are
>    processed determines the final state of the overlapping layout.
> The
>    order is determined by the "seqid" returned in each operation: the
>    operation with the higher seqid was executed later.
>
>    It is permissible for the client to send in parallel multiple
>    LAYOUTGET operations for the same file or multiple LAYOUTRETURN
>    operations for the same file, and a mix of both.
>
>    It is permissilble for the client to use the current stateid (see
>    Section 16.2.3.1.2) for LAYOUTGET operations for example when
>    compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs.  It is
>    also permissible to use the current stateid when compounding
>    LAYOUTRETURNs.
>
>    It is permissible for the client to current stateid when combining
>    LAYOUTRETURN and LAYOUTGET operations for the same file in the same
>    COMPOUND request since the server MUST process these in order.
>    However, if a client does send such COMPOUND requests, it MUST NOT
>    have more than one outstanding for the same file at the same time
> and
>    MUST NOT have other LAYOUTGET or LAYOUTRETURN operations
> outstanding
>    at the same time for that same file.
>
> ----
>
> 12.5.5.2.1.2.  Client Considerations
>
>    Consider a pNFS client that has sent a LAYOUTGET and before it
>    receives the reply to LAYOUTGET, it receives a CB_LAYOUTRECALL for
>    the same file with an overlapping range.  There are two
>    possibilities, which the client can distinguish via the layout
>    stateid in the recall.
>
>    1.  The server processed the LAYOUTGET before issuing the
> recall, so
>        the LAYOUTGET must be waited for because it may be carrying
>        layout information that will need to be returned to deal with
> the
>        CB_LAYOUTRECALL.
>
>    2.  The server sent the callback before receiving the LAYOUTGET.
> The
>        server will not respond to the LAYOUTGET until the
>        CB_LAYOUTRECALL is processed.
>
>    If these possibilities cannot be distinguished, a deadlock could
>    result, as the client must wait for the LAYOUTGET response before
>    processing the recall in the first case, but that response will not
>    arrive until after the recall is processed in the second case.
> Note
>    that in the first case, the "seqid" in the layout stateid of the
>    recall is two greater than what the client has recorded and in the
>    second case, the "seqid" is one greater than what the client has
>    recorded.  This allows the client to disambiguate between the two
>    cases.  The client thus knows precisely which possibility applies.
>
>    In case 1 the client knows it needs to wait for the LAYOUTGET
>    response before processing the recall (or the client can return
>    NFS4ERR_DELAY).
>
>    In case 2 the client will not wait for the LAYOUTGET response
> before
>    processing the recall, because waiting would cause deadlock.
>    Therefore, the action at the client will only require waiting in
> the
>    case that the client has not yet seen the server's earlier
> responses
>    to the LAYOUTGET operation(s).
>
>    The recall process can be considered completed when the final
>    LAYOUTRETURN operation for the recalled range is completed.  The
>    LAYOUTRETURN uses the layout stateid (with seqid) specified in
>    CB_LAYOUTRECALL.  If the client uses multiple LAYOUTRETURNs in
>    processing the recall, the first LAYOUTRETURN will use the layout
>    stateid as specified in CB_LAYOUTRECALL.  Subsequent LAYOUTRETURNs
>    will use the highest seqid as is the usual case.
>
> _______________________________________________
> nfsv4 mailing list
> nfsv4 at ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
nfsv4 at ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4
_______________________________________________
nfsv4 mailing list
nfsv4 at ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4