[pnfs] [nfsv4] layout stateid handling/processing updates
William A. (Andy) Adamson
andros at citi.umich.edu
Fri Mar 21 13:42:00 EDT 2008
On Fri, Mar 21, 2008 at 1:32 PM, Noveck, Dave <Dave.Noveck at netapp.com>
wrote:
> > I think I missed a bad stateid case
> >
> > - operation layout stateid sequence number is greater than the
> server stored value.
>
> Fun with wraparound. I assume this only applies if the seqid has nto
> wrapped around
good point.
> I think you could enforce this, but it seems odd to just pick this one
> constraint to enforce. If the server does not use the value, and I
> don't see where it does, why put this constarint on it, jsut so the
> server can, in this one case, say "gotcha".
The reason comes from Spencer noting that the client must send a layout
stateid.seqid that was valid in the past, which in turn comes from section
12.5.5.2.1.3 which describes the server processing of the operation
stateid.seqid WRT the CB_LAYOUTRECALL layout stateid.seqid. This is where
the server uses the seqid value.
-->Andy
>
>
> -----Original Message-----
> From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu]
> Sent: Friday, March 21, 2008 12:57 PM
> To: Noveck, Dave
> Cc: pnfs at linux-nfs.org
> Subject: Re: [nfsv4] layout stateid handling/processing updates
>
>
>
>
>
> On Fri, Mar 21, 2008 at 12:44 PM, William A. (Andy) Adamson
> <andros at citi.umich.edu> wrote:
>
> here is what I come up with:
>
> NFS4ERR_STALE_STATEID should apply to the layout stateid (?)
> NFS4ERR_BAD_STATEID:
> - initial open/lock/delegation stateid is not valid
> - layout stateid sequence number is 0
> - opaque portion of layout stateid changed (layout stateid can't be
> found....)
>
> I think I missed a bad stateid case
>
> - operation layout stateid sequence number is greater than the
> server stored value.
>
>
> -->Andy
>
>
>
>
> NFS4ERR_OLD_STATEID does not apply to layout stateid.
>
>
>
> On Fri, Mar 21, 2008 at 11:59 AM, Noveck, Dave <Dave.Noveck at netapp.com>
> wrote:
>
> I don't get it.
>
> Are you saying that the processing rules that you sent out on 3/7 are
> not enforced by the server and I can send any random seqid I choose
> (except zero perhaps).
>
> The client can't use any old seqid:
>
>
> ---- from Spencer ----
>
> Once the client receives a layout stateid, it MUST use the correct
> "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations. The
> correct "seqid" is defined as the highest "seqid" value from fully
> processed responses to LAYOUTGET or LAYOUTRETURN operations or fully
> processed arguments of a CB_LAYOUTRECALL operation.
>
> ---------
>
> But, the server can't check that the above MUST is followed by the
> client.
>
>
>
> But if any other value is OK, why isn't zero OK.
>
> I guess the server can use ordering info as well?
>
>
> ---- from Spencer ----
>
> The fundemental
> requirement in client processing is that the "seqid" is used to
> provide the ordering of processing.
>
> ---------
>
> -->Andy
>
>
>
>
> If that is so, then you would need to say so explicitly and that
> OLD_STATEID (or BAD_STATEID because of a seid other than zero) is not
> returned.
>
> The section you mention just lists cases in which using the current
> stateid is permissible and doesn't seem to say anything about things
> which are impermissable.
>
>
> -----Original Message-----
> From: Spencer Shepler [mailto:Spencer.Shepler at Sun.COM]
> Sent: Friday, March 21, 2008 11:31 AM
> To: NFSv4
> Subject: Re: [nfsv4] layout stateid handling/processing updates
>
>
>
> On Mar 21, 2008, at 9:58 AM, Noveck, Dave wrote:
>
> > Caveat: I haven't got a chance to read Spencer's email on this subject
> > in detail;
> >
> > My expectation though was that OLD_STATEID would follow the rules
> > Spencer set out in his mail of 3/7:
> >
> >> The processing rules would be something like this:
> >> If the client is sending layoutgets, the stateid.seqid for
> >> the requests must be greater than or equal to the last value
> >> returned for a layoutreturn. If the client is sending
> >> layoutreturns, the stateid.seqid must be greater than or equal
> >> to the last received layoutget or the value provided in the
> >> cb_layoutrecall.
> >
> > My expectation was that if the seqid you sent did not follow the
> > appropriate "equal or greater" clause, you would get OLD_STATEID while
> > if it greater than the last one currently given out, you would get
> > BAD_STATEID. Chapter 8 would say that the rules for seqid on layouts
> > are slightly different from those for IO, opens, byte-range locks,
> > etc.
> > and have an xref to chapter 12 where this was discussed.
> >
> > Appropriate wordsmithing of the above to deal with the fact that
> > seqids
> > are subject to wraparound is left as an exercize for anybody who
> > doesn't
> > pass the buck quickly enough :-)
>
>
> Having reviewed all of the layout stateid material in the I-D I came
> to the conclusion that seqid checking/validation the server needed
> to do was limited to what is stated in:
> 12.5.5.2.1. Layout Recall and Return Sequencing
> that I sent out (mainly unchanged). Otherwise, the server just
> processes the requests as they arrive and ensures that the seqid
> is updated for each resposne. It is then the client's responsibility
> to sort out the responses to deal with overlapping layoutget/
> layoutreturn
> processing.
>
> My apologies for not making this clear.
>
> I would suggest this wording be added to the second paragraph of
> "12.5.3. Layout Stateid"
>
> Note the seqid processing for layout stateid differs from the stateid
> types. To support the parallelism desired, seqid processing rules
> are limited to those specified in the pNFS chapters (mainly this
> section and Section 12.5.5.2).
>
>
> Spencer
>
>
> > -----Original Message-----
> > From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu]
> > Sent: Friday, March 21, 2008 10:05 AM
> > To: Spencer Shepler
> > Cc: NFSv4
> > Subject: Re: [nfsv4] layout stateid handling/processing updates
> >
> >
> > What about NFS4ERR_OLD_STATEID? If the client sends parallel
> > LAYOUTGET/RETURN operations, they could all have the same stateid. The
> > server will process the first one, and all remaining will have old
> > stateids. Should the server ignore this error for layout stateids?
> >
> > --->Andy
> >
> >
> > On Wed, Mar 19, 2008 at 3:06 PM, Spencer Shepler
> > <Spencer.Shepler at sun.com> wrote:
> >
> > Included is the updated/proposed text to clarify layout stateid
> > handling.
> > The diffs were a little difficult to read so I have included the
> > sections
> > that have been updated.
> >
> > Spencer
> >
> > ----
> >
> > 12.5.3. Layout Stateid
> >
> > As with all other stateids, the layout stateid consists of a
> > "seqid"
> > and "other" field. Once a layout stateid is changed, the "other"
> > field will stay constant unless the stateid is revoked, or the
> > client
> > returns all layouts on the file and the server disposes of the
> > stateid. The "seqid" field is initially set to one, and is never
> > zero on any NFSv4.1 operation that uses layout stateids, whether it
> > is fore channel or backchannel operation. After the layout stateid
> > is established, the server increments by one the value of the
> > "seqid"
> > in each subsequent LAYOUTGET and LAYOUTRETURN response, and in each
> > CB_LAYOUTRECALL request.
> >
> > Given the design goal of pNFS to provide parallelism, the layout
> > stateid differs from other stateid types in that the client is
> > expected to send LAYOUTGET and LAYTOUTRETURN operations in
> > parallel.
> > Since the server is incrementing the "seqid" value on each layout
> > operation, the client may determine the order of operation
> > processing
> > by inspecting the "seqid" value. In the case of overlapping ranges
> > in the client's requests, the ordering information will provide the
> > client the knowledge of which layout ranges are held. Additional
> > layout stateid sequencing requirements are provided in
> > Section 12.5.5.2.
> >
> > Once the client receives a layout stateid, it MUST use the correct
> > "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations. The
> > correct "seqid" is defined as the highest "seqid" value from fully
> > processed responses to LAYOUTGET or LAYOUTRETURN operations or
> > fully
> > processed arguments of a CB_LAYOUTRECALL operation.
> >
> > The client's receipt of a "seqid" is not sufficient for subsequent
> > use. The client must fully process the operations before the
> > "seqid"
> > can be used. For LAYOUTGET results, if the client is not using the
> > forgetful model (Section 12.5.5.1), it MUST first update its record
> > of what ranges of the file's layout it has before using the seqid.
> > For LAYOUTRETURN results, the client MUST delete the range from its
> > record of what ranges of the file's layout it had before using the
> > seqid. For CB_LAYOUTRECALL arguments, the client MUST send a
> > response to the recall before using the seqid. The fundemental
> > requirement in client processing is that the "seqid" is used to
> > provide the ordering of processing. LAYOUTGET results may be
> > processed in parallel. LAYOUTRETURN results may be processed in
> > parallel. LAYOUTGET and LAYOUTRETURN responses may be processed in
> > parallel as long as the ranges do not overlap. CB_LAYOUTRECALL
> > request processing MUST be processed in "seqid" order at all times.
> >
> > Once a client has no more layouts on a file, the layout stateid
> > is no
> > longer valid, and MUST NOT be used. Any attempt to use such a
> > layout
> > stateid will result in NFS4ERR_BAD_STATEID.
> >
> >
> > ------
> >
> > 12.5.5.2.1. Layout Recall and Return Sequencing
> >
> > One critical issue with regard to layout operations sequencing
> > concerns callbacks. The protocol must defend against races between
> > the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
> > CB_LAYOUTRECALL. A client MUST NOT process a CB_LAYOUTRECALL that
> > implies one or more outstanding LAYOUTGET or LAYOUTRETURN
> > operations
> > to which the client has not yet received a reply. The client
> > detects
> > such a CB_LAYOUTRECALL by examining the "seqid" field of the
> > recall's
> > layout stateid. If the "seqid" is not one higher than what the
> > client currently has recorded, and the client has at least one
> > LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client
> > knows
> > the server sent the CB_LAYOUTRECALL after sending a response to an
> > outstanding LAYOUTGET or LAYOUTRETURN. The client MUST wait before
> > processing such a CB_LAYOUTRECALL until it processes all replies
> > for
> > outstanding LAYOUTGET and LAYOUTRETURN operations for the
> > corresponding file with seqid less than the seqid given by
> > CB_LAYOUTRECALL (lor_stateid, see Section 20.3.)
> >
> > In addition to the seqid-based mechanism, Section 2.10.5.3
> > describes
> > the sessions mechanism for allowing the client to detect callback
> > race conditions and delay processing such a CB_LAYOUTRECALL. The
> > server MAY reference conflicting operations in the CB_SEQUENCE that
> > precedes the CB_LAYOUTRECALL. Because the server has already sent
> > replies for these operations before issuing the callback, the
> > replies
> > may race with the CB_LAYOUTRECALL. The client MUST wait for all
> > the
> > referenced calls to complete and update its view of the layout
> > state
> > before processing the CB_LAYOUTRECALL.
> >
> > 12.5.5.2.1.1. Get/Return Sequencing
> >
> > The protocol allows the client to send concurrent LAYOUTGET and
> > LAYOUTRETURN operations to the server. The protocol does not
> > provide
> > any means for the server to process the requests in the same
> > order in
> > which they were created. However, through the use of the "seqid"
> > field in the layout stateid, the client can determine the order in
> > which parallel outstanding operations were processed by the server.
> > Thus, when a layout retrieved by an outstanding LAYOUTGET operation
> > intersects with a layout returned by an outstanding LAYOUTRETURN on
> > the same file, the order in which the two conflicting operations
> > are
> > processed determines the final state of the overlapping layout.
> > The
> > order is determined by the "seqid" returned in each operation: the
> > operation with the higher seqid was executed later.
> >
> > It is permissible for the client to send in parallel multiple
> > LAYOUTGET operations for the same file or multiple LAYOUTRETURN
> > operations for the same file, and a mix of both.
> >
> > It is permissilble for the client to use the current stateid (see
> > Section 16.2.3.1.2) for LAYOUTGET operations for example when
> > compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs. It is
> > also permissible to use the current stateid when compounding
> > LAYOUTRETURNs.
> >
> > It is permissible for the client to current stateid when combining
> > LAYOUTRETURN and LAYOUTGET operations for the same file in the same
> > COMPOUND request since the server MUST process these in order.
> > However, if a client does send such COMPOUND requests, it MUST NOT
> > have more than one outstanding for the same file at the same time
> > and
> > MUST NOT have other LAYOUTGET or LAYOUTRETURN operations
> > outstanding
> > at the same time for that same file.
> >
> > ----
> >
> > 12.5.5.2.1.2. Client Considerations
> >
> > Consider a pNFS client that has sent a LAYOUTGET and before it
> > receives the reply to LAYOUTGET, it receives a CB_LAYOUTRECALL for
> > the same file with an overlapping range. There are two
> > possibilities, which the client can distinguish via the layout
> > stateid in the recall.
> >
> > 1. The server processed the LAYOUTGET before issuing the
> > recall, so
> > the LAYOUTGET must be waited for because it may be carrying
> > layout information that will need to be returned to deal with
> > the
> > CB_LAYOUTRECALL.
> >
> > 2. The server sent the callback before receiving the LAYOUTGET.
> > The
> > server will not respond to the LAYOUTGET until the
> > CB_LAYOUTRECALL is processed.
> >
> > If these possibilities cannot be distinguished, a deadlock could
> > result, as the client must wait for the LAYOUTGET response before
> > processing the recall in the first case, but that response will not
> > arrive until after the recall is processed in the second case.
> > Note
> > that in the first case, the "seqid" in the layout stateid of the
> > recall is two greater than what the client has recorded and in the
> > second case, the "seqid" is one greater than what the client has
> > recorded. This allows the client to disambiguate between the two
> > cases. The client thus knows precisely which possibility applies.
> >
> > In case 1 the client knows it needs to wait for the LAYOUTGET
> > response before processing the recall (or the client can return
> > NFS4ERR_DELAY).
> >
> > In case 2 the client will not wait for the LAYOUTGET response
> > before
> > processing the recall, because waiting would cause deadlock.
> > Therefore, the action at the client will only require waiting in
> > the
> > case that the client has not yet seen the server's earlier
> > responses
> > to the LAYOUTGET operation(s).
> >
> > The recall process can be considered completed when the final
> > LAYOUTRETURN operation for the recalled range is completed. The
> > LAYOUTRETURN uses the layout stateid (with seqid) specified in
> > CB_LAYOUTRECALL. If the client uses multiple LAYOUTRETURNs in
> > processing the recall, the first LAYOUTRETURN will use the layout
> > stateid as specified in CB_LAYOUTRECALL. Subsequent LAYOUTRETURNs
> > will use the highest seqid as is the usual case.
> >
> > _______________________________________________
> > nfsv4 mailing list
> > nfsv4 at ietf.org
> > https://www.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> nfsv4 at ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
> _______________________________________________
> nfsv4 mailing list
> nfsv4 at ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://linux-nfs.org/pipermail/pnfs/attachments/20080321/886f7c93/attachment-0001.htm
More information about the pNFS
mailing list