[pnfs] [nfsv4] layout stateid handling/processing updates
William A. (Andy) Adamson
andros at citi.umich.edu
Fri Mar 21 12:56:30 EDT 2008
On Fri, Mar 21, 2008 at 12:44 PM, William A. (Andy) Adamson <
andros at citi.umich.edu> wrote:
> here is what I come up with:
>
> NFS4ERR_STALE_STATEID should apply to the layout stateid (?)
> NFS4ERR_BAD_STATEID:
> - initial open/lock/delegation stateid is not valid
> - layout stateid sequence number is 0
> - opaque portion of layout stateid changed (layout stateid can't be
> found....)
I think I missed a bad stateid case
- operation layout stateid sequence number is greater than the
server stored value.
-->Andy
>
>
> NFS4ERR_OLD_STATEID does not apply to layout stateid.
>
>
> On Fri, Mar 21, 2008 at 11:59 AM, Noveck, Dave <Dave.Noveck at netapp.com>
> wrote:
>
> > I don't get it.
> >
> > Are you saying that the processing rules that you sent out on 3/7 are
> > not enforced by the server and I can send any random seqid I choose
> > (except zero perhaps).
>
>
> The client can't use any old seqid:
>
> ---- from Spencer ----
> Once the client receives a layout stateid, it MUST use the correct
> "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations. The
> correct "seqid" is defined as the highest "seqid" value from fully
> processed responses to LAYOUTGET or LAYOUTRETURN operations or fully
> processed arguments of a CB_LAYOUTRECALL operation.
> ---------
>
> But, the server can't check that the above MUST is followed by the client.
>
>
> But if any other value is OK, why isn't zero OK.
>
>
> I guess the server can use ordering info as well?
>
> ---- from Spencer ----
> The fundemental
> requirement in client processing is that the "seqid" is used to
> provide the ordering of processing.
> ---------
>
> -->Andy
>
>
> >
> > If that is so, then you would need to say so explicitly and that
> > OLD_STATEID (or BAD_STATEID because of a seid other than zero) is not
> > returned.
> >
> > The section you mention just lists cases in which using the current
> > stateid is permissible and doesn't seem to say anything about things
> > which are impermissable.
> >
> > -----Original Message-----
> > From: Spencer Shepler [mailto:Spencer.Shepler at Sun.COM]
> > Sent: Friday, March 21, 2008 11:31 AM
> > To: NFSv4
> > Subject: Re: [nfsv4] layout stateid handling/processing updates
> >
> >
> >
> > On Mar 21, 2008, at 9:58 AM, Noveck, Dave wrote:
> >
> > > Caveat: I haven't got a chance to read Spencer's email on this subject
> > > in detail;
> > >
> > > My expectation though was that OLD_STATEID would follow the rules
> > > Spencer set out in his mail of 3/7:
> > >
> > >> The processing rules would be something like this:
> > >> If the client is sending layoutgets, the stateid.seqid for
> > >> the requests must be greater than or equal to the last value
> > >> returned for a layoutreturn. If the client is sending
> > >> layoutreturns, the stateid.seqid must be greater than or equal
> > >> to the last received layoutget or the value provided in the
> > >> cb_layoutrecall.
> > >
> > > My expectation was that if the seqid you sent did not follow the
> > > appropriate "equal or greater" clause, you would get OLD_STATEID while
> > > if it greater than the last one currently given out, you would get
> > > BAD_STATEID. Chapter 8 would say that the rules for seqid on layouts
> > > are slightly different from those for IO, opens, byte-range locks,
> > > etc.
> > > and have an xref to chapter 12 where this was discussed.
> > >
> > > Appropriate wordsmithing of the above to deal with the fact that
> > > seqids
> > > are subject to wraparound is left as an exercize for anybody who
> > > doesn't
> > > pass the buck quickly enough :-)
> >
> >
> > Having reviewed all of the layout stateid material in the I-D I came
> > to the conclusion that seqid checking/validation the server needed
> > to do was limited to what is stated in:
> > 12.5.5.2.1. Layout Recall and Return Sequencing
> > that I sent out (mainly unchanged). Otherwise, the server just
> > processes the requests as they arrive and ensures that the seqid
> > is updated for each resposne. It is then the client's responsibility
> > to sort out the responses to deal with overlapping layoutget/
> > layoutreturn
> > processing.
> >
> > My apologies for not making this clear.
> >
> > I would suggest this wording be added to the second paragraph of
> > "12.5.3. Layout Stateid"
> >
> > Note the seqid processing for layout stateid differs from the stateid
> > types. To support the parallelism desired, seqid processing rules
> > are limited to those specified in the pNFS chapters (mainly this
> > section and Section 12.5.5.2).
> >
> >
> > Spencer
> >
> >
> > > -----Original Message-----
> > > From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu]
> > > Sent: Friday, March 21, 2008 10:05 AM
> > > To: Spencer Shepler
> > > Cc: NFSv4
> > > Subject: Re: [nfsv4] layout stateid handling/processing updates
> > >
> > >
> > > What about NFS4ERR_OLD_STATEID? If the client sends parallel
> > > LAYOUTGET/RETURN operations, they could all have the same stateid. The
> > > server will process the first one, and all remaining will have old
> > > stateids. Should the server ignore this error for layout stateids?
> > >
> > > --->Andy
> > >
> > >
> > > On Wed, Mar 19, 2008 at 3:06 PM, Spencer Shepler
> > > <Spencer.Shepler at sun.com> wrote:
> > >
> > > Included is the updated/proposed text to clarify layout stateid
> > > handling.
> > > The diffs were a little difficult to read so I have included the
> > > sections
> > > that have been updated.
> > >
> > > Spencer
> > >
> > > ----
> > >
> > > 12.5.3. Layout Stateid
> > >
> > > As with all other stateids, the layout stateid consists of a
> > > "seqid"
> > > and "other" field. Once a layout stateid is changed, the "other"
> > > field will stay constant unless the stateid is revoked, or the
> > > client
> > > returns all layouts on the file and the server disposes of the
> > > stateid. The "seqid" field is initially set to one, and is never
> > > zero on any NFSv4.1 operation that uses layout stateids, whether it
> > > is fore channel or backchannel operation. After the layout stateid
> > > is established, the server increments by one the value of the
> > > "seqid"
> > > in each subsequent LAYOUTGET and LAYOUTRETURN response, and in each
> > > CB_LAYOUTRECALL request.
> > >
> > > Given the design goal of pNFS to provide parallelism, the layout
> > > stateid differs from other stateid types in that the client is
> > > expected to send LAYOUTGET and LAYTOUTRETURN operations in
> > > parallel.
> > > Since the server is incrementing the "seqid" value on each layout
> > > operation, the client may determine the order of operation
> > > processing
> > > by inspecting the "seqid" value. In the case of overlapping ranges
> > > in the client's requests, the ordering information will provide the
> > > client the knowledge of which layout ranges are held. Additional
> > > layout stateid sequencing requirements are provided in
> > > Section 12.5.5.2.
> > >
> > > Once the client receives a layout stateid, it MUST use the correct
> > > "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations. The
> > > correct "seqid" is defined as the highest "seqid" value from fully
> > > processed responses to LAYOUTGET or LAYOUTRETURN operations or
> > > fully
> > > processed arguments of a CB_LAYOUTRECALL operation.
> > >
> > > The client's receipt of a "seqid" is not sufficient for subsequent
> > > use. The client must fully process the operations before the
> > > "seqid"
> > > can be used. For LAYOUTGET results, if the client is not using the
> > > forgetful model (Section 12.5.5.1), it MUST first update its record
> > > of what ranges of the file's layout it has before using the seqid.
> > > For LAYOUTRETURN results, the client MUST delete the range from its
> > > record of what ranges of the file's layout it had before using the
> > > seqid. For CB_LAYOUTRECALL arguments, the client MUST send a
> > > response to the recall before using the seqid. The fundemental
> > > requirement in client processing is that the "seqid" is used to
> > > provide the ordering of processing. LAYOUTGET results may be
> > > processed in parallel. LAYOUTRETURN results may be processed in
> > > parallel. LAYOUTGET and LAYOUTRETURN responses may be processed in
> > > parallel as long as the ranges do not overlap. CB_LAYOUTRECALL
> > > request processing MUST be processed in "seqid" order at all times.
> > >
> > > Once a client has no more layouts on a file, the layout stateid
> > > is no
> > > longer valid, and MUST NOT be used. Any attempt to use such a
> > > layout
> > > stateid will result in NFS4ERR_BAD_STATEID.
> > >
> > >
> > > ------
> > >
> > > 12.5.5.2.1. Layout Recall and Return Sequencing
> > >
> > > One critical issue with regard to layout operations sequencing
> > > concerns callbacks. The protocol must defend against races between
> > > the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
> > > CB_LAYOUTRECALL. A client MUST NOT process a CB_LAYOUTRECALL that
> > > implies one or more outstanding LAYOUTGET or LAYOUTRETURN
> > > operations
> > > to which the client has not yet received a reply. The client
> > > detects
> > > such a CB_LAYOUTRECALL by examining the "seqid" field of the
> > > recall's
> > > layout stateid. If the "seqid" is not one higher than what the
> > > client currently has recorded, and the client has at least one
> > > LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client
> > > knows
> > > the server sent the CB_LAYOUTRECALL after sending a response to an
> > > outstanding LAYOUTGET or LAYOUTRETURN. The client MUST wait before
> > > processing such a CB_LAYOUTRECALL until it processes all replies
> > > for
> > > outstanding LAYOUTGET and LAYOUTRETURN operations for the
> > > corresponding file with seqid less than the seqid given by
> > > CB_LAYOUTRECALL (lor_stateid, see Section 20.3.)
> > >
> > > In addition to the seqid-based mechanism, Section 2.10.5.3
> > > describes
> > > the sessions mechanism for allowing the client to detect callback
> > > race conditions and delay processing such a CB_LAYOUTRECALL. The
> > > server MAY reference conflicting operations in the CB_SEQUENCE that
> > > precedes the CB_LAYOUTRECALL. Because the server has already sent
> > > replies for these operations before issuing the callback, the
> > > replies
> > > may race with the CB_LAYOUTRECALL. The client MUST wait for all
> > > the
> > > referenced calls to complete and update its view of the layout
> > > state
> > > before processing the CB_LAYOUTRECALL.
> > >
> > > 12.5.5.2.1.1. Get/Return Sequencing
> > >
> > > The protocol allows the client to send concurrent LAYOUTGET and
> > > LAYOUTRETURN operations to the server. The protocol does not
> > > provide
> > > any means for the server to process the requests in the same
> > > order in
> > > which they were created. However, through the use of the "seqid"
> > > field in the layout stateid, the client can determine the order in
> > > which parallel outstanding operations were processed by the server.
> > > Thus, when a layout retrieved by an outstanding LAYOUTGET operation
> > > intersects with a layout returned by an outstanding LAYOUTRETURN on
> > > the same file, the order in which the two conflicting operations
> > > are
> > > processed determines the final state of the overlapping layout.
> > > The
> > > order is determined by the "seqid" returned in each operation: the
> > > operation with the higher seqid was executed later.
> > >
> > > It is permissible for the client to send in parallel multiple
> > > LAYOUTGET operations for the same file or multiple LAYOUTRETURN
> > > operations for the same file, and a mix of both.
> > >
> > > It is permissilble for the client to use the current stateid (see
> > > Section 16.2.3.1.2) for LAYOUTGET operations for example when
> > > compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs. It is
> > > also permissible to use the current stateid when compounding
> > > LAYOUTRETURNs.
> > >
> > > It is permissible for the client to current stateid when combining
> > > LAYOUTRETURN and LAYOUTGET operations for the same file in the same
> > > COMPOUND request since the server MUST process these in order.
> > > However, if a client does send such COMPOUND requests, it MUST NOT
> > > have more than one outstanding for the same file at the same time
> > > and
> > > MUST NOT have other LAYOUTGET or LAYOUTRETURN operations
> > > outstanding
> > > at the same time for that same file.
> > >
> > > ----
> > >
> > > 12.5.5.2.1.2. Client Considerations
> > >
> > > Consider a pNFS client that has sent a LAYOUTGET and before it
> > > receives the reply to LAYOUTGET, it receives a CB_LAYOUTRECALL for
> > > the same file with an overlapping range. There are two
> > > possibilities, which the client can distinguish via the layout
> > > stateid in the recall.
> > >
> > > 1. The server processed the LAYOUTGET before issuing the
> > > recall, so
> > > the LAYOUTGET must be waited for because it may be carrying
> > > layout information that will need to be returned to deal with
> > > the
> > > CB_LAYOUTRECALL.
> > >
> > > 2. The server sent the callback before receiving the LAYOUTGET.
> > > The
> > > server will not respond to the LAYOUTGET until the
> > > CB_LAYOUTRECALL is processed.
> > >
> > > If these possibilities cannot be distinguished, a deadlock could
> > > result, as the client must wait for the LAYOUTGET response before
> > > processing the recall in the first case, but that response will not
> > > arrive until after the recall is processed in the second case.
> > > Note
> > > that in the first case, the "seqid" in the layout stateid of the
> > > recall is two greater than what the client has recorded and in the
> > > second case, the "seqid" is one greater than what the client has
> > > recorded. This allows the client to disambiguate between the two
> > > cases. The client thus knows precisely which possibility applies.
> > >
> > > In case 1 the client knows it needs to wait for the LAYOUTGET
> > > response before processing the recall (or the client can return
> > > NFS4ERR_DELAY).
> > >
> > > In case 2 the client will not wait for the LAYOUTGET response
> > > before
> > > processing the recall, because waiting would cause deadlock.
> > > Therefore, the action at the client will only require waiting in
> > > the
> > > case that the client has not yet seen the server's earlier
> > > responses
> > > to the LAYOUTGET operation(s).
> > >
> > > The recall process can be considered completed when the final
> > > LAYOUTRETURN operation for the recalled range is completed. The
> > > LAYOUTRETURN uses the layout stateid (with seqid) specified in
> > > CB_LAYOUTRECALL. If the client uses multiple LAYOUTRETURNs in
> > > processing the recall, the first LAYOUTRETURN will use the layout
> > > stateid as specified in CB_LAYOUTRECALL. Subsequent LAYOUTRETURNs
> > > will use the highest seqid as is the usual case.
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > nfsv4 at ietf.org
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> > _______________________________________________
> > nfsv4 mailing list
> > nfsv4 at ietf.org
> > https://www.ietf.org/mailman/listinfo/nfsv4
> > _______________________________________________
> > nfsv4 mailing list
> > nfsv4 at ietf.org
> > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://linux-nfs.org/pipermail/pnfs/attachments/20080321/ef078a73/attachment-0001.htm
More information about the pNFS
mailing list