[pnfs] [nfsv4] layout stateid handling/processing updates
Benny Halevy
bhalevy at panasas.com
Mon Mar 24 07:56:56 EDT 2008
Here the current definitions of the relevant error codes:
15.1.5.2. NFS4ERR_BAD_STATEID (Error Code 10026)
A stateid does not properly designate any valid state. See
Section 8.2.4 and Section 8.2.3 for a discussion of how stateids are
validated.
15.1.5.5. NFS4ERR_OLD_STATEID (Error Code 10024)
A stateid with a non-zero seqid value does match the current seqid
for the state designated by the user.
15.1.16.5. NFS4ERR_STALE_STATEID (Error Code 10023)
A stateid generated by an earlier server instance was used.
I think that NFS4ERR_STALE_STATEID and NFS4ERR_BAD_STATEID are appropriate
if the the client uses the wrong opaque part of the layout stateid.
NFS4ERR_OLD_STATEID makes sense to return for layout ops that were sent
before a CB_LAYOUTRECALL and are processed after it. Wraparound processing
is important here to determine what is "before" or "after" a particular seqid.
To prevent a situation where LAYOUTRETURNs sent before a CB_LAYOUTRECALL are
mistaken to be in response to the CB_LAYOUTRECALL the server shouldn't just
accept any seqid. During normal operation, without any layout recalls,
the server must allow only seqids which are in the range of the current seqid
(as maintained by the server) minus some threshold to the current seqid, all modulo
|seqid| (2^32). If the seqid is too old or too new the server can return either
NFS4ERR_OLD_STATEID or NFS4ERR_BAD_STATEID, respectively.
Benny
On Mar. 21, 2008, 19:42 +0200, "William A. (Andy) Adamson" <andros at citi.umich.edu> wrote:
>
> On Fri, Mar 21, 2008 at 1:32 PM, Noveck, Dave <Dave.Noveck at netapp.com> wrote:
>
>
> > I think I missed a bad stateid case
> >
> > - operation layout stateid sequence number is greater than the
> server stored value.
>
>
> Fun with wraparound. I assume this only applies if the seqid has nto
> wrapped around
>
>
> good point.
>
>
>
>
> I think you could enforce this, but it seems odd to just pick this one
> constraint to enforce. If the server does not use the value, and I
> don't see where it does, why put this constarint on it, jsut so the
> server can, in this one case, say "gotcha".
>
>
> The reason comes from Spencer noting that the client must send a layout stateid.seqid that was valid in the past, which in turn comes from section 12.5.5.2.1.3 which describes the server processing of the operation stateid.seqid WRT the CB_LAYOUTRECALL layout stateid.seqid. This is where the server uses the seqid value.
>
> -->Andy
>
>
>
>
> -----Original Message-----
> From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu]
>
> Sent: Friday, March 21, 2008 12:57 PM
> To: Noveck, Dave
> Cc: pnfs at linux-nfs.org
> Subject: Re: [nfsv4] layout stateid handling/processing updates
>
>
>
>
>
> On Fri, Mar 21, 2008 at 12:44 PM, William A. (Andy) Adamson
> <andros at citi.umich.edu> wrote:
>
> here is what I come up with:
>
> NFS4ERR_STALE_STATEID should apply to the layout stateid (?)
> NFS4ERR_BAD_STATEID:
> - initial open/lock/delegation stateid is not valid
> - layout stateid sequence number is 0
> - opaque portion of layout stateid changed (layout stateid can't be
> found....)
>
> I think I missed a bad stateid case
>
> - operation layout stateid sequence number is greater than the
> server stored value.
>
>
> -->Andy
>
>
>
>
> NFS4ERR_OLD_STATEID does not apply to layout stateid.
>
>
>
> On Fri, Mar 21, 2008 at 11:59 AM, Noveck, Dave <Dave.Noveck at netapp.com>
> wrote:
>
> I don't get it.
>
> Are you saying that the processing rules that you sent out on 3/7 are
> not enforced by the server and I can send any random seqid I choose
> (except zero perhaps).
>
> The client can't use any old seqid:
>
>
> ---- from Spencer ----
>
> Once the client receives a layout stateid, it MUST use the correct
> "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations. The
> correct "seqid" is defined as the highest "seqid" value from fully
> processed responses to LAYOUTGET or LAYOUTRETURN operations or fully
> processed arguments of a CB_LAYOUTRECALL operation.
>
> ---------
>
> But, the server can't check that the above MUST is followed by the
> client.
>
>
>
> But if any other value is OK, why isn't zero OK.
>
> I guess the server can use ordering info as well?
>
>
> ---- from Spencer ----
>
> The fundemental
> requirement in client processing is that the "seqid" is used to
> provide the ordering of processing.
>
> ---------
>
> -->Andy
>
>
>
>
> If that is so, then you would need to say so explicitly and that
> OLD_STATEID (or BAD_STATEID because of a seid other than zero) is not
> returned.
>
> The section you mention just lists cases in which using the current
> stateid is permissible and doesn't seem to say anything about things
> which are impermissable.
>
>
> -----Original Message-----
> From: Spencer Shepler [mailto:Spencer.Shepler at Sun.COM]
> Sent: Friday, March 21, 2008 11:31 AM
> To: NFSv4
> Subject: Re: [nfsv4] layout stateid handling/processing updates
>
>
>
> On Mar 21, 2008, at 9:58 AM, Noveck, Dave wrote:
>
> > Caveat: I haven't got a chance to read Spencer's email on this subject
> > in detail;
> >
> > My expectation though was that OLD_STATEID would follow the rules
> > Spencer set out in his mail of 3/7:
> >
> >> The processing rules would be something like this:
> >> If the client is sending layoutgets, the stateid.seqid for
> >> the requests must be greater than or equal to the last value
> >> returned for a layoutreturn. If the client is sending
> >> layoutreturns, the stateid.seqid must be greater than or equal
> >> to the last received layoutget or the value provided in the
> >> cb_layoutrecall.
> >
> > My expectation was that if the seqid you sent did not follow the
> > appropriate "equal or greater" clause, you would get OLD_STATEID while
> > if it greater than the last one currently given out, you would get
> > BAD_STATEID. Chapter 8 would say that the rules for seqid on layouts
> > are slightly different from those for IO, opens, byte-range locks,
> > etc.
> > and have an xref to chapter 12 where this was discussed.
> >
> > Appropriate wordsmithing of the above to deal with the fact that
> > seqids
> > are subject to wraparound is left as an exercize for anybody who
> > doesn't
> > pass the buck quickly enough :-)
>
>
> Having reviewed all of the layout stateid material in the I-D I came
> to the conclusion that seqid checking/validation the server needed
> to do was limited to what is stated in:
> 12.5.5.2.1. Layout Recall and Return Sequencing
> that I sent out (mainly unchanged). Otherwise, the server just
> processes the requests as they arrive and ensures that the seqid
> is updated for each resposne. It is then the client's responsibility
> to sort out the responses to deal with overlapping layoutget/
> layoutreturn
> processing.
>
> My apologies for not making this clear.
>
> I would suggest this wording be added to the second paragraph of
> "12.5.3. Layout Stateid"
>
> Note the seqid processing for layout stateid differs from the stateid
> types. To support the parallelism desired, seqid processing rules
> are limited to those specified in the pNFS chapters (mainly this
> section and Section 12.5.5.2).
>
>
> Spencer
>
>
> > -----Original Message-----
> > From: William A. (Andy) Adamson [mailto:andros at citi.umich.edu]
> > Sent: Friday, March 21, 2008 10:05 AM
> > To: Spencer Shepler
> > Cc: NFSv4
> > Subject: Re: [nfsv4] layout stateid handling/processing updates
> >
> >
> > What about NFS4ERR_OLD_STATEID? If the client sends parallel
> > LAYOUTGET/RETURN operations, they could all have the same stateid. The
> > server will process the first one, and all remaining will have old
> > stateids. Should the server ignore this error for layout stateids?
> >
> > --->Andy
> >
> >
> > On Wed, Mar 19, 2008 at 3:06 PM, Spencer Shepler
> > <Spencer.Shepler at sun.com> wrote:
> >
> > Included is the updated/proposed text to clarify layout stateid
> > handling.
> > The diffs were a little difficult to read so I have included the
> > sections
> > that have been updated.
> >
> > Spencer
> >
> > ----
> >
> > 12.5.3. Layout Stateid
> >
> > As with all other stateids, the layout stateid consists of a
> > "seqid"
> > and "other" field. Once a layout stateid is changed, the "other"
> > field will stay constant unless the stateid is revoked, or the
> > client
> > returns all layouts on the file and the server disposes of the
> > stateid. The "seqid" field is initially set to one, and is never
> > zero on any NFSv4.1 operation that uses layout stateids, whether it
> > is fore channel or backchannel operation. After the layout stateid
> > is established, the server increments by one the value of the
> > "seqid"
> > in each subsequent LAYOUTGET and LAYOUTRETURN response, and in each
> > CB_LAYOUTRECALL request.
> >
> > Given the design goal of pNFS to provide parallelism, the layout
> > stateid differs from other stateid types in that the client is
> > expected to send LAYOUTGET and LAYTOUTRETURN operations in
> > parallel.
> > Since the server is incrementing the "seqid" value on each layout
> > operation, the client may determine the order of operation
> > processing
> > by inspecting the "seqid" value. In the case of overlapping ranges
> > in the client's requests, the ordering information will provide the
> > client the knowledge of which layout ranges are held. Additional
> > layout stateid sequencing requirements are provided in
> > Section 12.5.5.2.
> >
> > Once the client receives a layout stateid, it MUST use the correct
> > "seqid" for subsequent LAYOUTGET or LAYOUTRETURN operations. The
> > correct "seqid" is defined as the highest "seqid" value from fully
> > processed responses to LAYOUTGET or LAYOUTRETURN operations or
> > fully
> > processed arguments of a CB_LAYOUTRECALL operation.
> >
> > The client's receipt of a "seqid" is not sufficient for subsequent
> > use. The client must fully process the operations before the
> > "seqid"
> > can be used. For LAYOUTGET results, if the client is not using the
> > forgetful model (Section 12.5.5.1), it MUST first update its record
> > of what ranges of the file's layout it has before using the seqid.
> > For LAYOUTRETURN results, the client MUST delete the range from its
> > record of what ranges of the file's layout it had before using the
> > seqid. For CB_LAYOUTRECALL arguments, the client MUST send a
> > response to the recall before using the seqid. The fundemental
> > requirement in client processing is that the "seqid" is used to
> > provide the ordering of processing. LAYOUTGET results may be
> > processed in parallel. LAYOUTRETURN results may be processed in
> > parallel. LAYOUTGET and LAYOUTRETURN responses may be processed in
> > parallel as long as the ranges do not overlap. CB_LAYOUTRECALL
> > request processing MUST be processed in "seqid" order at all times.
> >
> > Once a client has no more layouts on a file, the layout stateid
> > is no
> > longer valid, and MUST NOT be used. Any attempt to use such a
> > layout
> > stateid will result in NFS4ERR_BAD_STATEID.
> >
> >
> > ------
> >
> > 12.5.5.2.1. Layout Recall and Return Sequencing
> >
> > One critical issue with regard to layout operations sequencing
> > concerns callbacks. The protocol must defend against races between
> > the reply to a LAYOUTGET or LAYOUTRETURN operation and a subsequent
> > CB_LAYOUTRECALL. A client MUST NOT process a CB_LAYOUTRECALL that
> > implies one or more outstanding LAYOUTGET or LAYOUTRETURN
> > operations
> > to which the client has not yet received a reply. The client
> > detects
> > such a CB_LAYOUTRECALL by examining the "seqid" field of the
> > recall's
> > layout stateid. If the "seqid" is not one higher than what the
> > client currently has recorded, and the client has at least one
> > LAYOUTGET and/or LAYOUTRETURN operation outstanding, the client
> > knows
> > the server sent the CB_LAYOUTRECALL after sending a response to an
> > outstanding LAYOUTGET or LAYOUTRETURN. The client MUST wait before
> > processing such a CB_LAYOUTRECALL until it processes all replies
> > for
> > outstanding LAYOUTGET and LAYOUTRETURN operations for the
> > corresponding file with seqid less than the seqid given by
> > CB_LAYOUTRECALL (lor_stateid, see Section 20.3.)
> >
> > In addition to the seqid-based mechanism, Section 2.10.5.3
> > describes
> > the sessions mechanism for allowing the client to detect callback
> > race conditions and delay processing such a CB_LAYOUTRECALL. The
> > server MAY reference conflicting operations in the CB_SEQUENCE that
> > precedes the CB_LAYOUTRECALL. Because the server has already sent
> > replies for these operations before issuing the callback, the
> > replies
> > may race with the CB_LAYOUTRECALL. The client MUST wait for all
> > the
> > referenced calls to complete and update its view of the layout
> > state
> > before processing the CB_LAYOUTRECALL.
> >
> > 12.5.5.2.1.1. Get/Return Sequencing
> >
> > The protocol allows the client to send concurrent LAYOUTGET and
> > LAYOUTRETURN operations to the server. The protocol does not
> > provide
> > any means for the server to process the requests in the same
> > order in
> > which they were created. However, through the use of the "seqid"
> > field in the layout stateid, the client can determine the order in
> > which parallel outstanding operations were processed by the server.
> > Thus, when a layout retrieved by an outstanding LAYOUTGET operation
> > intersects with a layout returned by an outstanding LAYOUTRETURN on
> > the same file, the order in which the two conflicting operations
> > are
> > processed determines the final state of the overlapping layout.
> > The
> > order is determined by the "seqid" returned in each operation: the
> > operation with the higher seqid was executed later.
> >
> > It is permissible for the client to send in parallel multiple
> > LAYOUTGET operations for the same file or multiple LAYOUTRETURN
> > operations for the same file, and a mix of both.
> >
> > It is permissilble for the client to use the current stateid (see
> > Section 16.2.3.1.2) for LAYOUTGET operations for example when
> > compounding LAYOUTGETs or compounding OPEN and LAYOUTGETs. It is
> > also permissible to use the current stateid when compounding
> > LAYOUTRETURNs.
> >
> > It is permissible for the client to current stateid when combining
> > LAYOUTRETURN and LAYOUTGET operations for the same file in the same
> > COMPOUND request since the server MUST process these in order.
> > However, if a client does send such COMPOUND requests, it MUST NOT
> > have more than one outstanding for the same file at the same time
> > and
> > MUST NOT have other LAYOUTGET or LAYOUTRETURN operations
> > outstanding
> > at the same time for that same file.
> >
> > ----
> >
> > 12.5.5.2.1.2. Client Considerations
> >
> > Consider a pNFS client that has sent a LAYOUTGET and before it
> > receives the reply to LAYOUTGET, it receives a CB_LAYOUTRECALL for
> > the same file with an overlapping range. There are two
> > possibilities, which the client can distinguish via the layout
> > stateid in the recall.
> >
> > 1. The server processed the LAYOUTGET before issuing the
> > recall, so
> > the LAYOUTGET must be waited for because it may be carrying
> > layout information that will need to be returned to deal with
> > the
> > CB_LAYOUTRECALL.
> >
> > 2. The server sent the callback before receiving the LAYOUTGET.
> > The
> > server will not respond to the LAYOUTGET until the
> > CB_LAYOUTRECALL is processed.
> >
> > If these possibilities cannot be distinguished, a deadlock could
> > result, as the client must wait for the LAYOUTGET response before
> > processing the recall in the first case, but that response will not
> > arrive until after the recall is processed in the second case.
> > Note
> > that in the first case, the "seqid" in the layout stateid of the
> > recall is two greater than what the client has recorded and in the
> > second case, the "seqid" is one greater than what the client has
> > recorded. This allows the client to disambiguate between the two
> > cases. The client thus knows precisely which possibility applies.
> >
> > In case 1 the client knows it needs to wait for the LAYOUTGET
> > response before processing the recall (or the client can return
> > NFS4ERR_DELAY).
> >
> > In case 2 the client will not wait for the LAYOUTGET response
> > before
> > processing the recall, because waiting would cause deadlock.
> > Therefore, the action at the client will only require waiting in
> > the
> > case that the client has not yet seen the server's earlier
> > responses
> > to the LAYOUTGET operation(s).
> >
> > The recall process can be considered completed when the final
> > LAYOUTRETURN operation for the recalled range is completed. The
> > LAYOUTRETURN uses the layout stateid (with seqid) specified in
> > CB_LAYOUTRECALL. If the client uses multiple LAYOUTRETURNs in
> > processing the recall, the first LAYOUTRETURN will use the layout
> > stateid as specified in CB_LAYOUTRECALL. Subsequent LAYOUTRETURNs
> > will use the highest seqid as is the usual case.
> >
> > _______________________________________________
> > nfsv4 mailing list
> > nfsv4 at ietf.org
> > https://www.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> nfsv4 at ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
> _______________________________________________
> nfsv4 mailing list
> nfsv4 at ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>
>
>
>
More information about the pNFS
mailing list