[pnfs] linux-pnfs-2.6-latest file layout data server session management

[William A. (Andy) Adamson]

I'm porting the pNFS code forward from 2.6.18.3 to linux-pnfs-2.6-latest.
Here is an overview of what I am planning.
Please comment...

-->Andy

Data Server Session Management

Current 2.6.18.3
-------------------------

Per superblock list of deviceID's
       - each deviceID is a stripe which stores it's own list of data server
         structs (nfs4_pnfs_ds) - not shared between stripes... (ugly
draft-13 bakeathon hack)

struct nfs4_client contains exchange_id session bookeeping.
struct nfs4_session is in struct nfs4_client.

1) device_create: use MDS struct nfs_server
        - contains the ds_wsize, ds_rsize etc
        - pnfs_mountid, pnfs_curr_ld pointers.
        - pass MDS struct nfs_server to device_create with decoded device
info
        from GETDEVICELIST/INFO.

2) device_create:
        - create own nfs4_client struct which has exchange id session (one
          slot) bookkeeping and it's own session struct for server
connection.
        - nfs4_client passed to nfs4_proc_exchange_id

struct nfs4_client contains
        - data server exchange id bookeeping and client id info
        - data server struct nfs4_session.
        - data server struct rpc_clnt.
this works for data server.

Problem: struct nfs4_client is wrong place to put non-data server session
because:
        - multiple exports to same server (e.g. more than one
          struct nfs_server) should not share the same session.

Current linux-pnfs-2.6-latest
-----------------------------------------
Will port the draft-13 data server bookeeping changes from 2.6.18.3 with
proposed changes below.

struct nfs_client contains exchange_id session bookeeping and clientid info.
struct nfs4_session moved to struct nfs_server

1) device_create: use MDS nfs_server struct
        - contains the ds_wsize, ds_rsize etc
        - pnfs_mountid, pnfs_curr_ld pointers.
        - pass MDS nfs_server struct to device_create with decoded device
info
        from GETDEVICELIST/INFO.

2) device_create:
        - create own nfs_client struct which has exchange id session
          bookeeping (one slot)
        - no struct nfs_server for data servers

struct nfs_client contains
        - data server exchange id bookeeping and client id info
        - data server struct rpc_clnt.
        - struct nfs_client used by nfs4_proc_exchange_id,
nfs4_proc_destroy_session


linux-pnfs-2.6.latest Proposed Changes
-----------------------------------------------------------

1) add a list of data servers to nfs4_pnfs_dev_hlist, hashed by ip_addr.
        - data servers will be used by multiple stripes(device IDs) per FSID
(e.g. per MDS struct nfs_server)
        - each data server will have a session used for every stripe (device
ID) the data
        server is a member of.

add dev_dslock and dev_dslist to struct nfs4_pnfs_dev_hlist which hangs off
MDS struct nfs_server

struct nfs4_pnfs_dev_hlist {
        rwlock_t                dev_lock;
        struct hlist_head    dev_list[NFS4_PNFS_DEV_HASH];
        rwlock_t                dev_dslock;
        struct hlist_head     dev_dslist[NFS4_PNFS_DEV_HASH];
}

2) add a struct nfs4_sessions *cl_ds_session; to nfs_client to hold data
   server session.
        - there will always be only one nfs_client per data server.
        - (re)create data server session => will need nfs_client.


3) add a count field and list_head field for nfs4_pnfs_ds
        - need to know when last stripe has de-referenced nfs4_pnfs_ds

struct nfs4_pnfs_ds {
        struct list_head        ds_hnode; /* for ds_hlist */
        u32                     ds_ip_addr;
        u32                     ds_port;
        struct nfs4_client      *ds_clp;
        atomic_t                ds_count; /* for destroy session */
};

4) change the array of struct nfs4_pnfs_ds (the data server struct)
   to an array of pointers to struct nfs4_pnfs_ds

struct nfs4_pnfs_dev {
        u32 stripe_index;        i
        int num_ds;
        struct nfs4_pnfs_ds *ds_list[];
};

5) will not create data server session on GETDEVICELIST. will create data
server
session on first I/0.



Questions
---------

1) should we plan for multiple sessions to a data server?

2) we could create a new struct to hold only exchange id and client id
bookeeping.
this would mean changing the interface to nfs4_proc_exchange_id, and to
nfs4_state_recovery_ops. not worth it?

3) do we need a separate rwlock_t for the dev_dslist, or can dev_lock
protect both lists?

4) is there a better way to have a data server session created before I/O
and not have any data server
sessions created but not used? small file performance issue?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://linux-nfs.org/pipermail/pnfs/attachments/20071016/5d0b0cf1/attachment.htm