GFS2 Setup Notes - cluster3, 2.6.27 kernel

From Linux NFS

You do not have permission to edit this page, for the following reason:

You can view and copy the source of this page:

With the release of Fedora 10 in early October 2008 ('''Update:''' since delayed until December), Red Hat's newest version of their cluster suite ("'''cluster3'''") will go prime-time. In the last couple months, one of cluster3's main dependencies split into two parts, '''corosync''' and '''openAIS''', and building has been problematic at times. These are my notes from my latest GFS2 setup, and the first time I'm moving everything to 2.6.27. ==The parts:== As of this moment, cluster3's dependencies aren't yet packaged as RPMs, so building from source is a must. The particular revisions are changing almost every day, so you'll have to consult the [http://sources.redhat.com/cluster/wiki/ cluster project wiki] for current versions. Even overnight, this just changed for me. Yikes! I'm now basing off of cluster3 '''cluster-2.99.10''', '''corosync svn r1667''', and '''openAIS svn r1651'''. * I used yum to get: '''libvolume_id-devel''', '''libxml2''', '''libxml2-devel''', '''openldap''', '''openldap-devel''', '''readline''' (likely installed already), and '''readline-devel'''. ** (Note: I only needed the readline stuff when I went back to <tt>LVM2.2.02.39</tt>, as ...40 was broken.) * get [ftp://sources.redhat.com/pub/dm/ the latest device-mapper] * use svn to clone the corosync repository ** <tt>$ svn checkout --revision 1667 http://svn.fedorahosted.org/svn/corosync</tt> *** or <tt> svn update --revision...</tt> ** <tt>$ cd corosync && svn export . ../corosync-checkout/</tt> ** the build stuff is in the "trunk" directory. * use svn to clone the openAIS repository ** <tt>$ svn checkout --revision 1651 http://svn.fedorahosted.org/svn/openais</tt> ** <tt>$ cd openais && svn export . ../openais-checkout/</tt> ** the build stuff is in the "trunk" directory. * use git to clone my cluster3 repository ** <tt>$ git-clone git://git.linux-nfs.org/projects/richterd/cluster.git</tt> ** my "pnfs-gfs2-dev" branch is where current development goes. * get [ftp://sources.redhat.com/pub/lvm2 the latest LVM2] ** '''note''': <tt>LVM2.2.02.40</tt> is broken (?!?) -- a minor build issue, but I can't believe they released something that has an undefined symbol. ==The build:== * build/install the device-mapper * build/install corosync (shouldn't even need to configure it; this is nice and easy now!) * build/install openAIS (shouldn't even need to configure it) * before doing the cluster3 stuff, you'll need to be running a 2.6.27-based kernel and have its sources available * build/install the cluster3 stuff ** I had to point it at openAIS, and I generally disable the <tt>rgmanager</tt> stuff. Also now disabling the perl/python bindings, and I build the kernel module separately anyway. ** <tt>$ ./configure --openaislibdir=/usr/lib/openais --openaisincdir=/usr/include --without_rgmanager --without_bindings --without_kernel_modules</tt> * build/install LVM2, making sure to specify the <tt>clvmd</tt> type ** <tt>$ ./configure --with-lvm1=none --with-clvmd=cman --prefix=/usr</tt> =====The rest of the cluster:===== So far, I'd built all of this on a single cluster node. I set that node up as an NFS server and exported my top-level build directory. Then, on each cluster node, I mounted the export and just did the <tt>make install</tt> steps. ==Shared storage:== For testing, I use ATA over ethernet and have had fairly good results with it. * yum-installed AoE initiator (client) '''<tt>aoetools-23-1</tt>''' across the cluster * downloaded AoE target (server) [http://internap.dl.sourceforge.net/sourceforge/aoetools/vblade-15.tgz vblade-15.tgz] and installed it on a separate host (<tt>rhclhead</tt>). ** with <tt>dd</tt>, I created a 1GB empty file, <tt>AOE_SHARED_STORAGE</tt> ** .. and exported it: <tt>$ sudo vbladed 2 3 eth0 AOE_SHARED_STORAGE (major dev num 2, minor 3)</tt> ==Creating the filesystem:== * prep the volume with LVM2 metadata: <tt>$ sudo pvcreate -M 2 /dev/etherd/e2.3</tt> * create the volume group '''DMRVolGroup''': <tt>$ sudo vgcreate -M 2 -s 1m -c y DMRVolGroup /dev/etherd/e2.3</tt> * edit <tt>/etc/lvm/lvm.conf</tt> across the cluster and set the locking type to <tt>DLM</tt>. * make sure you have a properly configured <tt>/etc/cluster/cluster.conf</tt> also set up across the cluster ('''DMRCluster''', in my case). * now, bring up the cluster: <tt>pdsh -w guest[1-3] sudo service cman start && pdsh -w guest[1-3] sudo service clvmd start</tt> * create the logical volume '''DMRVolume''': <tt>$ sudo lvcreate -n DMRVolume -l 100%VG DMRVolGroup</tt> * create the GFS2 filesystem '''DMRFS''': <tt> $ sudo gfs2_mkfs -j 4 -p lock_dlm -t DMRCluster:DMRFS /dev/DMRVolGroup/DMRVolume</tt> ** note the <tt>-j</tt> ("number of journals") argument needs to be appropriate for your cluster size.

Return to GFS2 Setup Notes - cluster3, 2.6.27 kernel.