Configuring DRBD with OCFS2 on Suse Linux 11.4 - Dual Primary Mode

Home | Profile 

Configuring DRBD with OCFS2 on Suse Linux 11.4  - Dual Primary Mode

I started with the single primary mode configuration, as I explained in my previous article. This way, I get the DRBD configuration right and tested out before moving to oracle cluster configuration (OCFS2 + DRBD). First thing is to get OCFS2 and O2CB installed, so go to software.opensuse.org and search for ocfs2.  You will need to install the ocfs2* and *o2cb* rpm packages. 

 
-- Download and Install the Software

silicon:/home/raj/downloads/ocfs2 # wget http://software.opensuse.org/search/download?base=openSUSE%3A11.4&file=openSUSE%3A%2F11.4%2Fstandard%2Fx86_64%2Focfs2console-1.6.3-5.3.x86_64.rpm

silicon:/home/raj/downloads/ocfs2 # wget http://software.opensuse.org/search/download?base=openSUSE%3A11.4&file=openSUSE%3A%2F11.4%2Fstandard%2Fx86_64%2Focfs2-tools-o2cb-1.6.3-5.3.x86_64.rpm

silicon:/home/raj/downloads/ocfs2 # wget http://software.opensuse.org/search/download?base=openSUSE%3A11.4&file=openSUSE%3A%2F11.4%2Fstandard%2Fx86_64%2Focfs2-tools-1.6.3-5.3.x86_64.rpm

silicon:/home/raj/downloads/ocfs2 # zypper in ocfs2console-1.6.3-5.3.x86_64.rpm

silicon:/home/raj/downloads/ocfs2 # zypper in ocfs2-tools-o2cb-1.6.3-5.3.x86_64.rpm

--Create a directory called ocfs2 under /etc/

silicon:/home/raj/downloads/ocfs2 # mkdir -p /etc/ocfs2/

--Edit cluster.conf file with below entries. You can choose any randpom ports that are not in use. Refering back to my previous article (DRBD - Single Primary Mode) , silicon (192.168.5.100) is my primary host and graphics (192.168.5.101) is my secondary one.

silicon:/etc/ocfs2 # cat cluster.conf

node:

    ip_port = 7777

    ip_address = 192.168.5.100

    number = 0

    name = silicon

    cluster = ocfs2

node:

    ip_port = 7777

    ip_address = 192.168.5.101

    number = 1

    name = graphics

    cluster = ocfs2

cluster:

    node_count = 2

    name = ocfs2

   

-- Configure the o2cb service.

Remember to give the cluster name as "ocfs2" on the prompt, or the name you have given in the cluster.conf file.

silicon:/etc/drbd.d # /etc/init.d/o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot.  The current values will be shown in brackets ('[]').  Hitting

<ENTER> without typing an answer will keep that current value.  Ctrl-C

will abort.

Load O2CB driver on boot (y/n) [y]:

Cluster stack backing O2CB [o2cb]:

Cluster to start on boot (Enter "none" to clear) []: ocfs2

Specify heartbeat dead threshold (>=7) [31]:

Specify network idle timeout in ms (>=5000) [30000]:

Specify network keepalive delay in ms (>=1000) [2000]:

Specify network reconnect delay in ms (>=2000) [2000]:

Writing O2CB configuration: OK

Setting cluster stack "o2cb": OK

Starting O2CB cluster ocfs2: OK

-- Add below to resource.res

…

resource data01 {

  on silicon {

    device    /dev/drbd01;

    disk      /dev/vgPQ1/lvPQ1_dfs06;

    address   192.168.5.100:7790;

    meta-disk internal;

  }

  on graphics {

    device    /dev/drbd01;

    disk      /dev/vgPQ1/lvPQ1_dfs06;

    address   192.168.5.101:7790;

    meta-disk internal;

  }

}

…

-- scp the cluster.conf and resource.res to the 2nd node - graphics.

silicon:/etc/ocfs2 # scp cluster.conf graphics:/etc/ocfs2/

cluster.conf                                                                     100%  269     0.3KB/s   00:00

silicon:/etc/drbd.d # scp resource.res graphics:/etc/drbd.d/

resource.res                                                                     100%  795     0.8KB/s   00:00

-- Create a LVM Partition. You can use the device partitions instead of LVM's.

silicon:/etc/drbd.d # lvcreate -L 10G -n lvPQ1_dfs06 vgPQ1

  Logical volume "lvPQ1_dfs06" created

-- Create metadata, sync and attach.
 

silicon:/etc/drbd.d # drbdadm create-md data01

Writing meta data...

initializing activity log

NOT initialized bitmap

New drbd meta data block successfully created.

Success

-- Check the status.
Note: "data00" is what I configured for single-primary-mode. "data01" is the one we currently should pay attention to - the dual-primay-mode.

silicon:/etc/drbd.d # drbd-overview.pl

  0:data00  Connected Primary/Secondary UpToDate/UpToDate C r-----

  1:data01

silicon:/etc/drbd.d # drbdadm syncer data01

silicon:/etc/drbd.d # drbdadm attach data01

-- Check the status after connect

silicon:/etc/drbd.d # drbd-overview.pl

  0:data00  Connected  Primary/Secondary UpToDate/UpToDate     C      r-----

  1:data01  StandAlone Secondary/Unknown Inconsistent/DUnknown r----s

-- Repeat same steps on the 2nd node - graphics.

graphics:/etc/drbd.d # drbdadm create-md data01

Writing meta data...

initializing activity log

NOT initialized bitmap

New drbd meta data block successfully created.

Success

graphics:/etc/drbd.d # drbdadm syncer data01

graphics:/etc/drbd.d # drbdadm attach data01

graphics:/etc/drbd.d # drbd-overview.pl

  0:data00  Connected  Secondary/Primary UpToDate/UpToDate     C      r-----

  1:data01  StandAlone Secondary/Unknown Inconsistent/DUnknown r----s

-- Connect the device on Primary - silicon. 

silicon:/etc/drbd.d # drbdadm connect data01

Monitor the progress. Pay attention to the changes that occurs on the "cs" (Connection Status), "ds" (Disk Status), "ns" Netowrk Status.

silicon:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:76 nr:0 dw:116132 dr:2189 al:36 bm:15 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404

-- Connect the device on Secondary - graphics.

graphics:/etc/drbd.d # drbdadm connect data01

graphics:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

    ns:0 nr:76 dw:116304 dr:1485 al:4 bm:7 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404

-- Run the below on one of the nodes.

silicon:/etc/drbd.d # drbdadm -- --overwrite-data-of-peer primary data01

silicon:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:76 nr:0 dw:116132 dr:2189 al:36 bm:15 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----

    ns:3072 nr:0 dw:0 dr:3744 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10482332

        [>....................] sync'ed:  0.1% (10236/10236)M

        finish: 1:49:11 speed: 1,536 (1,536) K/sec

graphics:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

    ns:0 nr:76 dw:116304 dr:1485 al:4 bm:7 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----

    ns:0 nr:261760 dw:261760 dr:0 al:0 bm:15 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10223644

        [>....................] sync'ed:  2.5% (9984/10236)M

        finish: 2:06:27 speed: 1,324 (1,280) want: 250 K/sec

You can see the progress of the sync by running cat /proc/drbd. Once the sync is complete, you can see the change of status in "ds" for both primary and secondary nodes. It will show up as "UpToDate/UpToDate".

graphics:/etc/drbd.d #  cat /proc/drbd                                                                 Thu Jun 30 07:16:01 2011

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:76 nr:0 dw:116132 dr:2189 al:36 bm:15 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:10618904 nr:0 dw:134792 dr:10486946 al:120 bm:640 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

-- Once the sync is done, create the file system on the drbd device. /dev/drbd01
   

silicon:/etc/drbd.d # mkfs -t ocfs2 -N 2 -L ocfs2_drbd01 /dev/drbd01

mkfs.ocfs2 1.6.3

Cluster stack: classic o2cb

Label: ocfs2_drbd01

Features: sparse backup-super unwritten inline-data strict-journal-super xattr

Block size: 4096 (12 bits)

Cluster size: 4096 (12 bits)

Volume size: 10737053696 (2621351 clusters) (2621351 blocks)

Cluster groups: 82 (tail covers 8615 clusters, rest cover 32256 clusters)

Extent allocator size: 8388608 (2 groups)

Journal size: 67108864

Node slots: 2

Creating bitmaps: done

Initializing superblock: done

Writing system files: done

Writing superblock: done

Writing backup superblock: 2 block(s)

Formatting Journals: done

Growing extent allocator: done

Formatting slot map: done

Formatting quota files: done

Writing lost+found: done

mkfs.ocfs2 successful

-- Add the "startup" and "net" portions to the resource file on both the nodes (silicon & graphics).

startup {
    become-primary-on both;
 }
net {
    allow-two-primaries;
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
  }

resource data01 {

  on silicon {

    device    /dev/drbd01;

    disk      /dev/vgPQ1/lvPQ1_dfs06;

    address   192.168.5.100:7790;

    meta-disk internal;

  }

  on graphics {

    device    /dev/drbd01;

    disk      /dev/vgPQ1/lvPQ1_dfs06;

    address   192.168.5.101:7790;

    meta-disk internal;

  }

}

-- Bounce nodes on both primary and secondary. 
  

silicon:/etc/drbd.d # drbdadm down data01

silicon:/etc/drbd.d # drbdadm up data01

silicon:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C r-----

    ns:0 nr:2832 dw:2832 dr:0 al:0 bm:17 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:872

        [===================>] sync'ed:100.0% (872/3704)K

        finish: 0:00:00 speed: 1,416 (1,416) want: 250 K/sec

-- Make the first node primary.

silicon:/etc/drbd.d # drbdadm primary data01

       

silicon:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----

    ns:0 nr:3704 dw:3704 dr:672 al:0 bm:81 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0  

-- Bounce the secondary node.
   

graphics:/etc/drbd.d # drbdadm down data01

graphics:/etc/drbd.d # drbdadm up data01

graphics:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----

    ns:3704 nr:0 dw:0 dr:3704 al:0 bm:81 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

-- Convert the 2nd node to primary.

graphics:/etc/drbd.d # drbdadm primary data01

graphics:/etc/drbd.d # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----

    ns:3704 nr:0 dw:0 dr:4376 al:0 bm:81 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

-- Mount the device to a directory on both nodes.

silicon:/etc/drbd.d # mount /dev/drbd01 /u06 -t ocfs2

graphics:/etc/drbd.d # mount /dev/drbd01 /u06 -t ocfs2

-- Touch a test file, you can see the file appearing on the secondary node. You can run a "tail -f afile" command on one of the nodes on a different ssh/telnet session and redirect some data to this file on the other node where in you can monitor a live transfer of the data.

silicon:/u06 # touch afile

graphics:/u06 # ls -ltr

total 388

drwxr-xr-x 2 root root   3896 Jun 30 22:32 lost+found

-rw-r--r-- 1 root root 394065 Jun 30  2011 afile

Run the tail -f afile first and do the redirect on the other node. As this is dual primary mode you can swap these commands between both nodes.

graphics:/u06 # tail -f afile

silicon:/u06 # cat /var/log/messages > afile

If you get the configuration right, you will be able to see output right away as and when you redirect the output to this file on the silicon.

Below are some additional commands with which you can admin the o2cb service.

• "/etc/init.d/o2cb offline" - Takes the clusters offline.
• "/etc/init.d/o2cb online"  - Brings the clusters online.
• "/etc/init.d/o2cb load"    - Loads all OCFS2 modules.
• "/etc/init.d/o2cb unload"  - Unloads all OCFS2 modules.

References:

OCFS2 Documentation - http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.2/ocfs2_users_guide.pdf

Here's one of the issues I ran into - a likely split-brain scenario where I couldn't get both nodes to talk. I ran drbdadm up data01 and here's the output I got.

silicon:~ # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/Outdated C r----s
    ns:0 nr:0 dw:0 dr:1680 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4194140

Same on the secondary node too..

graphics:~ # cat /proc/drbd

version: 8.3.9 (api:88/proto:86-95)

srcversion: A67EB2D25C5AFBFF3D8B788

 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/Outdated   r----s

    ns:0 nr:0 dw:0 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4194140

I haven't dig deeper into how to resolve this issue, but here's how I got both back. Ran the disconnect on the primary (one of the nodes) and discarded the data, which put the connection status (cs) to "WFConnection"


silicon:~ # drbdadm secondary data01

silicon:~ # drbdadm disconnect data01
silicon:~ # drbdadm -- --discard-my-data connect data01
silicon:~ # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/Outdated C r----s
    ns:0 nr:0 dw:0 dr:1680 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4194140
silicon:~ # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/Outdated C r----s
    ns:0 nr:0 dw:0 dr:1680 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4194140

Once done, I ran the connect on the 2nd node, i.e secondary.



graphics:~ # drbdadm connect data01
graphics:~ # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:2176 nr:0 dw:0 dr:3856 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4191964
        [>....................] sync'ed:  0.1% (4191964/4194140)K
        finish: 0:29:06 speed: 2,176 (2,176) K/sec



silicon:~ # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:8320 dw:8320 dr:1680 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4185820
        [>....................] sync'ed:  0.3% (4185820/4194140)K
        finish: 0:49:49 speed: 1,384 (1,384) want: 250 K/sec

silicon:~ # drbdadm primary data01
silicon:~ # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:4194140 dw:4194140 dr:2352 al:0 bm:256 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

silicon:/u06 # service o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Not active

silicon:/u06 #  mounted.ocfs2 -f

Device                FS     Nodes

/dev/mapper/vgPQ1-lvPQ1_dfs06  ocfs2  Not mounted

/dev/drbd1            ocfs2  Not mounted

silicon:/u06 # mount /dev/drbd01 /u06 -t ocfs2

silicon:/u06 # mounted.ocfs2 -f

Device                FS     Nodes

/dev/mapper/vgPQ1-lvPQ1_dfs06  ocfs2  silicon

/dev/drbd1            ocfs2  silicon

graphics:/u06 # cat /proc/drbd
version: 8.3.9 (api:88/proto:86-95)
srcversion: A67EB2D25C5AFBFF3D8B788
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:4194140 nr:0 dw:0 dr:4195852 al:0 bm:256 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

graphics:/u06 # mounted.ocfs2 -f

Device                FS     Nodes

/dev/mapper/vgPQ1-lvPQ1_dfs06  ocfs2  Not mounted

/dev/drbd1            ocfs2  Not mounted

graphics:/u06 # mount /dev/drbd01 /u06 -t ocfs2

graphics:/u06 # mounted.ocfs2 -f

Device                FS     Nodes

/dev/mapper/vgPQ1-lvPQ1_dfs06  ocfs2  silicon, graphics

/dev/drbd1            ocfs2  silicon, graphics

silicon:/u06 # service o2cb status

Driver for "configfs": Loaded

Filesystem "configfs": Mounted

Stack glue driver: Loaded

Stack plugin "o2cb": Loaded

Driver for "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster ocfs2: Online

Heartbeat dead threshold = 31

  Network idle timeout: 30000

  Network keepalive delay: 2000

  Network reconnect delay: 2000

Checking O2CB heartbeat: Active

graphics:/u06 # service o2cb status

Driver for "configfs": Loaded

Filesystem "configfs": Mounted

Stack glue driver: Loaded

Stack plugin "o2cb": Loaded

Driver for "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster ocfs2: Online

Heartbeat dead threshold = 31

  Network idle timeout: 30000

  Network keepalive delay: 2000

  Network reconnect delay: 2000

Checking O2CB heartbeat: Active

graphics:/u06 # touch file_created_at_graphics

graphics:/u06 # ls -ltr

total 0

drwxr-xr-x 2 root   root     3896 Jun 30 22:32 lost+found

drwxr-xr-x 6 oracle oinstall 3896 Jul 17 18:07 oracle

-rw-r--r-- 1 root   root        0 Aug 21 23:07 file_created_at_graphics

silicon:/u06 # ls -ltr

total 0

drwxr-xr-x 2 root   root 3896 Jun 30 22:32 lost+found

drwxr-xr-x 6 oracle dba  3896 Jul 17 18:07 oracle

-rw-r--r-- 1 root   root    0 Aug 21 23:07 file_created_at_graphics

Hope this helps.

Raj...