Setup up mirroring for datavgs and rootvg
Use split mirror copies to reduce downtime for backup
Avoid unwanted deactivation of your VGs by the quorum mechanism
Understand LVM-related entries in the error log
Replace a failed disk in rootvg and data VGs
Volume Group with Logical Partions pointing to Physical Partitions
Stale Partitions (1 of 2)
The main purpose of mirroring is to withstand disk failures
Some disk failures are temporary (for example, power loss).
Data on the failed disk becomes outdated (stale)
After repair, the copies must be resynchronized
Copying the entire disk would cause much I/O load
How does LVM handle this?
State information is kept per physical partition
varyonvg VGname calls syncvg -v VGname:
– Only partitions marked stale are copied
– syncvg runs as a background process
VGSA – Volume Group Status Area (127 bytes * 8 bits = 1016 on each disk)
Note: This is one bit for each partition (indicates if the partition is stale)
Run varyonvg when you have stale partitions even if the volume group is already variedon. It will only update stale partitions.
Stale Partitions (2 of 2)
syncvg works only if the PV is still the same.
The resychronization does not start by itself.
Using varyonvg may be better than invoking syncvg directly.
Consumes a significant part of I/O resources.
Information is kept in the Volume Group Status Area (VGSA):
– Unlike the VGDA, the VGSA is a per-disk structure.
– The VGSA cannot track more than 1016 PPs per PV (incl. AIX 4.2.1)
Newly created mirrors are in the stale state initially.
(Unless sychronization was explicitly requested)
Note: Running syncvg with update all partitions. Running varyonvg will only update the stale partitions (takes less time).
Scheduling Policy (Sequential vs Parallel)
Situation 1: Total Power Failure
If the disks were working on the same LP, all copies can be corrupted.
Remedy: UPS or Sequential Mirroring Policy
Second physical write operation to disk2 is not started unless disk1 has completed successfully. However, you will always have at least one good copy of the data.
Sequential always goes back to the primary disk to read data.
Notes on Parallel Mirroring (Default Setting):
May increase overall performance because it begins the write to each disk in the mirror at the same time. However, may not have a good copy of the data.
How much I/O is effectively done in parallel depends on the hardware and software configuration.
Parallel always goes back to the fasted accessible disk to read data.
Mirror Write Consistency (On by default)
Total power failure (especially with sequential mirroring) or
All disks are intact, but the host itself fails during a write
System must distinguish between the old and new copy.
Solution: Mirror Write Consistency Cache (MWC)
Allows identifying the correct PP after reboot.
Not needed for paging spaces.
Separate area of each PV (located at low phys. sector #s):
– Low PSNs mean “outer edge” in LVM terms.
– Place LVs with mirror write consistency on the outer edge.
Creating Mirrored LVs
smit – Add a Logical Volume
Adding Mirror Copies to Existing LVs
smit – Add Copies to a Logical Volume
Extending a File System in a Mirrored LV
chfs invokes extendlv with default arguments.
Strictness forces new partitions to separate PVs only.
PVs chosen may still reside on the same bus, loop and so forth
Step 1: Use extendlv to enlarge the LV only:
– Calculate the additional space needed in terms of LPs.
– Specify PVs (or use a map file) as required.
Step 2: Use chfs to increase the size of the fiel system:
– Specify exactly the same size (but in 512 byte blocks) as before.
– chfs will detect the unused space and not invoke extendlv again.
Officially supported in AIX 4.1.4 or later
Is best done using mirrorvg -m rootvg hdisk1
Steps necessary after mirrorvg:
– bosboot -a -d /dev/hdisk0 (rebuilds the boot logical volume)
– bootlist -m normal hdisk0 hdisk1 (tell the system which disks to boot from)
bosboot will note that the boot LV hd5 is mirrored and
– Takes care of all necessary processing for the second disk.
– Does not need to be invoked twice.
Prior to AIX 4.3.3 dump devices could not be mirrored:
– Created to unmirrored dump LVs (hd7, hd71 instead).
– sysdumpdev -e displays an estimated dump size.
Mirroring an Entire Volume Group
smit – Mirror a Volume Group
Turn off Quorum Checking when mirroring a volume group so that it doesn’t get varied off.
Online Mirror Backup
– Data needs to be in a consistent state for backup.
– Backups may take a long time.
– In many cases the available downtime is too short for a backup.
– Use a mirrored LV
– Stop the application ( and umount the FS) for a few seconds only.
– Split the mirror copies: One for backup and one for the restarted application.
History of applied techniques:
– V3.x.y: use the same alloc. map file for the mirror and the unmirrored LV
– V4.1.4: splitlvcopy (automates splitting and subsequent LV creation)
– V4.3.1: chlvcopy (no permanent split, but partitions marked stale)
– V4.3.3: chfs -a splitcopy= /splitfs (based on chlvcopy)
Note: Having three mirrored LVs you can temporarily split one off to backup the file system and still have the data being mirrored on the remaining two LVs. Then you can resync the third one after the backup.
VGs containing one or two disks: (two VGDAs on one disk, one on the second disk)
Problem: asymmetrical failure of PV1:
Only 1/3 of the VGDAs available after failure of PV2:
still 2/3 of the VGDAs available
VGs containing three or more disks:
even distribution after failure of a PV:
still 2/3 (or more with bigger VGs) of all VGDAs available
One of the most-discussed features of AIX
Enforces that more than 50% of the VGDAs are available
If this is not ture, the VG
– Is either not varied on at all or
– Is closed during operation:
System may show strange behavior
LVM_SA_QUORCLOSE logged via error reporter.
Pros and Cons of Quorum
Enforces the use of additional hardware: More disks mean more security
With only 1 valid VGDA left, bad sectors in this area would be disastrous.
Quorum busters are expensive and difficult to implement in HACMP environments
Introduces unnecessary problems for some diaster recovery scenarios.
Turning off Quorum Checking
Command: chvg -Qn VGname
lsvg VGname shows 1 in the QUORUM field
– Seems to be in effect immediately
– Really needs a varyoffvg /varyonvg for data VGs
– rootvg: Requires bosboot -a -d /dev/hdiskX and a reboot
– rootvg: If a regular varyon is not possible, a forced varyon is used.
Number of VGDAs necessary to keep/bring a VG online:
VG Status Quorum Enabled Quorum Disabled
————— ————————- ————————-
Running > 50% VGDAs 1Disk
varyonvg > 50% VGDAs 100% VGDAs or if environment variable MISSINGPV_VARYONVG=TRUE > 50% VGDAs
Command: varyonvg -f VGname
A summary like the following is displayed:
PV Status hdisk2 … PVREMOVED
hdisk3 … PVREMOVED
hdisk4 … PVACTIVE
hdisk5 … PVACTIVE
varyonvg: Volme group VGname is varied on.
Defective PVs are “locked out” for now and later varyons:
– Don’t count during future quorum checks
– lspv hdiskX shows zero VGDAs.
For 2-disk VGs, the active disk becomes the 2-VGDA-disk.
This does not change even if the failed disk can be repaired.
Physical Volume States
varyonvg VGname –> active
varyonvg -f VGname (varyon forced)
repair the disk
chpv -v a hdiskX
Quorum Buster Disks
Definition: the 2n+1)th disk in a volume group
Can be an old disk not suitable for production use.
Should have independent power supply and adapter.
Often ideal solution for standalone servers.
Quorum and HACMP
In some cases it is necessary to disable quorum
Turning off quorum checking may cause takeover to fail:
– HACMP doesn’t use varyonvg -f
– Undetected disk failures violate the “100%-at-varyon” rule
– Remedy: Use MISSINGPV_VARYON=TRUE with nonquorum VGs
Problems with quorum busters:
– Buster disk must also be connected to every host accessing the VG
– Often no further slot available for an additional adapter.
– Use error notification to be aware of disk failures as soon as possible
– Exploit the redundant power supplies of 7133 SSA and 7135 RAIDiant
– For SSA: Find a third location for a disk tower and add it to the loop.
– Consider using HAGeo for true disaster recovery.
Geographic Mirroring Device (GMD)
Application – GeoMirror – LVM – LV (Local GMD copy)
Geographic networks – GeoMessage
Application – GeoMirror – LVM – LV (Remote GMD copy)
Subsystem which adds the geographic remote mirroring functional.k, onto a file system or logical volume