AIX 5L SysAdmin II: (Unit 04) – System Initialization – Part 2

Unit Objectives:
1. Identify the steps in system initialization from loading the boot image to boot completion.
2. Identify how devices are configured during the boot process
3. Analyze and solve boot problems

System Software Initializations – Overview:
Load Kernel and pass control
Create RAM file system – /, /etc, /dev, /mnt, /usr (Boot Logical Volume hd5)
Start init-process from BLV
– rc.boot1 – Configure base devices
– rc.boot2 – Activate rootvg
– rc.boot3 – Configure remaining devices
Start “real” init process (from rootvg) /etc/inittab

Failure LED (rootvg is not active)
Process 1 – init (F5) – (Boot Logical Volume – kernel, ODM, rc.boot, cmds)
restbase (Successful=510) – hd5 Restores the ODM
cfgmgr -f (Config_Rules phase=1 only, RAM file system ODM)
bootinfo -b (Sucessful =511)
Devices to activate rootvg are configured

If Errors occur at this point including c06 and 548 then
Boot from mksysb from tape into Maintenance Mode, Restore options
bosboot -ad /dev/hdisk0 (to restore the boot logical volume)
Then reboot

rc.boot2 (Part 1):
rc.boot2 (Success=551)

activate rootvg
ipl_varyon (varyon rootvg) (Success=517, Failure=552, 554 or 556)
If failure then Try bosboot -ad /dev/hdisk0 from Maintenance Mode and reboot

Check integrity of the filesystem and start mounting filesystems
fsck -f /dev/hd4 (-f fast check, Failure=555)
mount /dev/hd4 / (mount on the root in RAMFS) (Failure=557)
fsck -f /dev/hd2 (Failure=518)
mount /usr (contains commands in /usr)
fsck -f /dev/hd9var (Failure=518)
mount /var
copycore (if dump it is stored in /var/adm/ras/vmcore0)
umount /var
swapon /dev/hd6 (Activate paging space)

rc.boot2 (Part 2):
Copy RAM /dev files to disk: mergedev
Copy RAM ODM files to disk: cp /./etc/objrepos/Cu* /etc/objrepos
mount /var
Copy boot messages to alog (error messages)
mount /dev/hd4
Kernel removes RAMFS (RAM FileSystem)

rc.boot3 (Part 1):
Process 1 – init (Here we work with rootvg)
/sbin/rc.boot3 (Failure=553)

fsck -f /dev/hd3
mount /tmp

syncvg rootvg & (syncronize the rootvg if mirrored)

Configure Devices
Normal: cfgmgr -p2 – Config_Rules (phase=2 normal, 3=service) /etc/objrepos
Service: cfgmgr -p3

cfgcon – Start Console
rc.dt boot – Start CDE

savebase – Save the ODM settings in hd5 for next reboot

rc.boot 3 (Part 2):
syncd 60
errdeamon – Start recording errors

Turn off LEDs

rm /etc/nologin

chgstatus=3 CuDv (A device previously detected could not be found. Run “diag -a”)
System initialization completed
Execute next line in /etc/inittab

rc.boot Summary:
Where From Actions
rc.boot 1 – /dev/ram0 restbase, cfgmgr -f
rc.boot 2 – /dev/ram0 ipl_varyon rootvg, Merge /dev, Copy ODM
rc.boot 3 – rootvg cfgmgr -p2, cfgmgr -p3, savebase

Config_Rules Object Class:
phase seq bootmask rule
——— —– ————– ————————————-
cfgmgr -f (Configure base devices)
1 1 0 /etc/methods/defsys
1 2 0 /etc/lib/methods/deflvm

cfgmgr -p2 (Normal boot)
2 10 0 /etc/methods/defsys
2 10 0 /usr/lib/methods/deflvm
2 15 0 /etc/methods/ptynode
2 20 0 /etc/methods/startlft

cfgmgr -p3 (Service boot)
3 10 0 /etc/methods/defsys
3 10 0 /usr/lib/methods/deflvm
3 15 0 /etc/methods/ptynode
3 20 0 /etc/methods/startlft
3 25 0 /etc/methods/starttty

Note: The bootmask tells when to configure the devices depending upon the type of boot that is occuring.
0 – Always configure

Output of cfgmgr in the Boot Log Using alog:
# alog -t boot -o | pg
-t – type of log to display

/etc/inittab File:

runlevel values:
1=single user mode
2=normal mode
4-9=user defined run levels
::=run at all levels
0123456789:=run at all levels

action values:
off – don’t execute this command
wait – wait until the previous command has finished running
powerfail – run when a powerfail occurs
init – run only when rebooting the system

Note: Be careful when using an editor to change /etc/inittab.
Using mkitab, chitab, rmitab varifies the syntax.

System Hang Detection:
System hangs
– High priority process
– Other
What does shdaemon do?
– Monitors system’s ability to run processes
– Takes specified action if threshold is crossed

– Log Error in the Error Logging
– Display a warning message on the console
– Launch recovery login on a console
– Launch a command
– Automatically REBOOT system

Configuring shdaemon:
# shconf -E -l prio
ps -elf – display the priority levels

Resource Monitoring and Control (RMC):
Based on two concepts: conditions and responses
Associates predefined responses with predefined conditions for monitoring system resources.
– Example: Broadcast a message to the system administrator when the /tmp file system becomes 90% full.

Boot Problem Management:
1. LED 553
Check: File system full?
Access the rootvg.
Issue “df -k”
Check if /tmp, /usr or / are full.

2. LED 553
Check: /etc/inittab ?, /etc/environment ?
Access the rootvg. Check /etc/inittab (empty, missing or corrupt?)
Check /etc/environment

3. LED 551, 555, 557
Check: BLV corrupt?
Access the rootvg.
Re-create the BLV:
# bosboot -ad /dev/hdisk0

4. LED 551, 552, 554, 555, 556, 557
Check: JFS log corrupt?
Access rootvg before mounting the rootvg file systems.
Re-create the JFS log:
# logform /dev/hd8
Run fsck afterwards

5. LED 552, 554, 556
Check: Superblock corrupt?
Run fsck against all rootvg filesystems.
If fsck indicates errors (not an AIX file system), repair the superblock (notes)

6. LED 551
Check rootvg locked?
Access rootvg and unlock the rootvg:
# chvg -u rootvg

7. LED 523-534
Check: ODM files missing?
ODM files are missing or inaccessible.
Restore the missing files from a system backup.

8. LED 518
Check: Mount of /usr or /var failed?
Check /etc/filesystem.
Check network (remote mount), file systems (fsck) and hardware.

Q: In maintenance mode, when should one select the Access the volume group before mounting the file systems?
A: When corrective action such as manual execution of fsck is needed for a particular corrupted file system.? Also, when the JFS log is corrupted.? Finally, when there is a bad super block to correct in the file system.?
Q: In maintenance mode, when should one select Access this? volume group and? start a shell??
A: When attempting to change a forgotten root password.? Also, when there is a corrupted system file and when a file system that is out of space and will not allow the system to complete its boot process.?

Execute a script: /home/workshop/ex4pro1 – Causes some problems
The script powered down the server.
Reboot the server
Running the POST – Power On Self Test
Looking for a device to boot from hdisk0
Screen displays the firmware icons
LED showing code 557 – The machine will never boot
Lookup code: 557 – System is having problems mounting the root filesystem

Must use other bootable media such as a mksysb tape
Boot from mksysb
Press F5 – when the firmware icons appear to go into Maintenance Mode
Press 1, Enter – To use English during installation
Don’t select option #1 or #2
Select Option #3 – Start Mainenance Mode for System Recovery
Select Option #1 – Access root filesystem
Press 0, Enter – To Continue
Select the rootvg displayed on the screen
Select Option #2 – Access Volume Group and start a shell before mounting the file system

Received a prompt
# logform /dev/hd8 – Recreate the JFS log
logform: destroy /dev/hd8 (y)? y
fsck -y -V jfs /dev/hd1 (/home filesystem)
fsck -y -V jfs /dev/hd2 (/usr filesystem)
fsck -y -V jfs /dev/hd3 (/tmp filesystem)
fsck -y -V jfs /dev/hd4 (/ root filesystem)
fsck -y -V jfs /dev/hd9var (/var filesystem)
fsck -y -V jfs /dev/hd10opt (/opt filesystem)
shutdown -Fr
/usr/bin/ksh: shutdown: not found
Reboot the machine

Another problem:
LED = 553 – problems with the /etc/inittab file
Reboot the machine
Press F5 – Enter Maintenance Mode
Press F1 – To define the System Console
Select option#1 – Use English during install
Select option #3 – Maintanace Mode
Access a Root Volume Group
Select option #1 – Access this Volume Group and start a shell
# export TERM=lft
# df – Check the space of each file system check out ok
more /etc/inittab
Discovered that the /etc/inittab is corrupted
# sed ‘s/;/:/’ /etc/inittab > (change all ; to :, output to
# more (Verify changes)
# cp /etc/inittab (Copy changed file to inittab)
# more /etc/inittab (Verify copy)
# bootlist -m normal hdisk0 (Set bootlist to boot from hdisk0)
# shutdown -Fr

Leave a Reply

Your email address will not be published. Required fields are marked *