AIX 5L Configuring TCP/IP (Unit 09) – Troubleshooting

Unit Objectives
Describe debugging alternatives
Document network and system configuration
Use tools to diagnose TCP/IP problems

Debugging Networks
Work from a picture (diagram the network)
Use a methodology (method to track down the problem)
What are your assumptions? (Be careful)

1. Scope of Impact – Who is having problems?
2. How far can I get? – Ping local interface the outward, traceroute command
3. Has anything changed?
4. Look for patterns, consistent or not

TCP/IP Data Flow (Layers of the Architecture)
Send(Down) Receive(Up)
———————- ———————-
Application Write Buffer Read Buffer
TCP Socket Send Socket Receive
IP IP Input Queue
Network Interface Transmit Queue Receive Queue
Hardware Transport Medium

Note: Checklist of things to look at.

Overview of Troubleshooting Commands
Document System – lscfg, lsdev, lsattr, snap
Document Network – lsattr, lsdev, snap (tcp/ip – ifconfig, no)
Problem Determination – iptrace, errpt (tcp/ip – ping, ifconfig, arp, netstat)
Tuning – chdev (tcp/ip – no, ifconfig)

lscfg and lsdev – gives you the devices
lsattr – gives you the attributes
snap – collectin of configuration files

Note: When things are running smoothly, document the system capturing information in a booklet for future reference when things are not going right.

List System Configuration Information
lscfg – list of devices on the platform, location codes, description
lscfg -l tok0 -v (-v means vital product data)

List Device Information
# lsdev -C -c if (-C = Predefined Database)
# lsdev -C -c adapter

List Device Attributes
# lsattr -E -H -l en0 (-E effective attributes, -H field headers, -l interface or object
# chdev -l en0 mtu -1500 (change the above settings)
lsattr -R -H -l en0 -a mtu (view acceptable values)
Note: Can use Smit to change these values too.

Network Interface Parameters Display
# ifconfig tr0
# ifconfig en0

Note: Check the kernel values, may be different than ODM at boot up.

Display Network Options
# no -a (list all options)
# no -o thewall=value (set the value of a single attributes)
# no -o the wall | grep dgd

snap Command
Gathers system configuration information
Compresses information into pax file
Requires root authority to use
# snap [] [] [] [] [] [] [] [] [] []
-t = tcp information
-n = nfs information

Test Network Connectivity
# ping -c 5 sys1 ( -c 5 – count of 5)
ping -f (flood echo requests)
ping -s (increase packet sizes)

Check Address Resolution
# arp -a (view the arp cache to find duplicate IP address on the local network)
Every 20 minutes will flush the cache and rebroad cast.
# arp -n (just ip addresses)

Isolating Name Resolution Problems with DNS
Should /etc/resolv.conf exist? No –> Does it? Yes –> Remove it
Does it? No –> Create it
Does it contain DNS pointer? No –> Check for valid nameserver addresses, update file
Is entry valid? Yes –> Is named running? No –> Start it, Check database entries

Display Network Interface Information (Kernel not ODM)
# netstat -in (list of interfaces in pairs of lines)

Display Routing Table
# netstat -rn

Monitor State of an Interface
# netstat -I tr1 2 (continous monitoring, ever 2 seconds)
Note: Shows traffic for tr1 on left compared to all adapters combined on the right.

Display Device Driver Statistics
# netstat -v (list of all adpaters)
# netstat -v tok1 (specific adapter)
# netstat -v tr0 (can give the interface tr0 instead of the adapter tok1)

Display Network Memory Usage
# netstat -m (memory)

Display Dropped Packets
# netstat -D (Device driver packets dropped)

Trace Packets with the iptrace Daemon
Records internet packets received from configured interfaces
Can be started as a command or under control of SRC
Can be stopped either with the kill or stopsrc command
Binary output sent to logfile specified at startup
ipreport command used to format the trace file for viewing

Recommended Book – TCP/IP Illustrated – by Stevens

iptrace Examples
# iptrace -b -d sys4 -s sys3 /tmp/telnet.trace
# startsrc -s iptrace -a “-i en0 /home/team01/iptrace.log” (-a arguments passed to iptrace)
# iptrace -e /tmp/iptrace.log

-b (bidirectional trace)
-d (destination)
-e (trace all traffic not just that for the adapter)
-s (source)

View a Trace Report
# ipreport -n -s /tmp/telnet.trace > /tmp/report1
# pg /tmp/report1

-n (packet numbering)
-s (prefix protocols)

Sample Problem Scenario
See example

Unit Summary
When debugging a network it is always a good idea to work from a picture of the network
A good methodology to use for problem determination is to work with the layering model of TCP/IP from the bottom up
Commands that can be used to ducument the system and netowrk include lsdev, lscfg, lsattr, ifconfig, ping -R, no, snap
Commands that are useful to diagnose TCP/IP problems are: ping, netstat, iptrace

Exercise 9 – Troubleshooting
Look at devices
lsdev -Cc adapter (-C customized database, -c class of adapter)

Look at attributes for a specific device
lsattr -E -l ent0 (effective attributes from customized database, logical device name of ent0)

Look at interfaces
lscfg -v -l ent0 (-v vital product data, logical device name)
lsdev -Cc if (-C customized database, c -if class of interface)

Look at detailed information in ODM database
lsattr -E -l en0

ifconfig en0

Note: ifconfig gives detailed information from the kernel which may have been changed since the ODM database was read at last boot time.

netstat -in (same information)

Look at network options
no -a | more (list all network options)
no -o extendednetstats (0=off, 1=on – display setting for memory utilization)
no -o extendednetstats=1 (0=off, 1=on – assign setting for memory utilization)

snap command (IBM support center may request)
snap -t (Checking space requirement for tcpip imformation………)
Gathering tcpip system information……..
Creates some directories /tmp/ibmsupt/tcpip…

cd /tmp/ibmsupt/tcpip
ls (shows a collection of configuration files and scripts)

netstat -m | more (mbufs – control block manages memory utilization)
Note: Will see additional information when the extendednetstats=1 is turned on.

netstat -v | more (detail statistics of the adapters)

netstat -D | more (device drivers information)

Monitor system
Create some activity using a script
netstat -I en0 2 (monitor en0 every 2 seconds, performance analysis)

Note: See the exercise

Performance Testing using ping
Note: See the exercise

running iptrace
arp -a | more (list the arp cache)
arp -d sys3 (delete sys3 or ipaddress from the arp cache)

startsrc -s iptrace -a “-b sys3 -d -s sys2 /tmp/trace1” (-b bidirectional tracing)
lssrc -s iptrace (check to see if still active)
ping -c3 sys3 (generate some activity)
stopsrc -s iptrace
ipreport -n -s /tmp/trace1 > /tmp/report1
view /tmp/report1

iptrace -a -b -s sys2 -d sys3 -P tcp -p 23 /tmp/trace2 (protocol, port 23)
telnet sys3 (generate some activity)
login: root
ls /usr/bin (just list some files)
ps -ef | grep iptrace
kill 21466 (defaults to signal 15)
ipreport -n -s /tmp/trace2 > /tmp/report2
view /tmp/report2
/login (search for login)

Note: This can be used to view the passwords during login. Anyone that is listening and view the packets can detect the passwords.

Leave a Reply

Your email address will not be published. Required fields are marked *