Checklist for Solving Head Scratching IT Problems [Favorites]

All IT folks have that problem/issue that you have been unsuccessfully staring at or thinking about for hours. This has happened to me with everything from simple workgroup networking to writing complex database reports. The worst part about such problems is that when you finally do have that “aha moment” you kick yourself for not seeing the resolution sooner. A list I often find myself referencing in such situations was written by Tom Limoncelli and can be seen on EverythingSysAdmin.

A List of Dumb Things to Check

Layer 0 – PEBKAC

Make sure CapsLock is off. (Same for ScrollLock and NumLock)

Type it again (without using cut-and-paste) and see if you get the same results. (good way to find a typo) (or a unicode “whitespace” char)

Use cut-and-paste to copy that variable name (or URL, commmand line, etc.) to see if it was entered correctly.

Are the binaries really the ones you think are running? (Did you install in single user mode when /opt wasn’t mounted? Can you check the md5 or sha1 checksum vs. a machine that is running properly?)

Check the file permissions.

Are you really on the host you think you are?

Are you doing the test from the right machine? Would the test be more effective from a different machine?

Does your test gear test what you think it tests? What happens when you run the exact same test on a known-good and a known-bad element? Do you get the results you expect both ways?

Is that a file, a directory, a hard link, a symbolic link, or a mountpoint?

Is the filename extention right? Should it be .php instead of .html?

Is the screen paused via Ctrl-S? (press CTRL-Q to find out)

You can get to the web site? Are you working in “off-line” mode?

Layer 1 – Physical

Are both ends of the (power/network/video/etc.) cable plugged in?

Is the cable plugged into the right jack? (Some jacks look the same: AUI and video, Some Sun’s have a ’stealth’ video jack that you aren’t supposed to use, etc.)

Did you unplug and re-plug in the cable to make sure its in right?

It’s too easy to answer “yes” when asked, “Is it plugged in.” It’s better to ask them to power it off, then power it back on OR ask them to “check both ends of the power cable” OR ask them if the power light is on, off, or blinking. (and if it’s blinking… RUN!)

Does plugging a lamp into that outlet make it light up? (If you use a radio, be careful of radios with battery backups)

Is the laptop running off battery? (therefore obscuring a power problem)

Is the device driver you’re trying to install for the device you actually have?

Layer 2 – Data Link

Is there a rogue DHCP server on the network messing with you?

Layer 3 – Network

Is the default route set? Is the netmask set correctly?

http://www.psc.edu/~mathis/MTU/index.html : Six classes of bugs limit network performance.

packet losses, corruption, congestion, bad hardware

IP Routing, long round trip times

Packet reordering

Inappropriate buffer space

Inappropriate packet sizes

Inefficient applications

Is someone else also at that IP address? (Unplug the network cable and ping the address)

(firewall ruleset issues) If you move the machine to another IP address does it still happen? If you move the machine to a different subnet does it still happen? If you put a different machine at that IP address does it still happen? If you boot the same machine on a different OS (like a CD-ROM based Linux or FreeBSD) does it still happen?

Does the same thing happen when you specify the IP address instead of the hostname? (Hint: the lmhost or /etc/hosts may be overriding)

Layer 4 – Transport

Traceroute from A to B. Traceroute B to A. Do they match up?

Layer 5 – Session

SSH, SCP, L2TP, PPTP

Layer 6 – Presentation

Is the program reading the last line of the file? Is it being processed right?

Is there a (invisible) CTRL-M at the end of each line in the text file?

Does the text file end with a newline?

Layer 7 – Application

Is DNS configured right? Misconfigured DNS masks other problems and appears as bizarre problems that will send you looking everywhere except /etc/resolv.conf

Check the environment variables (Use “strings” on the binary to find out what they really are supposed to be).

Are you running the same copy of the script that you are editing?

Is the software looking for the same copy of the configuration file that you are? (Does the new release look in /etc/example2/example.conf instead of the old /etc/example.conf location?)

Layer 8 – User/Political

Is the user pressing RETURN when you think they are? (Are they pressing it at all?)

Is the user typing a “/” or a “\”?

Does the user know which is the lessthan (<) and which is the greaterthan (>) symbol?

“Did you get permission to run crack against that password file?” … “Is it in writing?”

If All Else Fails

Did you remember to check this list?