Network Troubleshooting Tools

1. Network Troubleshooting Tools Kent Reuber ITS Networking Reuber@stanford.edu April 6, 2007

2. Outline • What problems do you need to solve? • Tool descriptions • Q&A time • Tool descriptions are in the “Software” section of the LNA Guide: http://lnaguide/software.html

3. What are the problems? • Are hosts online? (ping) • How do you get to hosts? (traceroute) • What are hosts running? (nmap) • Where/when have hosts been seen? (ipm) • “The network is slow” (Netspeed, iperf) • DHCP and DNS (SUNet reports) • Wireless problems (various) • Packet sniffing (wireshark), and batch NetDB changes (NetDB CLI)

4. Ping and traceroute

5. Ping: Are you there? • Ping sends ICMP echo requests to a host and asks for a reply. Reply time is also returned. • Some hosts may choose not to reply by security policy. It may not mean that they’re down. • Stanford de-prioritizes pings at some of our borders, so a long ping time or dropped pings does not indicate a poor connection. • Stanford maintains a special host: • “ping-me.stanford.edu” • Exempt from ping filter. • Have outside users ping “ping-me” if they claim that connections to Stanford are unavailable or slow.

6. Ping for Advanced Users • Can increase packet size to see duplex errors. (Unix: ping -s) • Default small (<60 byte) ping packets don’t generate enough traffic to show duplex problems. • Try using pings of 1000+ bytes. • Use nmap or similar utility for “ping sweeps” of entire networks: • “nmap -sP <network range>” (Ex. “nmap -sP 171.64.18.0/23”) • Nmap: http://insecure.org/

7. Traceroute: How do I get there? • How traceroute works: • Source sends a series of packets with increasing time-to-lives. (TTL is the allowed number of router hops.) Unix/Mac: UDP, Windows “tracert”: ICMP. • Routers will decrement TTL and respond with an ICMP “unreachable” message if TTL is 0. • Like ping, a timestamp is returned.

8. Traceroute notes • Routers need not reply to traceroutes. Lack of a reply does not mean that the router is down. • Return traffic doesn’t necessarily use the same path. • This can cause problems with firewalls and packet shapers that assume they see the whole conversation. • When troubleshooting connection problems, you may want to have the destination send traceroutes to you as well.

9. nmap

10. Nmap: Scanning nets • In addition to ping scans, you can scan for open ports on hosts. • This can be useful for seeing who is running a service (intentionally or otherwise!) • My recipe for scanning for open TCP ports: • ”nmap -P0 -sT net -p ports -oG - | grep open”

11. Getting nmap • Download from http://insecure.org • Unix and MacOS X usually require compiling from source. • Windows binary available.

12. ipm

13. IPM: IP <-> MAC addresses • Stanford-specific utility • How it works: • Devices broadcast ARP packets when they need to communicate locally. • Routers see these ARP and cache it. • Information is periodically harvested and kept in a database. • Using IPM, you can track when an IP/MAC was first and last seen and where.

14. IPM: What’s it good for? • You can find MAC addresses which aren’t in Netdb. • Find out where a particular device has been seen. • See if multiple devices are using a single IP address.

15. More on IPM • Where is it: • AFS: /usr/pubsw/sbin/ipm • Note: this directory is not in your default PATH. • Using IPM: • Wildcards: “_” (single character), “%” (multiple characters) • Run “ipm -h” to see list of options.

16. MAC vendor codes • MAC addresses are 48-bit (6 bytes) xx:xx:xx:xx:xx:xx, where each “x” is a hexadecimal number 0-9,a-f. • First 3 bytes are the Organizationally Unique Identifier (OUI), which tell you who made the network card. • Can look this up. My favorite site: http://www.coffer.com/mac_find/ • Can tell you when NetDB records are outdated. For example, a NetDB record for a Macintosh with MAC address 00:0b:db (Dell) is clearly wrong.

17. Netspeed and Iperf

18. Netspeed & Iperf: Speed testing • Often hear “the network is slow”. • Is it the client, the network or a server? • Where’s the bottleneck? • Useful tools: • Netspeed (Web based speed to campus backbone). • Iperf (command line tool for point-to-point).

19. Netspeed • Web based speed testing to Stanford backbone: http://netspeed.stanford.edu/ or http://iperf.stanford.edu/ • Useful for finding duplex errors (misconfigured hubs or switches) in the path.

20. Iperf • Command line testing tool. • Can also run speed tests against netspeed.stanford.edu and iperf.stanford.edu • Can be run in server mode for testing speed between arbitrary points (e.g., within your network) http://dast.nlanr.net/Projects/Iperf/

21. How fast can you go? • DSL: 1 Mbps (asymmetric) • 802.11b wireless: 1-5 Mbps • 802.11g wireless: 1-12 Mbps • Fast Ethernet: 80+ Mbps • Gigabit: ?? Note: consider these tests as upper bounds. For gigabit especially, you may not be able to transfer real data this fast.

22. DHCP

23. Troubleshooting DHCP • Many things can go wrong. Problems are rarely caused by DHCP server unavailability. • Things to check: • What IP is the host getting? • Netdb record for the host. • DHCP server logs, roaming pool utilization reports.

24. Understanding DHCP • Stanford has two DHCP servers: dusk and dawn. • Info from Netdb is uploaded approximately every 15 minutes. Give Netdb the time to upload data. • At Stanford, MAC address information is required for successful DHCP. • Initial DHCP is a four step process using broadcasts; renews are different.

25. Leases • DHCP addresses are valid for a limited period (wired and wireless). • Normal DHCP: 2 days • Roaming DHCP: 42 minutes • Hosts will re-confirm their leases halfway through the lease period. • Clients use unicast directly to the DHCP server (clients have an address and they know who their server is). • Renew message type is used.

26. DHCP roaming • If the Netdb record has a “home” IP address appropriate for the network where the device is located, DHCP servers will send it. • Can have “home” IP addresses and still be able to roam to other networks. • Can have multiple “home” addresses bound to each MAC address. • If no appropriate address is entered, DHCP will look for available roaming addresses on the local network. • Number of roaming address is specified by the LNA. Defined in the Netdb network record. • Usually there are only a handful of roaming addresses. Can easily run out of them.

27. What address did you get? • The address received may tell you what the problem is. • Self assigned (169.254.*.*): • NetDB record not set up properly. • No roaming address available. • Routing or DHCP server problem (less likely). • 10.x.x.x: • Used by Network self-registration system. (SNSR) • Could also be used by a rogue. • 192.168.*.*: • Probably a rogue DHCP server.

28. Finding rogues • Try pinging the gateway that’s being distributed. • Use “arp” command to get the MAC address of the gateway. Or use a sniffer if you have one. • Look at switch MAC tables and find the offending hosts. Shut off the port or go have a “chat”. • New Net-to-Switch configs block rogue DHCP servers!

29. Available DHCP reports • DCHP logs for a given host. • Type in MAC address and see the conversation. • Takes practice to read. • Roaming address utilization • How many roaming addresses were used in a day. • DHCP reports from dusk and dawn • Hourly logs show number of DHCP messages for hosts. • “No free leases” may indicate that you’re out of roaming addresses. • All reports are linked from LNA Guide software section: http://lnaguide/software.html

30. DNS

31. DNS at Stanford • Host information is entered in NetDB • Uploads to DHCP servers about every 15 minutes. • Uploads to DNS servers about every hour. • Starts at 5 minutes after the hour. • Takes about 20 minutes. Should be done by 30 minutes past the hour. • Specific info on timing is kept in the NetDB help files.

32. DNS inspection tools • Standard: “host”, “nslookup”, “dig”. • Stanford whois can show you most NetDB information: • “whois -h whois.stanford.edu <query>” • Use “%” and “_” as wildcards as per ipm. • Great for people who need “read-only” access, since you don’t need a NetDB account. • For host names, you need to end query in a “.” or specify “.stanford.edu” so that whois knows you want information on a host.

33. Wireless

34. Wireless problems • Wireless is slow or unavailable. • Reports can be vague. “Wireless is slow on the 2nd floor.” • Isolating the problem can speed resolution. • Exactly where is the problem occurring? • What access point is the user connecting to? • Do others have problem in the area?

35. Wireless tools • Access point association: • Mac: Internet Connect utility • PC: ?? • Access point discovery for seeing available AP’s and channels: NetStumbler, iStumbler • Iperf and Netspeed are useful for checking speed problems. • Often, a AP reboot will solve the problem. • AP jack (tso) information is in Netdb. • Can unplug and replug if necessary.

36. Packet sniffer

37. EtherPeek and Wireshark • Stanford has site license for Etherpeek, but it’s still expensive. • Wireshark (formerly Ethereal) is free. (Motto: “Sniff free or die!”) • X windows application for Unix/Mac. • Binary for Windows. • http://www.wireshark.org/ • Some books are available!

38. Advice on Sniffing • Need for a sniffer is rare, but invaluable when you need it. • Learn to use it before you need it! • You will need to set up special “span” ports on your switches to see all traffic. • No need if you’re interested in broadcasts and multicasts. • Most useful for seeing traffic entering and leaving your net.

39. NetDB Command Line

40. NetDB CLI overview • Designed for power users. • Provides a subset of NetDB functionality (mostly nodes) for batch changes. New features are periodically added. • Use with caution. Try one or two hosts before doing big batches.

41. How to run NetDB CLI • Located in AFS space: • /usr/pubsw/sbin/netdb (note: this directory is probably not in your PATH) • Use -h option to get command syntax • Stuff you can do (to a single machine or list of machines): • Change administrators, locations. • Change IP addresses. • Delete nodes.

42. Q&A??