Colleague System Maintenance

1. Colleague System Maintenance File System Administration & Maintenance – Part Two Cinda Goff Chuck Hauser 2005-10-30

2. Presentation Conventions • Names (files, users, daemons) are usually in bold:/etc/syslog.conf • System dependent or variable items are usually in italics: /var/sadm/patch/patchnumber/log • File entries and output are in mono-spaced type:> root 8036 c Tue Apr 26 23:59:00 2005 < root 8036 c Tue Apr 26 23:59:59 2005 • Ämarks a line wrapped to fit on the slide:mv Solaris_8_Recommended_log ÄSolaris_8_Recommended_log.yyyymmdd • ð marks a horizontal tab (09 hex) • Reference OE is Solaris 8 • Reference UniData is 6.0

3. Unix File System Maintenance • Log File Maintenance • Cleaning the File System (files that do not need to be retained) • File System Maintenance (fsck & UFS Logging)

4. Solaris 8 (and Earlier) Automatically Maintained Logs

5. Solaris 9 System Log Rotation Starting with Solaris 9, logadm is used to handle log rotation for: • /var/adm/messages • /var/cron/log • /var/log/syslog • /var/lp/logs/lpsched • /var/lp/logs/requests

6. Some Logs That Need Manual Maintenance These files will grow forever: • /var/adm/authlog • /var/adm/loginlog • /var/adm/sulog • /var/adm/wtmpx • /var/sadm/install_data/Solaris_8_Recommended_log • var/sadm/patch/patchnumber/log • /var/spool/lps/adm/activity_log (EasySpooler)

7. Trimming Logs • Most logs can be cleared by copying /dev/null or standard out to the log:cp /dev/null >logfil (or simply: >logfile) • Or save the file, then clear:mv logfile logfile.save.nameÄ && >logfile • Or save the latest entries in the log with the tail command:tail -50 logfile >logfile.tmp Ä && mv logfile.tmp logfile • To stop EasySpooler logging, remove the log file:rm /var/spool/lps/adm/activity_log

8. Patch Cluster Logs Recommendation If a Solaris_8_Recommended_log file already exists, the next cluster installation will append new entries at the end of the existing log. May want to save each log individually:mv Solaris_8_Recommended_log ÄSolaris_8_Recommended_log.yyyymmdd

9. Maintaining wtmpx Can be maintained by hand or script:cp /var/adm/wtmpx /var/adm/cis.wtmpx.2004cp /dev/null >/var/adm/wtmpx (Or simply >/var/adm/wtmpx) Note: /var/adm/utmpx is information about who is currently logged into the system. Used by who, whodo, w, users, and finger commands. Don’t mess with this file.

10. Files That May Need Manual Removal • core files • preserve files • backing-store snapshot files

11. Core Files • Core files are images dumped to a disk of a user process terminated by certain signals • coreadm command will configure or show settings (but won’t prevent core dumps) • To prevent core dumps, use Bourne Shell ulimit command with –c set to zero:ulimit –c 0

12. Removing Core Files As root, find and remove all core files:find / -name core –type f –exec rm –f {} \;(Note: Be sure to use the ‘–type f’ option; patches for third-party apps such as Apache sometimes have a directory named core.)

13. Crash Dump Files • A crash dump file is a disk copy of physical memory of the computer at the time of a fatal system error. • Usually not a problem – unless the machine has crashed. • The dump is usually saved in a swap file partition, then on reboot the savecore command copies the dump to the savecore directory.

14. Removing Crash Dump Files • dumpadm will show crash dump settings, including the savecore directory:# dumpadm Dump content: kernel pages Dump device: /dev/dsk/c2t0d0s3 (swap) Savecore directory: /var/crash/cis Savecore enabled: yes • Check the savecore directory for a crash dump file (vmcore.n), or use a find command with: –name “vmcore.*”.

15. Preserve Files • Should the vi editor (or the system) crash, a copy of the user’s file is stored in /var/preserve/username. • vi –r filename allows recovery of workfile. • Unlikely there are any files there, but may want to check and remove:find /var/preserve –exec rm –f {} \;

16. Snapshot File Cleanup • If the fssnap command is used without the unlink option, the backing-store file will still exist on the system and need to be manually removed after issuing the fssnap –d filesystem command. • If there are no active snapshots (check with fssnap –i), either: rm /backup-store-path/snapshot# Or find /backing-store-path –name “snapshot*” Ä–exec rm {} \;

17. Unix Disk-Based File Systems • Disk drives consists of slices (partitioned using the format command). • Each slice is either raw or contains a file system (constructed by newfs, front-end to mkfs). • Disk file systems consists of so many blocks • The default Solaris disk file system is UFS (Unix File System)

18. VTOC (Volume Table of Contents) • The first cylinder of the disk contains a VTOC describing the disk’s slices – slice number, tag, starting sector, size, & last sector. • Use the prtvtoc command to save the VTOC of a disk in case the entire disk fails: prtvtoc /dev/rdsk/c#t#d0s2 Ä>c#t#d0s2.vtoc • Use this file with the fmthard command to recreate the VTOC on a replacement disk: fmthard -s c#t#d0s2.vtoc Ä/dev/rdsk/c#t#d0s2

19. UFS File System Structure • Formatting a UFS file system divides the disk slice into cylinder groups. • Cylinder groups have four basic types blocks: bootblock, superblock, inode, and storage (or data block). • For further info, see man page fs_ufs (4) • Structure is documented in /usr/include/sys/fs/ufs_fs.h

20. Cylinder Group Layout Graphic from Sun Microsystems, Inc.

21. UFS Areas and Block Types

22. Boot Block • Used for booting system, holds bootstrap programs • Only appears in first cylinder group (blocks 0 -15) • Left blank if file system isn’t used for booting • Firmware boot program loads and executes bootblk, which then loads and executes /ufsboot. • Installed by installboot command.

23. Superblock • Critical data about the file system – a copy is replicated before each cylinder group • Sync command forces all file systems’ superblocks to be written to disk; • The shutdown command calls sync. • Structure is documented in /usr/include/sys/fs/ufs_fs.h • Superblock contains flags about file system state, including fs_clean.

24. Superblock fs_clean Flags

25. Some Inode (or I-node) Fields • File type (regular, directory, link, etc.) • File mode (read/write/execute permissions) • Hard link count • UID of owner, GID of group • Size (number of bytes) • Dates & times: created, last accessed, last modified • Array of 15 disk-block addresses

26. File Addressing Graphic from Sun Microsystems, Inc.

27. Boot Block Problems • If ‘The file just loaded does not appear to be executable’ message appears when booting, then the hard disk boot block is corrupted. • Boot Solaris Software 1 of 2 CD and install a new boot block on the boot disk: ok> boot cdrom –s // #installboot /usr/platform/`uname Ä- i`/lib/fs/ufs/bootblk /dev/rdsk/c#t#d0s0

28. Causes of File System Inconsistencies • Unclean shutdowns • Stop+A executed • System turned off without proper shutdown • System unplugged or power failure • Disk with mounted file systems removed • Software error in kernel • Hardware • Failing disk or controller • Other major component fails

29. Check & Repair File Systems: fsck • fsck is only a file-system checker; does not handle data-integrity by checking contents of regular file data blocks. • Runs automatically at boot, can be run manually • Never use fsck on a mounted file system! (Unless you want to cause a panic and test your crash dump settings …)

30. fsck Superblock Checks • File system size versus number of inodes and number of blocks used by superblock • Checks free blocks; blocks marked as free should not be claimed by any files • Count of summary block free inodes is compared to actually count of free inodes

31. fsck Inode Checks • Inodes are checked sequentially starting at first true inode (2). • Inodes are checked for: • Format and type • Link count (directories compared to inode) • Duplicate block • Bad block numbers (blocknumber >= first block & <= last block number) • Inode size (actual number of blocks compared to inode size field)

32. fsck Directory Block Checks • Inode number in directory points to unallocated inode • Inode number in directory entry points beyond inode list • First entry in directory list must be ‘.’ entry referencing itself • Second entry must be ‘..’ and equal to inode of parent directory • Directory must be linked to somewhere in the file system.

33. fsck and fs_clean • The fsck command uses the fs_clean flag to determine whether a file system needs checking. • To view a file system’s fs_clean flag:#fstyp –v /dev/rdsk/c#t#d#s#Ä| grep fscleanfilesystem state is valid, fsclean is 2 (be patient: fstyp –v returns a lot of information)

34. fsck at Boot Time • At boot time the /sbin/rcS script checks /, /usr and /var file systems (if /usr & /var are separate systems). • When the system boots, fsck runs in ‘preen’ mode: inconsistences consist with an unorderly shutdown are repaired, file systems are checked sequentially using the fsck pass field in /etc/vfstab. • If the system cannot be repaired interactively, /sbin/rcS will print a either a warning or fatal message that states ‘Run fsck manually’. • After non-fatal errors are repaired, the system will continue booting. After fatal errors, the system will reboot.

35. / (Root)and fsck • If the root file system isn’t mounted FSCLEAN or FSTABLE, then at boot time / is mount read-only and flagged FSBAD. • To fix, need to boot from alternate device such as CD, then use fsck on CD to repair. • If /usr is hosed, will usually also need to boot from CD.

36. UFS Logging • UFS logging first appeared in Solaris 7 • Previously (2.6 and earlier) file system logging required Solaris DiskSuite and a separate logging partition. • UFS logging writes all metadata changes first to the logging space, then actual data blocks are written. Metadata is the directory and Inode information • In other words, details of all changes to the file system are recorded in a log before the changes are actually written to the disk.

37. How to Log File Systems • To log a file system, either mount the system with the logging option:mount –o logging myfilesystemor change the option field in /etc/vfstab to logging: #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options ... /dev/md/dsk/d4 /dev/md/rdsk/d4 /vol1 ufs 1 yes logging • Using mount without arguments will include logging status:/vol1 on /dev/md/dsk/d4 Äread/write/setuid/intr/largefiles/logging/onerror Ä=panic/dev=1540004 on Sat Oct 29 14:14:21 2005

38. UFS Logging Space • The logging information is stored in the file system’s free blocks. • Log size is 1MB per 1GB of file system space, up to 64MB.

39. UFS Logging Advantages • Reduces risk of file system inconsistencies – if data blocks aren’t written, metadata changes are rolled back. • Increases boot speed – fsck doesn’t waste time on system where the fs_clean flag is FSLOG. • Most file system operations are significantly faster

40. Solaris Disk Suite (up to OE 8) andSolarisVolume Manager (OE 9 and later) • If using Solaris Disk Suite (SDS) or Solaris Volume Manager (SVM), periodically use the metastat command to check status of metadevices and hot spare pool. • Automate checking by using shell and awk scripts run by cron:# Check status of disk metadevices0 4,16 * * * /opt/local/sbin/dsmon.sh

41. dsmon.sh Shell Script #!/bin/sh # @(#)dsmon.sh 1.1 @(#) # Uses an awk script (dsmon.awk) to parse output of DiskSuite # metastat command in order to report errors. DSMON_SCRIPT=/opt/local/sbin/dsmon.awk DSMON_OUT="/tmp/dsmon.$$.out" DSMON_STATUS=0 # RECEPIENTS – sendmail alias of who to notify RECEPIENTS=sysadmin.list trap "rm -f $DSMON_OUT; exit 1" 1 2 3 15 if metastat | awk -f $DSMON_SCRIPT >$DSMON_OUT then : else DSMON_STATUS=$? mailx -s "Metastat Disk Error Report" sysadmin.list <$DSMON_OUT fi rm -f $DSMON_OUT exit $DSMON_STATUS

42. dsmon.awk Awk Script # DiskSuite Status Monitor # @(#)dsmon.awk 1.1 @(#) BEGIN { STATUS=0 } /State: / { if ($2 != "Okay" ) { if (prev ~ /^d/) print prev, $0 STATUS=9 } } { prev = $0} END { exit STATUS }