Symmetrix Connectivity Verification Level 2

1. Symmetrix Connectivity Verification Level 2

2. Course Objectives Upon completion of this course, you should be able to: Understand and troubleshoot Symmetrix connectivity issues (Point of Failure) Components of SYR Connectivity SymmConnect enablement Service Request (SR) Handling for no connect home (Incorrect/Improved) Understand the ConnectEMC GUI (Graphical User Interface) Understanding connectivity methods ( i.e. EMCRemote, Modem, SymmIP, ESRS, ConnectEMC, Firewall issues) End to End connect home triage What to look for using SYR for connectivity connect home information Detailed review of connectivity logs showing failures and possible resolutions Process to performing a test call home Process for Test Call Home verification Connectivity Transitions Acronyms

3. Lesson 1: Components of SYR Connectivity and SR handling Upon completion of this lesson, you should be able to: Understand Components of SYR Connectivity Understand the enablement of SymmConnect Understand the difference between incorrect and correct SR handling

4. Components of SYR Connectivity Components: SYR database contains: Current data on all EMC's hardware and maintenance records All connect home and SR related information Includes Customer Service Integration (CSI) database records on customer contacts, connection records, maintenance issues, etc. Lifeline One-way weekly inbound (from SYR to the Symmetrix) connectivity check for Symmetrix. Also includes collection of configuration files. These files are used for business analytics. SymmConnect SymmConnect is a one-way outbound (from the Symmetrix to EMC) connectivity check for Symmetrix. SymmConnect provides the best proactive and preemptive support in the industry

5. SymmConnect Set-up Enablement of SymmConnect: Default interval for Test Call is set to 7 days SymmConnect executable packages are available for each Enginuity family After a successful execution of the SymmConnect executable the system will automatically attempt to connect home Systems with Enginuity 5874 starting with 5874.248.194 and above, and 5875, only have the option of 1 or 7 days for Test Call interval Automatic enablement of SymmConnect on Enginuity 5875 has been simplified; any successful connect home can enable SymmConnect (now available in 5874.259.201 as well) The two SymmConnect files that need to connect home so that the SymmConnect process is enabled (pre 5875 and pre-5874.259.201) Schedule.ini or Schedinfo.xml (dependent on Enginuity family)

6. Incorrect SR Handling for no connect home (AAA1.58) During previous investigations, certain resolution summaries were used to close a AAA1.58 error code Service Request (SR) that did not resolve the connect home issue. This ultimately could cause and has caused Data Unavailable/Data Loss (DU/DL) events to occur. The following two slides list examples of real world investigation techniques that were used with descriptions of why they are not the proper resolution methods. None of the resolutions methods on the following two slides should be used for closing a AAA1.58 error code SR.

7. Incorrect SR Handling for no connect home (AAA1.58) Dialed into the system ok This does not contribute to the resolution because being able to connect in has nothing to do with connect home failures All checks are good (health check, environmental check, etc) These checks will not assist in determining the connect home issue as they do not check anything to do with connectivity Test call was successful The test call needs to be verified in SYR to be considered successful. The test call is not successful if it simply leaves the service processor successfully Dispatched to unknown issues/reasons Even if the failure cannot be determined, by not listing the actions taken and the steps to be done by the field will result in duplicate work and delay in resolution

8. Incorrect SR Handling for no connect home (AAA1.58 cont.) Closed with no reason/explanation All connect home issues require a resolution Closed to unrelated SR This can cause the connect home issue to not get resolved. If an SR is open or has been dispatched for something else or another PSE or field personnel are dialed into the system at the time, make sure either they check/correct the connect home issue or wait until they are done so you can investigate the connect home issue Closed to unresolved AAA1.58 SRs If another AAA1.58 SR is created, do not close to previous SR as issue is not be resolved. Further investigation is required Rebooting blindly This does not correct the connect home issue and will also reset services that could be used to determine cause of failure

9. Improved SR Escalation Process flow (AAA1.58 cont.) Simply stating unable to determine the problem, leaving the SR blank or vague and dispatching the SR to the field causes confusion and delays in getting the resolution to the connectivity issue. The history should be checked for any previous AAA1.58 errors. If there is a history of AAA1.58 SRs then verify the correct actions were taken. Provide a detailed description of the steps taken to troubleshoot and possibly correct the connect home issue. Provide a detailed description of the steps the field should take to correct the connect home issue. The next two slides will provide examples of what type of information should be documented in the SR.

10. Improved SR Escalation Process flow (AAA1.58 cont.) Below are examples of text that should be provided in the SR describing what steps have been taken to investigate the connect home issue. Do not simply dispatch the SR without placing details of what investigation has already been done. This information will provide a starting point for the field if further investigation is required. (actual text taken from real SRs) Tried with the desk phone found that the modem is ringing and not getting picked up Tried with the local modem and getting connection attempt failed Tried all the IP islands with RCC and no go Tried with the HOP IP Island 7 no go Verified the connect home interval is the same in SYR and Site Information Verified within ConnectEMC that all connection methods are configured correctly

11. Improved SR Escalation Process flow (AAA1.58 cont.) Detailed list of recommended actions to provide to the CE (actual text taken from real SRs) Page CE to verify if the modem and SP need to be rebooted (Important: A reboot should only be done if the field verifies the SP is unresponsive, such as blue screen, hung or dead). If the field can login to the SP but a remote connection cannot be established, the following steps need to be performed: Access the “Task Manager” Under the “Application” Tab, list any applications with the status of "Not Responding” Under the “Process” Tab, filter by “CPU’ and list the Top two applications using large amounts of CPU (not including System Idle Process) Under the “Process” Tab, filter by “Memory Usage” and list any Image Names with the memory usage over 200,000k(200MB) After recording the above information, contact a Connectivity SME in the PSE lab for assistance at the following escalation e-mail: SYMMSPAppsConnectivityEscalations@emc.com. Please provide as much details in the e-mail as possible.

12. Improved SR Escalation Process flow (AAA1.58 cont.) Have CE check that the Connection records are correct in IBG (Install Base) Have CE generate a test call home from the box Verify in SYR the test call was successful and resets next expected call home Refer to Primus emc114175 for more information If unsuccessful, this needs to be escalated to a Connectivity SME in the PSE lab to determine the root cause. Contact the SME at the following escalation e-mail: SYMMSPAppsConnectivityEscalations@emc.com. Provide as much details in the e-mail as possible. DO NOT close the SR unless a test call home is completed successfully

13. Lesson 2: Overview of ConnectEMC GUI (Graphical User Interface) Upon completion of this lesson, you should be able to: Understand each section of the ConnectEMC GUI Identify the components of the connect home process Tool Bar Left Menu Pane Status Pane Info Pane

14. Overview of ConnectEMC GUI(Graphical User Interface) Restore default GUI Menu Bar Toolbar Status Pane Left Menu Pane Info Pane

15. ConnectEMC GUI – Tool Bar Restore the default GUI Splitter toolbar buttons - Pressing an icon alters the GUI in that manner ALL – Test in sequence each current record in the Connection List TEST –the current ConnectEMC configuration sequence (Excluding Symmetrix) INFO – View the info Pane Stop or Resume the ConnectEMC data transfer • Stop, Run, register the GUI buttons are available when the Status View is active

16. ConnectEMC – Info Pane • Displays the characteristics of the site at which ConnectEMC is installed such as service processor name and serial number • Depending on the associated Client, some or all of these fields may be edited

17. ConnectEMC GUI - Left Menu Pane • Connection: identifies connection configuration categories • Events: four categories of Event files • Last Connection Log: shows the currently-active and most recent logs • Logging: shows the list of log files • Settings: lists all variables and attributes from configuration file • Status: invoke status view in GUI • Testing: invoke testing from GUI

18. ConnectEMC – Status Pane • Displays the registered applications producing file transfers • Displays the status of remote control software on that host • Displays the current transfer mode • Displays the status and attributes for each file queued for transfer

19. Lesson 3: End to End connect home triage Upon completion of this lesson, you should be able to: Understand the most probable root cause of a connect home failure Identify the components of the connect home process ConnectEMC triage Memory leak issue EMC Secure Remote Support (ESRS) triage ESRS - Gateway triage Firewall triage Modem triage The following slides will cover different techniques on how to solve AAA1 connect home issues with detailed information on the following topics.

20. ConnectEMC Triage at the Service Processor If a modem site – make sure local modem is set Modem must have Primary and Secondary numbers For the US the ConnectEMC Config must have 4 modem numbers listed All other countries it may be less than 4 modem numbers If modem type is not listed in the OS, modem may be powered off or not configured All modem issues – should check fw_dial.log or DialEMC.log At no time should modem drivers be de-installed or re-installed ( This will require an SP replace) If an ESRS site – make sure Gateway IP is set Verify Customer Network Line is properly set up The FTP and e-mail servers are properly configured Mode is set to Passive under FTP Understanding the possible Memory Leak issue in ConnectEMC Errors on the configuration setup Configuration error Incorrect phone number or IP address Incorrect connect home interval Incorrect information provided by the customer (i.e. IP address or SMTP server) ConnectEMC Service not running within the OS ConnectEMC Application either not installed or not installed in the correct location

21. ConnectEMC Triage at the Service Processor (cont.) Verify SymmConnect is enabled in SYR Verify that SymmConnect(SYR) and ConnectEMC(Site info on the Service Processor) are both set for the same interval (i.e. 1 or 7 days) Do not do a test call from ConnectEMC GUI, it is not supported Perform a test call from scheduler (This will create a Service Request) Can also perform a test call from site information (This does not create an SR) CEC.ini - Provides the version of ConnectEMC to SYR ConnectEMC_Config.xml – Provides ConnectEMC configuration details Here is a list of log files to assist in the triage of the root cause of AAA1 ConnectEMC_xxxx.log ConnectEMC_Dial.log – Provides analog modem connect home log information ConnectEMC_FTP.log – Provides ESRS FTP connect home log information ConnectEMC_SMTP.log – Provides ESRS e-mail connecthome log information ConnectEMC_HTTP.log – Provides Device Client connect home log information

22. Possible Memory Leak issue Check to see if ConnectEMC is running in Services Right click on my computer > Manage > Services and applications > Services Right hand side of the display Look for ConnectEMC service to be stopped Look at the Windows Event Viewer (Application or System) for errors Refer to Primus emc227604 to confirm the Symmetrix has the latest ConnectEMC version and SymmWin fix (apply fixes if needed) Versions with known memory leak issues 3.02.06 and below 4.02.08 and below 5.02.06 Recommended Enginuity level that include the appropriate fixes for memory leak issues: 5771.113.121 or above with 3.02.06 ConnectEMC 5772.105.94 or above with 3.02.06 ConnectEMC 5773.155.107 or above with 4.02.08 ConnectEMC 5874.210.168 or above with 4.02.08 ConnectEMC

23. ESRS Gateway Triage Refer to Master Primus solution emc235600 for all ESRS information Gateway Physical Power Verify the Gateway has power Verify the Gateway is powered on Gateway Server Verify the Gateway server is running Depending on the setup, either FTP or SMTP or both Verify the policy manager status is running correctly Can be verified in DRM (Device Relationship Management System) listed under Gateway Verify Network Connectivity Do we have network connection to the service processor Can be verified in DRM (Device Relationship Management System) listed under Device Has the customer changed the IP address or provided incorrect IP address

24. ESRS Gateway Triage (cont.) FTP Server If FTP is chosen, verify the FTP Server is Running Email (SMTP) Server Verify we have the correct SMTP from the customer for email setup Proxy Server (HTTPS) ConnectEMC_HTTPS.log (For Device Client) Field Watch Services This is a notification of Gateway issues via e-mail DRM (Device Relationship Management) verification Verify that DRM shows ESRS is communicating with the Service Processor Status should equal “ Good” with a green check mark If status is missing or offline using a single Gateway configuration, then communication with the SP is not possible via ESRS Servicelink (ESRS IP Solution) Status should display a green dot next to the serial number of the Symmetrix , there may be a delay up to a few hours displaying the green dot with new installs. If the status is “managed” on the Gateway, you may see a dot color other than green during the implementation. Test connection, if able to connect the Symmetrix, the connection is good.

25. ESRS Triage Determine cause of failure and include the cause of the failure in the SR for field personnel to troubleshoot on their end Timeout searching for xxx.xxx.xxx.xxx Port xx Cannot connect to FTP Server (check firewall section) Cannot connect to Email server (check firewall section) Methods to connect home FTP/s E-mail home is acceptable set as a secondary connection record on Symmetrix (do not set as primary). Please follow primus solutions emc275566 & emc278135 for e-mail home. Minimum Enginuity levels are required (5874.242.192 and above) and ESRS Gateway configured O:\emc\connectemc\logs\cURL.log file is a good file to review to see error messages on the Service Processor to see if test call homes are making it to the ESRS Gateway

26. Firewall Triage Service Processor Firewall Configuration Utility (FCU) Refer to CSP-2068 on the GS Intranet for information on the following checks. Navigate to Application Download Area>EMC Secure Remote Support Gateway>Implementation>Install Procedure ESRS Rules – verify the rules are active within FCU ConnectHome FTP Rules / E-mail Rules Follow the Privatewire / Firewall setup information Firewall Ports open on the customer end Device Client – 8443/443

27. Firewall Triage (cont.) Firewall issue Navigate to - Start>Programs>EMC Apps>Firewall utilities If ESRS is not connecting home, check firewall rules (must be set to active) If connect home leaves the SP but still logging AAA1.58 errors, then it is a Gateway issue (the SP to the Gateway is good, but the Gateway to SYR is bad) verify that SYR received the connect home (see test call home verification slides later in this presentation) There is a Policy Manager that is entirely customer controlled on their network which EMC does not have access. If the connect home is not passing through the Gateway, the customer will need to be contacted to make sure the Policy Manager is set to “Always Allow”. (Note: It is not uncommon for the customer to change the Policy Manager to “Not Allow”. And EMC is not made aware.)

28. Potential modem issues Serial Cable Connection Cable - not plugged in, missing or faulty PBX or POTS Telephone line verification Make sure the RJ11 telephone cable is plugged into the line port No Power You will see “no-modem attached” or “modem not present” in Windows OS modem configuration Verify the modem is plugged in Verify the modem has power Modem & device In modem configuration within Windows , only one modem must be configured Note: If the modem info has been changed or more than one modem has been configured at anytime, a replacement of the service processor is required or reimaged Check for VOIP (digital) – This will show similar symptoms as if using an unsupported modem. (Important: VOIP is not supported)

29. Potential modem issues (cont.) Line Error – Is phone line plugged in and functional PBX – analog/digital (dial home can only use analog) If there are constant phone line issues – fails and succeeds over and over, it is likely a PBX issue POTS (Plain old telephone service) line – usually no dial tone or busy signal issue Modem init string Verify the init string is not blank; listed are examples of the modem part numbers and their corresponding init strings MT2834BA (&E1&E4&E13$R1S10=1000&S0&SF1#F1&E7) MT5634ZBA-V92 (\N3&K3S10-100&S0%E1\X1%C0+MS=V34) MT9432ZBA (\N3&K3S10=100&S0%E1\X1%C0+MS=V34) Some countries require no dial tone detection Specifically to German and Austria - X3 is required at the beginning of the INIT string TAPI issues No Dial tone, Busy, or No Connection

30. Potential modem issues (cont.) Xmodemdebug.log Reports connection speeds, drops Reports basic init string information Modem Debug If you can connect into the system consistently, but the system does not connect home at all, the line quality is suspect and should be investigated. If unable to correct the connect home issue, escalated to a Connectivity SME in the PSE lab for assistance at the following escalation e-mail: SYMMSPAppsConnectivityEscalations@emc.com. Provide as much details in the e-mail as possible. If there are line errors, collect the following logs Xmodemdebug.log Fwdial.log ConnectEMCDial.log

31. Lesson 4: Log files Upon completion of this lesson, you should be able to: Describe the connect home log files for each supported Enginuity level Describe what information each log file contains Understand which log files to review for each connect home failure Locate the log files within the service processor

32. Connect home log file names for supported Enginuity levels 5267 and above Enginuity levels support SymmConnect Dialemc.log 5x67, 5568 and 5670 Enginuity levels, refer to the following files DialEMC.log FW_Dial.log ConnectEMC is supported in 5671 and above Enginuity levels, refer to the following files FW_Dial.log ConnectEMC_Dial.log ConnectEMCDial.log ConnectEMC_FTP.log ConnectEMCFtp.log cURL.log Xmodem.log

33. Connect home log file names for supported Enginuity levels (cont.) 5875 and above FW_Dial.log ConnectEMCFtp.log ConnectEMC_FTP.log 5875 also supports Email (as secondary) – Https – Device Client ConnectEMC_Email.log ConnectEMC_ HTTP.log ConnectEMC Device Client.log

34. Modem Logs Log files can be found under O:\emc\ConnectEMC\logs ConnectEMCDial.log - Contains most recent connection record both failed and successful connect homes ConnectEMCDial_xx.log – Contains Historical data from previous Connection attempts both failed and successful connect homes ConnectEMC_Dial.log – Contains the last connection record for the last attempted connect home fw_dial.log – Contains the connect home connection data provided by the modem status Additional modem information can be found on the GS Intranet: http://www.cs.isus.emc.com/ The path to this information is - Application Download Area>Call Home Information>Call Home Dialing Information>Country-Specific Toll Free Numbers

35. fw_dial.log (example) – Sign of only one phone number listed Description of failure ConnectEMC only has one number listed in the ConnectEMC configuration This is a preventable single point of failure in connect home process Possible cause or resolution List of all recommended connect home phone numbers to be used can be found on the GS Intranet Navigate to - Application Download Area>Call Home Information>Call Home Dialing Information>Country-Specific Toll Free Numbers

36. fw_dial.log (example)– No Dial Tone Detected Description of failure ‘No dial tone detected’ – This example shows a Plain Old Telephone Service (POTS) which does not require a prefix of an 8, or 9, to be used Possible cause or resolution Check Physical Line Connection Telephone line service provider issue using POTS (Plain old telephone service) Wrong or incorrect usage of dial prefix; for example 9, when using a direct line Possible issue with PBX (Private Branch Exchange )system Shared phone line where only one line is available for outgoing calls at a time

37. fw_dial.log (example) – Phone Line is Unavailable Description of failure The Phone Line is unavailable Possible cause or resolution Check Physical Phone Line Connection Possible telephone line service provider issue using POTS (Plain old telephone service) Possible issue with PBX system Verify wrong or incorrect usage of dial prefix; for example using the prefix ‘9,’ or ‘8,’ Verify any possible issue with PBX system on the customer end Shared phone line where only one line is available for outgoing calls at a time

38. fw_dial.log (example) – Busy Line Description of failure Example of the Symmetrix calling home and not getting an answer at EMC Possible cause or resolution Verify the number being dialed is correct Verify the prefix (if needed) is being used Check to see if all numbers are being utilized in the connection setup within ConnectEMC

39. fw_dial.log (example) – Connection Dropped Description of failure Connection dropped during a file transfer or at the connection handshake Possible cause or resolution Poor phone line quality Verify no damage to the hardware (i.e. phone line or serial cable) Verify maintenance is not being performed or was not being performed on the phone line at the time of failure

40. Dialemc.log Log files can be found under c:\Program Files\EMC\DialEMC\Logs Dialemc.log - Contains most recent connection record both failed and successful connect homes Dialemcxx.log - Contains Historical data from previous Connection attempts both failed and successful connect homes FTP-cURL.log - Contains the FTP connection data rscApi.log - Quick Overview of Error

41. Dialemc.log – FTP Description of failure Example below displays an indication the SP is trying to connect to the Gateway and is failing (IP address 151.191.176.107) On the 5th attempt – No file sent to EMC. Displaying ‘Error 3004’ Possible cause or resolution Verify FTP server is running on the Gateway Verify correct IP address is listed in ConnectEMC Verify the Port is open on the Firewall configuration tool and on the customers network (This is a common implementation error)

42. rscApi.log This log is useful to perform a quick overview or historical diagnosis to determine how many failures occurred over a period of time.

43. ESRS Logs – FTP Log files can be found under O:\emc\ConnectEMC\logs ConnectEMCFtp.log - Contains most recent connection record both failed and successful connect homes ConnectEMCFtp_xx.log - Contains Historical data from previous Connection attempts both failed and successful connect homes ConnectEMC_Ftp.log - Contains the last connection record for the last attempted connect home FTP-cURL.log - Contains the FTP connection data

44. ConnectEMCFtp.log – FTP Failure Description of failure ftp could not connect to host Possible cause or resolution Verify IP Address of the Gateway Verify FTP Server is running on the Gateway Verify the Port is open on the Firewall configuration tool and on the customers network Verify no network issues on the customers end occurred at the time of failure

45. FTP-cURL.log This log file shows connection failures as well as connections that succeeded Description of failure FTP Connection failure to the Gateway Possible cause or resolution Verify IP Address of the Gateway Verify FTP Server is running on the Gateway Verify the Port is open on the Firewall configuration tool and on the customers network Verify no network issues on the customers end occurred at the time of failure

46. ESRS Logs – SMTP Log files can be found under O:\emc\ConnectEMC\logs ConnectEMCEmail.log - Contains most recent connection record both failed and successful connect homes ConnectEMCEmail_xx.log - Contains Historical data from previous connection attempts both failed and successful connect homes ConnectEMC_Email.log - Contains the last connection record for the last attempted connect home EMAIL-cURL.log - Contains the SMTP connection data

47. ConnectEMCEmail.log – SMTP Server Failure Description of failure Failed to send email due to failure to connect to SMTP server Possible cause or resolution Verify correct SMTP IP Address for the Gateway Verify the SMTP server is running on the Gateway Verify port is open on the customers firewall and on the Service Processor

48. EMAIL-cURL.log Failure description No connection could be made because the target machine actively refused it Possible cause or resolution Verify the SMTP IP Address is correct for the Gateway Verify the SMTP Server is running on the Gateway Verify with the customer if the Policy Manager is set to “Always Allow”

49. Lesson 5: Test Call Home Upon completion of this lesson, you should be able to: Perform a Test Call from Site Information Perform a Test Call from Scheduler Verify the successful completion of a Test Call Home

50. Performing a Test Call Home From Site Information In SymmWin, navigate to Configuration > Site Information > Call Home > Test SymmWin Call Home now (see screenshot to the right) Before exiting SymmWin, click the Advisories tab and ensure you see the following message: The registration of the file C:\EMC\serial_Number\SymmWin\logs\testcall.log succeeded in the ConnectEMC • For ESRS systems, after initiating the test call, close the active session with SymmWin and terminate the connection to the system. (Exit out of the Symmetrix) • Always close the Symmetrix connection for all analog connect in connections • Proceed toSYR to verify that the Test Call Home was successful (Note: The actual time for the Test Dial Home to appear in SYR can be anywhere from 20 minutes to several hours)