Infrastructure to Support Extreme Self-Service

1. Infrastructure to Support Extreme Self-Service Session #20653 March 15, 2006 Alliance 2006 Conference Nashville, Tennessee

2. Presenter • Carol Jordan • I have been working on PeopleSoft applications since 1997, and have been at Stanford since November 2000. • I am the Infrastructure Manager for the team supporting Student and HR Systems. • In addition to the PeopleSoft products, our group also supports Kronos, Resource 25, OnBase Imaging, Resumix and a custom Workflow application.

3. Overview • Before PeopleSoft: • - Limited functionality for students • - …using a web-enabled mainframe application • …allowing only 35 concurrent student sessions • With PeopleSoft HR/SA and Portal: • Many functions for students • …and for faculty, advisors and staff • …and no portal sessions are turned away

4. Agenda/Contents • In this presentation: • Extreme self-service defined • Our rough start • Where we are now, how we got here • What we learned

5. Overview: Stanford University • Stanford University is a private university located 30 miles south of San Francisco and just north of Silicon Valley. • There are currently 6,700 undergraduate and 8,000 graduate students enrolled, with 1,775 faculty. • Founded in 1891, the university was established by Jane and Leland Stanford in memory of their son, Leland Stanford, Jr., who died of typhoid fever at the age of 15.

6. Overview: Application Administration The Application Administration team is part of Administrative Systems, supporting many of the applications used campus-wide. There are three AppAdmin teams, supporting PeopleSoft, Oracle Financials, and Reporting. The team supporting PeopleSoft is made up of 5 System Analysts, 3 Oracle DBAs and 3 Windows System Administrators. Developers and other support staff are part of a separate team within Administrative Systems.

7. Overview: Application Administration • PeopleSoft products currently installed: • Campus Solutions 8 SP1 • PeopleTools 8.22.05 • Enterprise Portal 8.8 SP1 • PeopleTools 8.44.03 • Enterprise Learning Management 8.8 SP1 • PeopleTools 8.45.12 • All on Oracle 9.2.0.6

8. Extreme Self-Service, defined • Lots of functionality for students, faculty, advisors and staff • and • Lots of users – close to 35,000 logins on our busiest days

9. Extreme Self-Service, defined • Lots of functionality for students • File or adjust study list and elect grading options • Request an official transcript to be mailed • Print history of courses and grades • Print Enrollment Certification • Declare major and minor • Apply to graduate • Update addresses, emergency contacts • Maintain FERPA elections • View and accept financial aid • View student bill and print statement • View advisors • View degree progress, milestones, program summary • Coming in March…enter or read course evaluations • Coming in April…pay student bill, add money to card plan • Links to other applications: Student Housing, IT Services bill (Pinnacle), Office of Development

10. Extreme Self-Service, defined • Lots of functionality • for Faculty • Review class lists • Send e-mail to students in a class • Submit and change grades • Assign a grade proxy • Review previously-submitted grades • Coming in March…read students’ course evaluations • for Advisors • View lists of current and past advisees • View study lists, grades and unofficial transcripts for current advisees • Place and release holds of current advisees • for Staff • View on-line pay statement • Make W-4 elections • Enter direct-deposit elections • Register for training, take on-line tests through PeopleSoft ELM • Link to applications: Kronos, on-line W-2

11. Extreme Self-Service, defined Lots of users:

12. Our Rough Start • Upgrade to Version 8 went OK • First couple of months went OK • …but the system could not support the sustained demand of our busiest days – the first two days of the new academic year

13. Our Rough Start • Stanford Daily articles with headlines like, “Axess Problems Plague First Week” • What students saw: • log-in attempts that failed • ‘connection refused’ errors between HR/SA and Portal • pages that didn’t load

14. Our Rough Start • CIO magazine article: ‘Big Mess on Campus’ • “Stanford's IT was still struggling with integrating the enterprise systems when the newly launched PeopleSoft Web portal (called Axess) crashed last fall…Axess couldn't handle the load of all the returning students trying to log in…”

15. Where we are now • We are able to handle high load without issues • Business offices are planning to roll out new functionality using the Axess portal • …and we’ve managed to stay out of the papers for a while (fingers crossed)

16. Obligatory Hardware Diagram

17. How We Got Here • Upgraded hardware, added hardware • Server hosting custom single sign-on was the biggest, earliest bottleneck • Original web servers were old and slow • Got the OK to order new hardware after capacity issues hit • Lessons: • Go with your gut…if you’re worrying about it, fix it; if you can’t fix it, have a contingency plan • Fight the requests to minimize costs by re-using old hardware

18. How We Got Here • Web-server changes for HR/SA and Portal • Upgraded JRE • Upgraded Jolt and WebLogic to the latest rolling patch • Reduce network disconnect timeout • Used WebLogic console to fine-tune JVM heapsize (768M) and thread-count (100) on Windows web servers • Set Windows PIA service to restart automatically if it goes down • Installed two logical web-servers per physical server to improve memory utilization under Windows • Separated administrative and self-service users, to set different time-outs, breadcrumbs, and other configuration parameters for different types of users

19. How We Got Here • Application-server changes for HR/SA and Portal: • Upgraded JRE • Upgraded Tuxedo to latest rolling patch • Reduced client-cleanup timeout • For Portal and ELM: implemented shared application-server cache • For Portal and ELM: re-configured Tuxedo domains to eliminate spawning – always start the maximum number of PSAPPSRVs • 35 per server for each of two Portal application servers • 8 per server for each of two ELM application servers • 15-25 per server for each of two HR/SA application servers

20. What we learned • We can’t predict load with certainty • High load is loosely tied to the beginning of the term, when fees are due and students need to know where their classes are • …so we implemented defensive monitoring to send an e-mail when at 400+ logins per quarter-hour • Sample e-mail text:The Axess login count as of 00:16, 03/13/2006 is 550 in a 15-minute interval.

21. What we learned Because we count logins, we can see patterns in user activity: 03/12/2006 22:30 57 03/12/2006 22:45 66 03/12/2006 23:00 94 03/12/2006 23:15 80 03/12/2006 23:30 105 03/12/2006 23:45 240 03/13/2006 00:00 550 03/13/2006 00:15 197 03/13/2006 00:30 181 03/13/2006 00:45 128 03/13/2006 01:00 100 03/13/2006 01:15 90 03/13/2006 01:30 76 03/13/2006 01:45 57 Spring enrollment opened at 12:01am on Monday 3/13

22. What we learned Monitor at many levels • At the web-server layer: • We monitor the load-balanced site • …and individual (logical) web servers • …and the HTTP port used by the load-balancer probe • At the application-server layer: • We monitor for Tuxedo processes • UNIX SysAdmins monitor memory usage, add swap space as needed • We monitor for CPU utilization (sample e-mail follows)

23. CPU monitor – sample e-mail Info from sar ====================================== Time: 00:40:10 usr%: 77 sys%: 10 wio%: 0 idle%: 14 Top output ====================================== load averages: 4.76, 9.29, 15.18 00:40:10 106 processes: 101 sleeping, 1 zombie, 4 on cpu Memory: 8192M real, 1143M free, 6577M swap in use, 18G swap free PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 26073 a2k_prd 4 0 0 203M 187M cpu0 4:40 10.43% PSAPPSRV 9800 a2k_prd 4 58 0 256M 238M sleep 13:49 8.83% PSAPPSRV 10991 a2k_prd 4 50 0 245M 227M sleep 11:18 7.61% PSAPPSRV 10819 a2k_prd 4 58 0 267M 251M sleep 13:33 6.82% PSAPPSRV

24. What we learned Use your PeopleSoft tools! • PeopleSoft Performance Monitor for Portal and ELM • WebLogic console for web servers

25. What we learned • Keep Tools up-to-date – watch for BEA updates • Move AppMessaging to a dedicated host – keep back-end processing out of the transactional infrastructure • Nobody hits ‘logoff’ – configure your web servers to close idle connections quickly • Performance testing is mandatory for new projects • Configuration review, top-to-bottom, with PeopleSoft Consulting was helpful

26. Questions?

27. Contacts • Carol Jordan • Infrastructure Manager, Student and HR Systems • Administrative Systems • Stanford University • E-mail: cajordan@stanford.edu • Jeff Butler • Lead System Analyst, Student & HR Systems • Stanford University • E-mail: jbutler@stanford.edu • Ross Shimabukuro • System Analyst, Student & HR Systems • Stanford University • E-mail: ryshima@stanford.edu

28. This presentation and all Alliance 2006 presentations are available for download from the Conference Site Presentations from previous meetings are also available