The mechanics of EPCC

1. The mechanics of EPCC How EPCC learnt to do technology transfer and software engineering the hard way

2. Structure of talk • What is EPCC today • Learning to deliver on time • The structure of a commercial project • Software development and OGSA-DAI • Project management • Questions and discussion The mechanics of EPCC

3. HPC Research Facilities Technology Transfer Training European Coordination Visitor Programme EPCC Activities • Europe’s largest, most successful supercomputing centre – 15 years old • Vital statistics: • 65 staff • £3.2M turnover (almost) all from external sources • Multidisciplinary and multi-funded • with a large spectrum of activities • and a critical mass of expertise • Strong engagement with industry • from local SMEs to large multinationals • project based consultancy services • Supports research at University of Edinburgh via • access to facilities • training and support • TRACS visitor programme • NeSC • founding partner of National e-Science Centre • Wide variety of leading-edge systems • 1,600 processor HPCx system • 2,000 processor IBM Bluegene/L • 12,000 processor QCDOC • New investment in Advanced Computing Facility • EPCC has a unique breadth of expertise in high performance computing The mechanics of EPCC

4. Commercial activities today • Bespoke software development and software project management for business • network, cluster and high-performance computing • novel application areas • from mushrooms to internet packets • Start-to-finish projects • full software development lifecycle, 3 - 12+ months • most commercial projects are < 6 months • Operate like a business • Commercial Group brings in business, Software Development Group delivers • charge at commercial rates ($1,000 per day) • very delivery focused • all commercial contracts are fixed cost • funded by cash contracts, public funds and European Commission, EU • many of the smaller projects are supported by SE The mechanics of EPCC

5. Clients UK: o Almond Engineering Ltd o Altamira Ltd o Arran Aromatics Ltd o Callanders Sawmills Ltd o Calman Ltd o CB Technology Ltd o Centre for Customer Awareness Ltd o CERN o Cheltenham & Gloucester plc o DTI o Digital Bridges Ltd o Elektrobit Ltd o First Group plc o Golden Crumb Ltd o High Speed Productions Ltd o Integriti Solutions Ltd o IP Technology Ltd o Ironside Farrar Ltd o Jardine Technology Ltd o Pepper’s Ghost Productions Ltd o Radar World Ltd o Red Lemon Ltd o Rosti (Scotland) Ltd o Quadstone Ltd o SCI Ltd o Scottish Enterprise o The Crown Office o TSB Bank Scotland Ltd o UK Meteorological Office o Upstream Systems Ltd o Alpha Data Parallel Systems Ltd o Nallatech Ltd UK: o Almond Engineering Ltd o Altamira Ltd o Arran Aromatics Ltd o Callanders Sawmills Ltd o Calman Ltd o CB Technology Ltd o Centre for Customer Awareness Ltd o CERN o Cheltenham & Gloucester plc o DTI o Digital Bridges Ltd o Elektrobit Ltd o First Group plc o Golden Crumb Ltd o High Speed Productions Ltd o Integriti Solutions Ltd o IP Technology Ltd o Ironside Farrar Ltd o Jardine Technology Ltd o Pepper’s Ghost Productions Ltd o Radar World Ltd o Red Lemon Ltd o Rosti (Scotland) Ltd o Quadstone Ltd o SCI Ltd o Scottish Enterprise o The Crown Office o TSB Bank Scotland Ltd o UK Meteorological Office o Upstream Systems Ltd o Alpha Data Parallel Systems Ltd o Nallatech Ltd 2000 - 2004 USA: o Cisco Systems Inc o Sun Microsystems Inc o IBM Corporation o Oracle Corporation o Hewlett Packard o Microsoft o Xilinx Corporation Japan: o Hitachi o NEC Europe o Fujitsu Labs Europe Europe: o European Commission + many EU project partners The mechanics of EPCC

6. Business Strategy • to solve business problems NOT sell technology • … individual solutions for clients Technology Push Down Academic Research Project size OGSA-DAI £ X,000,000 SunDCG PGPGrid £ X00,000 First Group CCA IPO Microsoft £ X,000 Autoscreen C&G The mechanics of EPCC

7. How do we work? Heroic Programming! All-nighters! Work weekends! No. The mechanics of EPCC

8. How do we work? • Take pride in a professional approach • Work in small project teams • Project leader, 1-6 developers, technical reviewers • Use documented engineering & management processes • Project management based on PRINCE2 • Engineering using agile methods • Built from experience and industry best practice • Iterative/staged development techniques • Requirements triage • Test-driven development • Tuned to the leading edge of innovative software development The mechanics of EPCC

9. Who does the work? • Currently around 4 business development staff • 2 focus on business development • 2 focus on marketing and publicity • Currently around 20 engineering staff • Three full-time project managers, two software architects • c. 15 consultants and principal consultants • Staff backgrounds – maths , physics, computer & life sciences • Over 100 staff-years of experience, over 1/3 from industry • Typical skills • Java, C/C++, Visual Basic/C#, Perl, Fortran • Distributed computing, webservices, XML, J2EE, MPI, OpenMP • Databases, SQL, JDBC, XML-DB • Software engineering, OO design, UML The mechanics of EPCC

10. EPCC’s early history • Established in 1990 • focus for interest in parallel computing within Physics and CS • Early years largely supported by UK Government “Parallel Applications Programme” • made lots of money working with large UK corporations to optimise/parallelise their codes • How did our funding model come about? • from a belief in the self-funding of Universityresearch • we’ve shown it can be done but it’s verydifficult • it did mean we had to work with industry fromthe beginning The mechanics of EPCC

11. EPCC history (continued) • 1990-1994 • funded by UK Government Parallel Applications Programme • grew to 65 staff • many parallelisation projects with UK industry – aerospace, nuclear, oil & gas etc etc • span out company – Quadstone • 1995-1996 • as Gov money dried up so did projects • had to move from long term projects (18 months) to much shorter projects (3-6 months) • major problem – project / cost overruns • nearly had to make many staff redundant The mechanics of EPCC

12. EPCC history (continued) • 1997-2000 • successfully moved markets from large-scale industry to SMEs • opportunities focussed around successful EU TTN project • projects 3-6 months in duration • embarked on having a repeatable process • 2000-now • over the past few years moved into Grid computing • continued to work with industry • wide variety of projects: • OGSA-DAI – data access & integration for the Grid • Intersim – packet level modelling of differentiated services • Golden Crumb – automatic mushroom selection in factory • Cheltenham & Gloucester – data mining for mortgage industry The mechanics of EPCC

13. How does EPCC work today? • We have well developed project processes • Two linked processes • software development process • project management process • Will illustrate software development process using OGSA-DAI as example • Recently moved to PRINCE2 project management methodology The mechanics of EPCC

14. The project lifecyle • Commercial Group identifies clients and initiates discussions • Following initial discussions CD and technical staff visit company to discuss requirements • High level design written – timings / costs agreed • may involve free code survey at this point • Contract negotiated – fixed price – includes detailed workplan based on design • Project handed to IS – staff scheduled according to skills • All projects have • Project Leader, Applications Consultant, Technical Reviewer • Regular meetings between IS and CG • CG act as account manager to company / funder The mechanics of EPCC

15. OGSA-DAI • Data Access and Integration for databases resources on the Grid • Aim to deliver application mechanisms that: • Meet the data requirements of Grid applications • Functionally, performance and reliability • Reduce development cost of data centric Grid applications • Provide consistent interfaces to data resources • Acceptable and supportable by database providers • Trustable, imposed demand is acceptable, etc. • Provide a standard framework that satisfies standard requirements • A base for developing higher-level services • Data federation / Distributed query processing • Data mining • Data visualisation The mechanics of EPCC

16. ESNW, Manchester OGSA-DAI team NeSC, Edinburgh EPCC Team, Edinburgh NEReSC, Newcastle IBM Dissemination Team IBM Development Team, Hursley The mechanics of EPCC

17. Software Process and Teams REVIEW Programme Board Technical Review Board Peer Review and Inspection Technical Reviewer Users’ Group Continual process → Reqs. Design Implement QA Ingest DEVELOPERS Deep track features Nightly unit + system tests Release Dissem. Prototype Testing System tests based on reqs Additional test cases Test Cases Fix Bugs Support Training USERS Use Cases Prioritisation Contribs Requests The mechanics of EPCC

18. Working together • No more heroes any more • the lone researcher can get into trouble • so don’t do it! • use teams even for small projects • a task leader to keep the bigger picture in mind • a “reviewer” as a technical foil for the developer • distributed extreme programming doesn’t work • be sensible! • Code needs owners • and joint ownership doesn’t work • Java packages and CVS module provide useful boundaries • “buddy” system worked well for a team of 10-12, not as well for 5 • we now have 80,000 lines of Java code + 30,000 lines of documentation The mechanics of EPCC

19. An agile approach to development • Agility is all • Grid/HPC environments and problems = complex systems • complex systems = big, complex projects • big, complex projects = high risk of failure • adopting incremental approaches to requirements, design, and implementation helps minimise risk • delivering small increments regularly is good • good for quality, for visibility, for morale • Keep your eyes on the road • keep an active eye on project risks • think about what happens if this goes wrong • just thinking about it reduces the likelihood it’ll happen! The mechanics of EPCC

20. Releasing software • No release schedule = no releases • don’t timebox research, but do timebox development • HPC is fun and exciting - beware feature creep! • “how’s the project?” • “oh, we’re 95% there” (and always will be…) • frequent release milestones focus developers • but don’t overspecify what will be released • OGSA-DAI had the opposite problem • three months too short • six months about right • major/minor/patch/”special brew” • set your testing timetable in stone The mechanics of EPCC

21. Know your requirements • Requirements, requirements, requirements • write ‘em down! Give ‘em numbers! • remember, requirements aren’t just functional! • whatever they are, they are always testable • tests on HPC systems may be tricky, but thatmakes it fun! • MoSCoW notation is good • Must, Should, Could, Won't • “how important are Priority 3 requirements again..?” • OGSA-DAI had lots of requirements • but make sure you can understand their worth • real users are often better than good ideas • a user group helps to focus development as software matures The mechanics of EPCC

22. Return, recycle, reuse • Throwaway prototypes never are • “once I’ve proved this, I’ll junk the code” • no, you won’t (or your grad student won’t) • apply some basic process even to trivial codes • even reuse of “good” code is sometimes wrong • OGSA-DAI started with high ideals • beware the big ball of mud • patterns in architecture • “Shantytown” • enables quick exploration of feature territory • must be built on a strong central foundation • must include council legislation aka testing The mechanics of EPCC

23. OGSA-DAI Dashboard The mechanics of EPCC

24. Can I see your documents please? • Document! Document! Document! • Imagine trying to program without a language reference • structure and stability is good • Get people who like writing documents to do them • but get everyone to doc their code • a single editor can provide guidance • Good code documentation can be used by the tooling • Good human documentation will win your users support • Make sure you don’t underestimate the cost • code maintenance and documentation takes longer than code development • make it part of the process The mechanics of EPCC

25. People power • Social engineering is the key • Push decisions down to the developers • “Too many chiefs” • make sure you know what are the key battles to win • Have a process for change • or one person will become very unpopular • developers and managers both think they know what’s best • Understand your teams • different people like working in different ways • no one style for management in OGSA-DAI • Competition is good • go one better The mechanics of EPCC

26. The big picture • Balance the hype • software engineering is about vision vs effort vs requests • expectation management is important • researchers, developers, users and funders are all different • and all want different things • the larger the project, the harder it falls • Listen to your users • useability is good • it has to install easily • don’t change your interface • client tooling helps • support helpdesk is better • user groups are interesting The mechanics of EPCC

27. Software development summary • Agile methods are very sympathetic • the Agile founders disliked Rigid Inflexible Processes too! • Adopt a simple process and toolset • even lightweight process really pays off • scoping, requirements and risk analysis up front • incremental approach to design, develop, test • learn some basic tools (they’re even free!) • distributed teams are hard to manage strictly • distributed management is even harder • Listen to your customers – they always know best The mechanics of EPCC

28. Project Management • All technical staff have a line manager and at least one project leader • Procedures are well documented and have grown up over time • Recently we have moved to PRINCE2 project management methodology for commercial projects • seems to work well but is a bit of a culture shock • We employ staff specifically for project management • All staff time is logged – planned and actual • A working day has two blocks of 3 hours • Staff can bid for time to do research / proposal writing The mechanics of EPCC

29. What is PRINCE2? • PRojects IN Controlled Environments version 2 • A project management standard produced by UK’s Office of Government Commerce (part of DTI) • “PRINCE2 is a process-based approach for project management providing an easily tailored, and scalable method for the management of all types of projects” • PRINCE2 is a de facto UK PM standard • becoming mandatory in the public sector (Gov, NHS, Police) • becoming PM method of choice in business • Unilever, GlaxoWellcome, Tesco, BT, Sun, TSB, NatWest, Norwich Union, Centrica, Cable & Wireless… • becoming widespread in Europe too • PRINCE2 is internationally recognised and respected The mechanics of EPCC

30. What is PRINCE2 not? • PRINCE2 is not a software engineering method • but it grew out of an IT environment • and it fits well with traditional or agile development methods alike • PRINCE2 will not help you code better • but it will help you deliver better quality products, on time • and will stop you falling out with your boss/staff • PRINCE2 will not tell you how to write software • but it will leave you alone to write software your way • PRINCE2 is not a silver bullet • but it’s general, flexible and tailorable and – most importantly – it’s based on common sense The mechanics of EPCC

31. PRINCE2 in a nutshell • Projects have a clear Business Case or they don’t happen • “remind me again why we’re doing this project?” • Projects have a beginning, a middle and an end • clearly defined – they start and they stop – they don’t weeble on forever • Projects run in stages with clearly defined boundaries • get a clear picture of how we’re all doing • Product-based planning focuses on deliverables not tasks • think “what do we have to make?” • Layered management: corporate, board, project, team • each level has a clearly defined interface with the others • Management by exception • if there are no problems, just carry on – management don’t meddle • Change is fundamental: change management is intrinsic • assume things will change and plan accordingly The mechanics of EPCC

32. The PRINCE2 process diagram Corporate or Programme Management Project Mandate Directing a Project Starting up a Project Initiating a Project Controlling a Stage Managing Stage Boundaries Closing a Project Managing Product Delivery Planning The mechanics of EPCC

33. PRINCE2 Components • As well as the processes there are several complementary components… • The Business Case • a key driver – the Why? for the project • either a genuine (commercial) business case or at least a set of compelling reasons • owned by the Executive • monitored throughout the project • if the BC goes away, the project should be stopped • The Project Organisation • describes the four management layers • corporate, board, project, team • everyone should have a job description • make roles and responsibilities clear The mechanics of EPCC

34. PRINCE2 Components (2) • Plans • product-based, as discussed above • write product descriptions for key products • Controls • divide the project into Management Stages • a Stage is “as far ahead as you can plan in reasonable detail” • typically a few months • define reports, meetings etc. • Tolerances • allowed variations in time, budget, scope before escalation triggered • “you have six months, +/- 1 month” • “you must satisfy these requirements; those are optional this stage” The mechanics of EPCC

35. PRINCE2 Components (3) • Quality • the project must define methods for QC and test • quality checks should be built in to the MP process • Risk • think about it, monitor it • one of the best management tools is to ask “what might go wrong?” • and create plans to handle it if it does • Configuration Management • keep track of product versions and histories • software version control tools are a good way of implementing this The mechanics of EPCC

36. PRINCE2 Summary • PRINCE2 is a powerful, flexible, scalable PM approach • It’s based on industry best practice • rooted in software development projects • Provides good, intelligent layers of management control • Formalises, in a positive way, customer relations • Can fit easily with agile software development • It’s the only PM approach with internationally recognised qualifications The mechanics of EPCC

37. Final comments on working with industry • Wear a tie! • Remember that the person you’re meeting is just as nervous of meeting a mad academic as you are of meet a rapacious capitalist • The managing director of the company may be drunk • Always apply Denis Healey’s law of holes: “When in one stop digging” • If you’re going to deliver late – tell the customer straightaway • Listen listen listen!!! It’s the only way to get business The mechanics of EPCC

38. Questions / discussion ? The mechanics of EPCC