N-Tuples within Gaudi

1. N-Tuples within Gaudi Definition Usage Run through simplified example The persistent back-end

2. ConversionSvc PHist PNTup N-tuple Service JobOptionsSvc AppManager Converter Alg Properties MessageSvc AlgFactory EventSelector Algorithm1 Algorithm1 PObj Algorithm1 PObject PObj PObject PObj PDetElem EventDataSvc DetDataSrv Obj1 TObj1 TObjContainer PersistencySvc TObjContainer DetPerstySvc ObjContainer TDetElem1 TDetElem1 Converter TDetElem1 Converter Converter Converter Converter TObj TObj Converter TObj TObj Obj2 Obj1 T Detector Store AnotherPercySvc NTupleSvc HistogramSvc Transient Event Store HistPerstySvc PHist NTup Hist1 PHist PObj NTup Hist1 PObj NTup1 Hist1 uses PObj creates Converter T Histogram Store T N-Tuple Store navigability What are we talking about? Gaudi N-Tuples

3. What was an N-Tuple in CERNLIB • Collection of identifiable n-dim. primitive items • LOGICAL*4, INTEGER*4, REAL*4, REAL*8 • See HBOOK • Row wise N-Tuples • REAL*4 array(SOME_DIMENSION) • Items are single numbers identified by tags • Column wise N-Tuples • Structure mapped to a common block • Items (number, array, …) are identified by tags • Allow for variable size arrays Gaudi N-Tuples

4. What is an N-Tuple in Gaudi • Collection of N-tuple Items • Items represent primitives: bool, long, float, double • N-tuples reside in a data store • Can be accessed like any other DataObject • Data management (e.g. writing records) is done by the N-tuple service Things did not change dramatically Gaudi N-Tuples

5. What’s the job: Wishes • Stick to architecture • We want to independent from the “back-end” storage mechanism (HBOOK, ROOT, OBJY...) • Cernlib will die • Encapsulate implementation: Interfaces or ABC • Easy to use • Items must behave like ordinary numbers • Must support basic arithmetic operations and type conversions • N-tuples should not go all into one single file • possibly, but not always • Connect several streams to the data store Gaudi N-Tuples

6. The N-Tuple Data Store • Top Entry point /NTUPLES • Logical file names • User defined • First useable directory/MC/REC/USER • Directory structure • N-tuples • 1,2,3,4 Gaudi N-Tuples

7. The Example • Simplified version of GaudiExample/Ntuples • Explain how to write and read a column wise N-tuple • Differences to row wise N-tuples are minor • Only record structure differs • N-tuples should be easy to use • Tell me if you think stuff is overcomplicated • I do not use them daily Gaudi N-Tuples

8. Easy to Use… (Hopefully) • Define data itemse.g. as members of filling Algorithm • Items are based on primitives • bool, long, float, double // Items for n-tuple // _________________ // // 0 dimensional item (=number) NTuple::Item<long> m_ntrk; // 1 dimensional items (Arrays) NTuple::Array<float> m_px, m_py, m_pz; // 2 dimensional items (Matrices) NTuple::Matrix<long> m_hits; Gaudi N-Tuples

9. Easy to Use… (Hopefully) • Book and add the items you later want to fill • Book the N-tuple NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1"); if ( !nt ) { nt = ntupleService()->book("/NTUPLES/MC/1”, CLID_ColumnWiseTuple, "Hello World"); if ( nt ) { status = nt->addItem ("Ntrack", m_ntrk, 0, 5000 ); status = nt->addItem ("px", m_ntrk, m_px); // m_hits[0:m_ntrk][0:5]; numbers within [0,8] status = nt->addItem ("hit", m_ntrk, m_hits, 5, 0, 8 ); } } NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1"); if ( !nt ) { nt = ntupleService()->book("/NTUPLES/MC/1”, CLID_ColumnWiseTuple, "Hello World"); } • Add items Gaudi N-Tuples

10. Easy to Use… (Hopefully) • Fill all items and commit record • Simple calculations can be done directly using the items • Do not overrun dimensions for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; } for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; m_px[m_ntrk] = track[m_ntrk]->px(); m_hits[m_ntrk][0] = track[m_ntrk]->adcValue(0); ... m_hits[m_ntrk][4] = track[m_ntrk]->adcValue(4); } for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; m_px[m_ntrk] = track[m_ntrk]->px(); m_hits[m_ntrk][0] = track[m_ntrk]->adcValue(0); ... m_hits[m_ntrk][4] = track[m_ntrk]->adcValue(4); } // Commit record (here of a column wise N-tuple) status = ntupleService()->writeRecord("/NTUPLES/MC/1"); • Fill items • Reset values to defaults: • Is this good? • Commit record Gaudi N-Tuples

11. Easy to Use… (Hopefully) • At the end of the job • All N-tuples will automatically be closed • The directory structure of the data store is reflected by RZ directories • Access N-tuple for reading in the same way as for writing • Don’t forget to “cd //<lun>/<dir>” in PAW Gaudi N-Tuples

12. Easy to Use… (Hopefully) • Will open the corresponding logical file • Search for the directory • Load the N-tuple • Reading back (…or with PAW) NTuple::Item<long> ntrk; NTuple::Array<float> px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { } NTuple::Item<long> ntrk; NTuple::Array<float> px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { status = nt1->item ("Ntrack", ntrk ); status = nt1->item ("px", px); ... } NTuple::Item<long> ntrk; NTuple::Array<float> px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { status = nt1->item ("Ntrack", ntrk ); status = nt1->item ("px", px); ... do { float sumPx = 0; if(ntupleService()->readRecord(nt.ptr()).isSuccess()){ for ( long j = 0; j < ntrk; j++ ) sumPx += px[j]; } } while ( status.isSuccess() ); } • Assign the entries to the N tuple • Loop over records Gaudi N-Tuples

13. Easy to Use… (Hopefully) • Configuration (using job options file) • Reading N-tuples NTupleSvc.Input = { ”MC#tuple1.hbook", ”USER#tuple2.hbook" }; // Persistency type of the N-tuple service: 6=HBOOK NTupleSvc.Type = 6; • Writing N-tuples NTupleSvc.Output = { ”MC#tuple1.hbook", ”USER#tuple2.hbook" }; // Persistency type of the N-tuple service: 6=HBOOK NTupleSvc.Type = 6; Gaudi N-Tuples

14. Back-End: HBOOK Limitations • Limited type information • Cannot define all “C” data types • Therefor only: bool, long, unsigned long, float and double • NOT: char, short, int (+unsigned) • Does not affect disk space: HBOOK compresses internally • N-tuple name is an “integer” • “integer” translates later to HBOOK ID • Must be constant independent of booking time for kumacs • Do not mix up N-tuple IDs and Histogram Ids • There is only one PAWC common block! Gaudi N-Tuples

15. Back-End: HBOOK Limitations • Only one index variable per item • RZ directory names are CHARACTER*8 • … and upper case ONLY • double/REAL*8 data • PAW has trouble: better don’t use • REAL*8 must be quadword aligned (64 bits) • Padding must be done by user • Not more than 1000 REAL*8 entries per N tuple • Row wise N-tuples • Type information is stored as first “event” • Start reading at the second... Gaudi N-Tuples

16. Other Back-Ends • The Present: HBOOK • PAW is the most widely used viewer • N-tuples don’t help if they can’t be analyzed interactively • The Future (?): ROOT I/O • Will give ROOT a try as back-end • Implementation with trees/branches is straight forward • Would solve probably all the limitations of HBOOK Gaudi N-Tuples

17. Conclusions • It is possible to define dynamic structures (N-tuples) with Gaudi • If you think the usage is too complicated, please say so! • Otherwise it will never change • Depending on the back-end, limitations invade transient side Gaudi N-Tuples