1 / 37

Building an MXG PDB from NTSMF Data

Building an MXG PDB from NTSMF Data. Chuck Hopf chuck@chopf.com www.chopf.com/geekland.shtml. Building an MXG PDB from NTSMF Data. ‘Classic’ MXG PDB Structure Why it doesn’t work What needed to change How it is implemented for NTSMF Using BLDNTPDB. Classic MXG PDB Structure.

yepa
Download Presentation

Building an MXG PDB from NTSMF Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building an MXG PDB from NTSMF Data Chuck Hopf chuck@chopf.com www.chopf.com/geekland.shtml

  2. Building an MXG PDB from NTSMF Data • ‘Classic’ MXG PDB Structure • Why it doesn’t work • What needed to change • How it is implemented for NTSMF • Using BLDNTPDB

  3. Classic MXG PDB Structure • Read ALL the data and place it in a PDB with possibly CICSTRAN on the side • Run daily - daily contains detail of ALL data read • WEEKBLD consolidates dailies into weeklies • MONTHBLD consolidates weeklies into monthlies

  4. Classic MXG PDB Structure • DASD storage required: • 8 daily PDB • 5 weekly PDB • enough WORK space for the largest dataset (STEPS or TYPE74)

  5. Why It Doesn’t Work • 40 GB of SMF data daily • Needs 160GB of WORK space (SORTWK and WORK) • 320GB of space for daily datasets • 1+ Terror Bytes for weekly datasets • Time - Daily jobs that run for more than a day are not practical

  6. What Needed to Change • Multiple Parallel Threads • Run week-to-date • Run month-to-date • Eliminate weekly/monthly jobs • Exploit summarization macros where needed to reduce volume

  7. How Is It Implemented for NTSMF? • BLDNTPDB - a very large and complex collection of SAS MACROs that allows the user to choose how the PDB should be structured. • Can use the classic MXG structure or... • Can use virtually any structure the user chooses

  8. Using BLDNTPDB • Parameters and Meanings • Examples of use • Future Directions • ?????????????????????

  9. Using BLDNTPDB - Parameters and Meanings • NTSMF=C:\MXG\NTSMF\NTSMF.SMF - where is your RAW NTSMF data? • WEEKSTRT=MON - on what day of the week does your week start? The normal MXG default is Monday. This controls when weekly processing takes place. • RUNNTINT=YES - should we run the NTINTRV summarization?

  10. Parameters and Meanings • ASUMDUR=HOUR - what level of summarization should we apply? Valid values are QTRHOUR, HALFHOUR, HOUR, SHIFT, DATE, WEEK, MONTH • KEEPERS=null - A list of datasets to be ‘kept’ (usually not used) • DROPPERS=null - A list of datasets to be ‘dropped’ (built but not kept in the output PDB.)

  11. Parameters and Meanings • ZEROOBS=null - Similar to DROPPERS but datasets are created with 0 observations • MAXDRVS=26 - Maximum number of logical drives/physical drive (limits and array size.) • RERUN=null - If YES is specified forces the rewriting of an old PDB USE WITH CAUTION

  12. Parameters and Meanings • MON=C:\MXG\MON • TUE=C:\MXG\TUE • WED=C:\MXG\WED • THU=C:\MXG\THU • FRI=C:\MXG\FRI • SAT=C:\MXG\SAT • SUN=C:\MXG\SUN • Dataset placement for DAILY PDBs. If a null string is used, the dataset WILL NOT BE BUILT.

  13. Parameters and Meanings • WEEK=C:\MXG\WEEK • WEEK2=C:\MXG\WEEK2 • WEEK3=C:\MXG\WEEK3 • WEEK4=C:\MXG\WEEK4 • WEEK5=C:\MXG\WEEK5 • Placement for weekly datasets. If a null string is used, the dataset WILL NOT BE BUILT.

  14. Parameters and Meanings • TREND=C:\MXG\TREND • MONTH=C:\MXG\MONTH • MONTH1=null • Placement for monthly and TREND datasets. A null string results in the dataset NOT BEING BUILT.

  15. Parameters and Meanings • RUNDAY=YES - Run the BUILD process daily and build the specified daily databases • RUNWEEK=NO - YES means run the weekly BUILD process if it is the day specified by WEEKDATE and WTD means build WTD datasets daily • RUNMONTH=NO - YES means run the monthly BUILD process on the 2nd day of the month and MTD means build MTD datasets daily • RUNTREND=WEEKLY - Build the TREND datasets when the WEEKLY BUILD is run. If DAILY is specified, build the TREND datasets when the DAILY datasets are built.

  16. Parameters and Meanings • RPTRAW=NO - Build a table of the distribution of the RAW NTSMF record types • RPTDAY=PDB - Create reports for the most recent PDB at the hourly • RPTWEEK=NO - Create reports for the WEEKLY PDB if it has just been • RPTMONTH=NO - Create reports for the MONTHLY database if it has just been created • RPTTREND=NO - Create reports for the TREND database if it has just been updated

  17. Parameters and Meanings • GRAFDAY=PDB - Create reports for the most recent PDB at the hourly • GRAFWEEK=NO - Create reports for the WEEKLY PDB if it has just been • GRAFMNTH=NO - Create reports for the MONTHLY database if it has just been created • GRAFTRND=NO - Create reports for the TREND database if it has just been updated

  18. Parameters and Meanings • Any combination of RUNxxxx, RPTxxxx, and GRAFxxxx specifications may be used. You can do DAILY, WEEKLY, and MONTHLY runs in a single execution as well as creating reports and graphs at each level.

  19. Parameters and Meanings • Null strings passed as some parameters means ‘dont do this!’

  20. Examples of Use • Classic MXG Structure • PDB for each day of week, 5 weeks, monthly with week run on Monday • %BLDNTPDB(RUNDAY=YES, RUNWEEK=YES, RUNMNTH=YES, RUNTRND=WEEKLY);

  21. Examples of Use • Single daily PDB no WEEKLY/MONTHLY/TREND • %BLDNTPDB(RUNDAY=YES,MON=, TUE=, WED=, THU=, FRI=, SAT=, SUN=);

  22. Examples of Use • Single daily PDB with week-to-date, month-to-date, and trending done daily • %BLDNTPDB(RUNDAY=YES, RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MON=,TUE=,WED=, THU=, FRI=, SAT=, SUN=);

  23. A Few Words About MXG 16.04 and Later • Each dataset now has an up to 6 character mnemonic identifier. In the case of NTSMF data it starts with NT and then has 4 unique characters. • This is used to identify what datasets to KEEP DROP or set to ZEROOBS. • YOU MUST REVIEW THE 16.04 CHANGES and DOCUMENTATION!

  24. Examples of Use • Build and keep only the datasets needed for NTINTRV • %BLDNTPDB(KEEPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

  25. Examples of Use • Build only the datasets needed for NTINTRV and DROP after NTINTRV is created: • %BLDNTPDB(DROPPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

  26. Examples of Use • Build the datasets needed for NTINTRV but build the PROCESS dataset with zero observations: • %BLDNTPDB(KEEPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROR NTRDIR NTSERV NTSYST NTTCP, ZEROOBS=NTPROC);

  27. Examples of Use • Reporting can be added to any execution: • %BLDNTPDB(xxxxxxxxxxxxxxx, RPTDAY=YES);

  28. Examples of Use • Reporting can be done against any PDB: • %BLDNTPDB(xxxxxxxxxxxxxxx, RPTDAY=MON);

  29. Examples of Use • GRAPHS can be created for daily, weekly and trend datasets (CPU usage only at this time - more to come.) • %BLDNTPDB(xxxxxxxxxxxxx, GRAFDAY=YES, GRAFWEEK=YES);

  30. Examples of Use - A Case Study Suppose you want the following: Run the PDB DAILY keeping all datasets Maximum of 5 logical drives/physical drive Start the week on Sunday Keep a WTD/MTD/and TREND datasets Run reports of Daily usage Create Graphs of Daily usage

  31. Examples of Use- A Case Study %BLDNTPDB(MON=, TUE=,WED=,THU=,FRI=, SAT=,SUN=,WEEK2=,WEEK3=, WEEK4=,WEEK5=, RUNDAY=YES,WEEKSTRT=SUN, RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MAXDRVS=5, RPTDAY=PDB,GRAFDAY=PDB);

  32. Future Directions • Largely up to YOU! • My knowledge of the data is limited • My experience with the data is limited • I need to know what YOU want/need

  33. Future Directions • What reports are needed • Data sources • Data elements • Levels of summarization

  34. Future Directions • What graphs are needed • What data sources • What kinds of graphs • What levels of summarization

  35. Future Directions • Currently on the development list: • Selection of datasets to place in the WEEKLY/MONTHLY databases - a WEEKKEEP= operand or something along those lines • More automated summarization (this is a tough one because it supposes knowledge of the datasets and variables that is difficult to have programmatically.)

  36. Future Directions • Currently on the Development List • Ability to limit variables in WTD/MTD datasets using DROP= • Ability to output reports as HTML

  37. ???????????

More Related