2013-2015 Regional Energy Networks Impact Evaluation Research Plan

1. 2013-2015 Regional Energy Networks Impact Evaluation Research Plan July 20, 2016

2. Agenda • REN Program Impact Evaluation (Multifamily) • Program Overviews • Evaluation Overview, Activities, Calculations • REN Program Impact Evaluation (Single Family) • Program Overviews • Evaluation Overview, Methods and Samples • Schedule and Next Steps

3. How Did We Get Here? 7/15/2016: Draft plan released 7/20/2016: Research Plan Webinar 7/29/2016: Comment Period Closes 8/22/2016: Final Research Plan Released

4. Program Overview: REN Multifamily Whole Building Program

5. Program Overview – MF-WB Multifamily Whole Building Program (MF-WB) • Administered by BayRENand SoCalREN • Customized project scopes • Savings based on building simulations • Technical and financial assistance • Project incentives range: $3,750 - $594,000

6. Program Overview – MF-WB

7. Program Overview – MF-WB • Incented ~248 properties in 2013-2015 • >15,000 dwelling units • BayREN claimed most projects (94%) and electricity savings (80%)

8. Evaluation Overview: REN Multifamily Whole Building Program

9. Evaluation Overview – MF-WB Evaluation Research Questions • What are the gross energy and demand savings (therms, kWh, kW) achieved by the BayREN and SoCalREN programs? • What are the net energy and demand savings achieved by the programs? • How can the RENs improve their ex ante savings claims so that they align with ex post values? • How do the REN savings and costs compare to those claimed from the IOU MF-WB programs?

10. Evaluation Overview – MF-WB Evaluation Activities • Telephone Surveys & Onsite Recruitment • Onsite Visits • Consumption Analysis • Calibrated Simulation Models • Savings Calculations

11. Evaluation Overview – Data Sources • REN provided project details • CPUC claim savings database • Consumption (billing) data • REN EnergyPro Models • Participant Survey (Property Managers/Owners) • 2013-2014 impact assessment completes • 2015 completes

12. Evaluation Overview – MF-WB

13. Evaluation Activities: Telephone Survey and Onsite Recruitment

14. Evaluation Activities– Telephone Survey Goals: • Recruit for onsite visits • Supplement 2013-2014 survey results • NFR Estimation • Baseline Assessment Sample: • All 2013-2015 participants eligible for onsite recruitment • Survey conducted only with those not contacted in 2013-2014 effort

15. Evaluation Activities– Telephone Survey Survey Topics: • Confirmation/verification of installed measures • Anticipated actions in absence of program intervention • Importance of program education and incentives on the decision to install high efficiency equipment • Working status and estimated age of replaced units • Timing for building maintenance/upgrades Survey Instrument: • Rely on 2013-2014 instrument • Comparability, consistency, holistic 2013-2015 results, vetted batteries and analysis

16. Evaluation Activities: Onsite Visits

17. Evaluation Activities– Onsite Visits Goals: • Verify key measure & building characteristics • Measures and characteristics that contribute most to project savings • Use in simulation models • Collect meter numbers • Link to IOU consumption information for project Sample: • All 2013-2015 participants eligible • Target 22 completed site visits

18. Evaluation Activities: Consumption Analysis

19. Evaluation Activities– Consumption Goals: • Comprehensive pre-program energy use at participant property. • Use in simulation models • Determine if meter numbers fully capture / link to IOU consumption information Sample: • Completed onsite visits where meter numbers were obtained

20. Evaluation Activities– Consumption Process: • Link meter numbers to IOU consumption information • Assess completeness and reasonableness of data

21. Evaluation Activities: Calibrated Simulation Models

22. Evaluation Activities– Models Goal: • Ex post gross savings values • Based on administrator project models • Using EnergyPro • Calibrated to pre-program consumption • Incorporate key measure and building characteristics found onsite • Adjusted for early replacement and replace on burnout scenarios Sample: • Completed site visit participants • Meter numbers were obtained • Reasonable consumption information

23. Evaluation Activities: Savings Calculations

24. Evaluation Review

25. Evaluation Activities– Calculations Goals: • Gross Program Realization Rate • Ex Post Gross Program Savings • Ex Post Net Program Savings

26. Program Overview: REN Single Family Home Upgrade Program

27. Overview: Single Family • 2 Regional Energy Networks (REN) and 4 Investor Owned Utilities (IOU) • Home Upgrade – deemed savings, deemed net to gross ratio (RENs) • Advanced Home Upgrade – modelled savings, estimated net to gross ratio (IOUs)

28. Overview: Prior Study Recap • Prior impact evaluation • Focused Impact Evaluation (PY 2013-2014) estimated gross savings only for Home Upgrade • Whole House Retrofit (PY 2010-2012) estimated savings gross and net savings for Advanced Home Upgrade • Outcome • Ex post saving estimates were less than ex ante but had wide ranges. • Issues with data quality and quantity limited the application of findings This research will estimate gross and net savings, NTG ratio of free riders, realization rate for Home Upgrade and Advanced Home Upgrade using projects from 2013, 2014 and 2015

29. Methods: Energy Savings • Billing analysis • Billing analysis using meter data and pooled model • Comparison group consists of future program participants • Variable duration for blackout period (min 30 days) • Consideration of interim activities and behavioral changes • Hourly data for kW • Daily data for kWh • Monthly data for therms • Expect population to be around 18,000 homes statewide Approach is similar to last impact evaluation to facilitate comparisons. Refinements will be made from lessons learned and to be as consistent as possible with CalTRACK effort

30. Methods: Savings Drivers • NTG and Exploratory Research • To date, the less than expected savings outcomes from HUP have not been fully explained • Combine data and findings from past and current studies • Conduct participant surveys • Topic examples: • Contractor effect / program messaging? • Geographic or demographic patterns? • Proportions and types of measures installed? • Primary motivations or significant barriers to adoption of one path or the other? • Other actions taken by participants? Primary data collection and mine existing data to uncover new insights

31. Single Family Schedule • Data Collection • Start now (July 2016) • Project data collection • Prior studies • Survey informed by billing data and prior studies • Interval billing data collection (Sep-Oct 2016) • Weather data • Analysis • Over the summer (Q3 and Q4 2016) • Reporting • Q1 2017 Interim results can be released as available

32. Schedule and Next Steps

33. Contact Information Apex Contact: Katie Parkinson, katiep@apexanalyticsllc.com (303) 590-9888 CPUC Contact: Jeremy Battis, jeremy.battis@cpuc.ca.gov (415) 703-3041 DNV GL Contact : Jon Vencil, jon.vencil@dnvgl.com (619) 929-3232