1 / 21

An On/Off Link Activation Method for Low-Power Ethernet in PC Clusters

An On/Off Link Activation Method for Low-Power Ethernet in PC Clusters. Michihiro Koibuchi(NII, Japan ） Tomohiro Otsuka(Keio U, Japan ） Hiroki Matsutani （U of Tokyo, Japan ） Hideharu Amano（Keio U/ NII, Japan）. GbE. 56%. Interconnects share@TOP500 (Nov 2008 ）. Gigabit Ethernet.

hallie
Download Presentation

An On/Off Link Activation Method for Low-Power Ethernet in PC Clusters

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An On/Off Link Activation Method for Low-Power Ethernet in PC Clusters Michihiro Koibuchi(NII, Japan） Tomohiro Otsuka(Keio U, Japan） Hiroki Matsutani （Uof Tokyo, Japan） Hideharu Amano（Keio U/ NII, Japan）

  2. GbE 56% Interconnects share@TOP500 (Nov 2008） Gigabit Ethernet HPC PC Clusterswith Ethernet • Host/CPU • Various low-power techniques are used • DVFS • Power Gating • Ethernet Switch • Always preparing (active) for packet injection PC Ethernet switch We propose, and evaluate a low-power technique of Ethernet switches for PC clusters

  3. Ethernet for HPC Link aggregation (channel group) + multi-paths On/Off link activation method Evaluations Overhead of On/Off link operation Performance and power consumption of PC clusters Outline

  4. Ethernet on HPC systems 　11 7 8 9 10 4 5 6 0 1 2 3 14 15 13 12 TREE 1 TREE 2 TREE 3 TREE 4 4paths • Increasing the number of ports of GbE switches - 24/48-port switches provide the lowest cost per port • Improving the computation power of host（ > 10GFlops) • Link aggregation [IEEE 802.3ad]　＋　multi-path topology[Kudoh, IEEE Cluster, 2004][Viking, Infocom2004] - drastically increasing the number of links Link aggr. using 3 links switch host

  5. Power cons is almost constant regardless of traffic load # of activated ports dominates the power cons of switches Power cons of port is reduced down to ZERO by port-shutdown operation Power cons of GbE switches Unit：Ｗ

  6. Overview of the on/off link method TREE 1 TREE 2 TREE 3 TREE 4 　11 　11 7 7 8 8 9 9 10 10 4 4 5 5 6 6 0 0 1 1 2 2 3 3 14 14 15 15 13 13 12 12 Switch ports consume 40-60% of the total power Network load is not always high (e.g. during computation time switch node Traffic load becomes low （turning off a part of links) TREE 1 TREE 2 TREE 3 TREE 4

  7. Ethernet for HPC Link aggregation (channel group) + multi-paths On/Off link activation method Evaluations Overhead of On/Off link operation Performance and power consumption of PC clusters Outline

  8. A framework of on/off link method Traffic monitoring No Low or high-load links appear TREE 1 TREE 2 TREE 3 TREE 4 　11 7 8 9 10 4 5 6 0 1 2 3 14 15 13 12 Yes Selection of on/off links and paths Update of on/off link operation Very crucial factor Eg： port monitor, IPTraf, pilot execution Low traffic load is detected Paths: Before & After the before path is deactivated How is it implemented on Ethernet?

  9. Requirements for the on/off link method 　11 7 8 9 10 4 5 6 0 1 2 3 14 15 13 12 After To achieve a practical on/off link activation method, • No update of the MPI communication library • Using existing functions of commercial switches • Hiding the overhead to activate the link • Stabilizing the MAC address tables during updating paths - Avoiding broadcast storms, and communication interruption TREE 1 TREE 2 TREE 3 TREE 4 Before Switch Host

  10. Changing the paths for on/off link op Using switch-tagged・VLAN routing method[Otsuka,ICPP06] Specifying the path by attaching the VLAN tag to a frame （Port VLAN ID: PVID) Each host sends and receives usual (untagged) frames When an frame arrives at a switch from a host, add a VLAN tag (PVID) to it When it leaves to a host, removes the VLAN tag PVID v0 1　 4 5 6 7 0 1 2 3 The path of PVID#v1 The path of PVID#v0 VLAN v0 VLAN v1 VLAN tag ＃v0is attached 4 5 6 7 0 1 2 3

  11. When a deactivated link is activated (1) Activating the target link Using no-shutdown command of switch (2) Create VLAN v0 for the new path set that includes the target link, and make its MAC address table (3) Update the PVIDs of the ports for connecting hosts to v0 4 5 6 7 0 1 2 3 Link On, VLAN v0 Step 1,2 PVID v0 Step 3 5 6 7 4 0 1 2 3 When the traffic increases 4 7 5 6 0 1 2 3 Before Updating PVID to v0

  12. When an activated link is deactivated (1) Create VLAN v1 for the new path set that avoids the target link, and make its MAC address table (2) Update the PVID of the ports for connecting hosts to v1 (3) Deactivating the link PVID #v0 v1 Step 3 4 7 5 6 0 1 2 3 The path of PVID v0 The path of PVID v1 Decreasing the traffic 4 5 6 7 5 6 7 0 1 2 3 4 0 1 2 3 Before Step 1,2 Deactivating

  13. Ethernet for HPC Link aggregation (channel group) + multi-paths On/Off link activation method Evaluations Overhead of On/Off link operation On/off link operation Overhead to modify the path set Performance and power consumption of PC clusters Outline Dell 5324, 6224(24 ports), 6248(48 ports), Netgear SF-G0420(24 ports) We can buy them at $1,000-3,000

  14. When enabling STP, the overhead becomes some dozens～1 min To hide this overhead, paths should be updated after completing the on/off operation Fund. eval：On/Off overhead a link is continuously operated:　 on off on 　　

  15. Measure the overhead to change paths using VLANs Communication is not interrupted!! Enabling the runtime on/off link activation Fund. eval(2)：overhead to update paths Before VLAN　v0 After VLAN　v1 Update PVID to v1

  16. Performance evaluation on a PC cluster PC Cluster 128 hosts, Dual Opteron 1.8GHz x2 MPICH 1.2.7p1 GbE switch Dell Power Connect6248 28host per switch 48port@8 Application NPB 3.2

  17. Topology of the cluster Peak: 4×2 torus, 6 links between switches Enabling the link aggregation (IEEE 803.ad) Pre-executing the applications for estimating traffic amount Set up the on/off link set before executing Two on/off link selection algorithms Conservative: maintain the maximum amount of traffic on a link Aggressive: further power reduction（details are the proceeding） Torus

  18. Results of NPB(64 procs, PC6248 SW） 26% of power reduction 26% of NW power cons is reduced w/o performance degradation peak(all links) conservative aggressive 1.1 1 0.9 Relative Power Cons(W) 0.8 0.7 0.6 EP IS LU SP Fig 2：Power Cons of NWs,PC6248s Fig 1：Performance The conservative policy maintained almost the peak performance

  19. Results of NPB(64 procs, other SWs） 37% of power reduction peak(all links) conservative aggressive peak(all links) conservative aggressive 1.1 1.1 1 1 0.9 0.9 Relative Power Cons(W) Relative Power Cons(W) 0.8 0.8 0.7 0.7 0.6 0.6 EP IS LU SP EP IS LU SP Fig 3：Power Cons,SF-420s Fig 4：Power Cons,PC5324 A small number of services in L2 switch（PC5324) is always running compared with that of L3 switch （PC6248) The L2 switches reduces the larger ratio of power cons

  20. On/Off interconnection networks Cannot be directly applied to Ethernet M.Alonso[IPDPS05],V.Soteriou[TPDS07] Our on/off link method enables to support some of them in Ethernet DVFS for interconnection networks L.Shang[HPCA03], J.M.Stine[CAL04] Using multi-speed Ethernet (10M/100M/GbE/10GE) is similar to the approach for DVFS Dell switch:PC6248, 10M: 1.1W 100M: 1.3W GbE: 2.1W Related Work

  21. We propose the on/off link method on Ethernet Using port-shutdown command for reducing power cons Switch ports consume up to 60% of power cons in GbE switch Stabilizing the update of the MAC address table Evaluations on the PC cluster with GbE switches No overhead to update paths Reducing down to up to 37% of NW power cons We will provide the total solution of Ethernet for Low-Power PC clusters Link aggre. + multi-path topology + on/off links Conclusions

More Related