Search below for 'logbook_sop' for help on usage.
Sections
- AMANDA - 417
- Computing - 183
- Hardware - 103
- Misc - 2
- Runs - 57
- Supernova - 39
- TWR - 7
- Temperature - 5
- Training - 6
- IceCube - 1932
- AURA - 14
- Commissioning - 81
- DAQ - 15
- DOMs - 66
- Deployment - 12
- HoseReel - 5
- ICL - 179
- IceTop - 114
- Inclinometer - 9
- Infrastructure - 128
- Misc - 220
- PBS - 2
- PNF - 17
- Power - 15
- Runs - 718
- RCAs - 297
- SPATS - 3
- SPOTS - 14
- SPS - 285
- StandardCandle - 2
- Testing - 8
- Training - 5
- MAPO - 124
- Misc - 41
- PowerOutages - 1
- SPADE - 1140
- SPASE - 6
Search
Archives
| November 2009 | ||||||
|---|---|---|---|---|---|---|
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| 8 | 9 | 10 | 11 | 12 | 13 | 14 |
| 15 | 16 | 17 | 18 | 19 | 20 | 21 |
| 22 | 23 | 24 | 25 | 26 | 27 | 28 |
| 29 | 30 | |||||
- November 2009 (61)
- October 2009 (52)
- September 2009 (59)
- August 2009 (57)
- July 2009 (60)
- June 2009 (61)
- May 2009 (66)
- April 2009 (52)
- March 2009 (81)
- February 2009 (30)
- January 2009 (85)
- December 2008 (118)
- November 2008 (136)
- October 2008 (143)
- September 2008 (116)
- August 2008 (219)
- July 2008 (228)
- June 2008 (269)
- May 2008 (236)
- April 2008 (160)
- March 2008 (145)
- February 2008 (141)
- January 2008 (131)
- December 2007 (104)
- November 2007 (103)
- October 2007 (8)
- September 2007 (9)
- August 2007 (19)
- July 2007 (10)
- June 2007 (23)
- May 2007 (15)
- April 2007 (34)
- March 2007 (83)
- February 2007 (17)
- January 2007 (3)
- December 2006 (14)
- November 2006 (42)
- October 2006 (40)
- September 2006 (37)
- August 2006 (30)
- July 2006 (57)
- June 2006 (27)
- May 2006 (40)
- April 2006 (50)
- March 2006 (56)
- February 2006 (38)
- January 2006 (93)
- December 2005 (4)
May 18, 2009
Power outage
Author: Erik
Mood: Excellent !
So here is the situation: Almost the whole rack 13 was OFF. Just OFF. The concerned machines were: * sps-fpslave03 * sps-fpslave04 * sps-fpslave05 * sps-fpslave06 * sps-fpslave07 * sps-fpslave08 * sps-fpslave09 * sps-fpslave10 * sps-fpmaster * sps-cvmaster Only sps-fpslave01, sps-build32, and all the blades were ON when I arrived. In addition, sps-itfreeze in rack 13 was OFF too. It is not the first time that this one powers OFF when a glitch appears somewhere else in the ICL. All machines are ON again, they are booting. Since these machines were powered OFF in a rather brutal way, they are now performing checkdisks, which takes a long time. I can't access sps-fpslave 05 and sps-fpslave06 with the KVM console, but they are up and running. Here is what most likely happened. Both UPSes in rack 16 are dead. A red light is ON on the upper one, and a red led is blinking on the lower one. This a pretty bad news, because the only spare UPS we had was moved to rack 9 last week to replace another failed one. According to the manual, the batteries are dead, and this is confirmed by the HP Power Management software. In the logs, on May 18 appeared (chronologically): * 15:10:27 The UPS is in a low battery condition * 15:10:27 The UPS is on battery * 15:10:34 The UPS battery has failed * 15:10:34 The UPS low battery condition has cleared * 15:10:34 Utility power has been restored * 15:12:46 A UPS battery has failed * 15:20:14 A UPS battery has failed * 15:20:16 Connection with the remote agent was lost (probably when the server was powered off) On another UPS I can see that there has been a little power cut for a few seconds. The good news is that rack 9 is still running. I'll talk to the power plant manager tomorrow morning, to confirm there was a problem.
Erik.
Erik Verhagen
|
18 May 2009 11:56 GMT
|
Ice Cube/Power
|
|
