Search below for 'logbook_sop' for help on usage.
Sections
- AMANDA - 417
- Computing - 183
- Hardware - 103
- Misc - 2
- Runs - 57
- Supernova - 39
- TWR - 7
- Temperature - 5
- Training - 6
- IceCube - 1932
- AURA - 14
- Commissioning - 81
- DAQ - 15
- DOMs - 66
- Deployment - 12
- HoseReel - 5
- ICL - 179
- IceTop - 114
- Inclinometer - 9
- Infrastructure - 128
- Misc - 220
- PBS - 2
- PNF - 17
- Power - 15
- Runs - 718
- RCAs - 297
- SPATS - 3
- SPOTS - 14
- SPS - 285
- StandardCandle - 2
- Testing - 8
- Training - 5
- MAPO - 124
- Misc - 41
- PowerOutages - 1
- SPADE - 1140
- SPASE - 6
Search
Archives
| November 2009 | ||||||
|---|---|---|---|---|---|---|
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| 8 | 9 | 10 | 11 | 12 | 13 | 14 |
| 15 | 16 | 17 | 18 | 19 | 20 | 21 |
| 22 | 23 | 24 | 25 | 26 | 27 | 28 |
| 29 | 30 | |||||
- November 2009 (61)
- October 2009 (52)
- September 2009 (59)
- August 2009 (57)
- July 2009 (60)
- June 2009 (61)
- May 2009 (66)
- April 2009 (52)
- March 2009 (81)
- February 2009 (30)
- January 2009 (85)
- December 2008 (118)
- November 2008 (136)
- October 2008 (143)
- September 2008 (116)
- August 2008 (219)
- July 2008 (228)
- June 2008 (269)
- May 2008 (236)
- April 2008 (160)
- March 2008 (145)
- February 2008 (141)
- January 2008 (131)
- December 2007 (104)
- November 2007 (103)
- October 2007 (8)
- September 2007 (9)
- August 2007 (19)
- July 2007 (10)
- June 2007 (23)
- May 2007 (15)
- April 2007 (34)
- March 2007 (83)
- February 2007 (17)
- January 2007 (3)
- December 2006 (14)
- November 2006 (42)
- October 2006 (40)
- September 2006 (37)
- August 2006 (30)
- July 2006 (57)
- June 2006 (27)
- May 2006 (40)
- April 2006 (50)
- March 2006 (56)
- February 2006 (38)
- January 2006 (93)
- December 2005 (4)
May 07, 2009
about power outage
Author: Erik
Mood: Excellent !
So as promised I did a little investigation on what happened yesterday.
Unfortunately this leaded to a switch-off of the whole rack 9 again, so
run #113667 crashed and #113668 is a failed attempt to restart. In the
meantime I rebooted the hubs on rack 9, pUp'ed the DOMs, recovered
firmware fuses and restarted a run, #113669.
Sorry about that.
BUT here is what I found out (my look turns towards IT) :
The facility engineer ensured me this morning that the power delivery
did not glitch of failed yesterday evening (1:09pm UTC). What Mr.
Powerplant saw was the current drop produced by a whole rack that shuts
down, and slowly ramps up again till all the hubs are back online. Quite
neat that their monitoring system is able to see this.
So apparently the problem is something local to the ICL.
Here is what I saw at the ICL when I arrived this morning. UPS of rack 9
is off (UPS battery circuit breaker tripped). No LEDs on the front
bezel. This means there is direct power to the output but as soon as
there is a power glitch, we'll loose rack 9 again.
I tried to switch it back ON and that is when the whole rack 9 (hubs)
powered off, 30 minutes ago. Fortunately the GPS unit is not in this
rack...
This weird state of the UPS is not described in the manual. But while it
was briefly ON, 2 LEDs were blinking, the general alarm (red) and the
utility LED (green) and no power on the output (hubs were off). This is
described in the manual as "REPO condition". It is not written how to
solve it. I tried pushing 10 seconds on each front button (like the
on/off etc.), but nothing happened.
I manually tripped the breaker again to restore the previously faulty
(but working) state and to restore data taking as quick as possible.
What does IT think off this UPS unit ? is it completely unusable ? if it
is, I think replacing it with the unit in Rack 4 once AMANDA is switched
off, as we agreed before, would solve the problem. This is in a few days
anyway.
Another matter, is that yesterday's problem switched off sps-itfreeze. I
don't know how that is possible but it happened. This is the only
computer in rack 13, so I switched it on again.
Erik.
Erik Verhagen
|
07 May 2009 19:19 GMT
|
Ice Cube/Power
|
|
