Home
South Pole Logbook

Search below for 'logbook_sop' for help on usage.

Sections

Search

Archives

November 2009
Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30          

RSS Feed

Powered by Blosxom


Root :: Ice Cube :: Runs :: { RCAs }

May 31, 2008

Run Failure Analysis: run 111047 - stringHub 69...

Over four hours into run 111047, the run watchdog decided stringHub#69 was stagnant and killed the run. Coincident with that DOMHubMonitor reported that one pair on sps-ichub69 dropped. 

DOMHubMonitor reports ic40 abnormal configuration on Fri May 23 04:05:06 2008


>>> sps-ichub69 : Card 2 pair 2 pwr check: plugged(ok)
>>> current(ERR_CURRENT_BELOW_LIMITS,ok) voltage(ok,ok)^G^G^G
From dash.log: 
DAQRun [2008-05-22 23:19:58.047628] Configuring run set...
DAQRun [2008-05-22 23:20:42.839577] Started run 111047 on run set 1
DAQRun [2008-05-22 23:20:43.288779]     0 physics events (0.00 Hz), 29016 moni events, 1871 SN events, 5 tcals
DAQRun [2008-05-22 23:21:13.516273]     46283 physics events (1412.83 Hz), 392307 moni events, 78116 SN events, 6576 tcals
DAQRun [2008-05-22 23:21:43.694565]     87808 physics events (1395.16 Hz), 506086 moni events, 151576 SN events, 82280 tcals
DAQRun [2008-05-22 23:22:13.896106]     127228 physics events (1366.00 Hz), 622234 moni events, 226567 SN events, 155438 tcals
.
.
.
DAQRun [2008-05-23 03:54:13.493728]     21679066 physics events (1319.25 Hz), 63000145 moni events, 40315776 SN events, 40267581 tcals
DAQRun [2008-05-23 03:54:43.727815]     21712201 physics events (1296.20 Hz), 63080971 moni events, 40368828 SN events, 40343541 tcals
DAQRun [2008-05-23 03:54:50.230899] ** Run watchdog reports stagnant components:
    secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 40368828
    secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 63080971
and starving components:
    dom->stringHub#69 sender.NumHitsReceived not changing from 10309695
    globalTrigger->eventBuilder backEnd.NumTriggerRequestsReceived not changing from 21712315
    stringHub->eventBuilder backEnd.NumReadoutsReceived not changing from 70475182
DAQRun [2008-05-23 03:55:00.254669] ** Run watchdog reports stagnant components:
    secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 40368828
    secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 63080971
and starving components:
    dom->stringHub#69 sender.NumHitsReceived not changing from 10309695
    globalTrigger->eventBuilder backEnd.NumTriggerRequestsReceived not changing from 21712315
    stringHub->eventBuilder backEnd.NumReadoutsReceived not changing from 70475182
DAQRun [2008-05-23 03:55:10.202407] #38: stringHub#69 inputs: Exception("dom->stringHub#69 sender.NumHitsReceived is not changing") in check() (RunWatchdog.py:80) <- checkValues() (RunWatchdog.py:206) <- checkList() (RunWatchdog.py:192) <- checkComp() (RunWatchdog.py:401)
DAQRun [2008-05-23 03:55:10.259228] #2: eventBuilder inputs: Exception("globalTrigger->eventBuilder backEnd.NumTriggerRequestsReceived is not changing") in check() (RunWatchdog.py:80) <- checkValues() (RunWatchdog.py:217) <- checkList() (RunWatchdog.py:192) <- checkComp() (RunWatchdog.py:401)
DAQRun [2008-05-23 03:55:10.264138] #1: secondaryBuilders outputs: Exception("secondaryBuilders->dispatch snBuilder.TotalDispatchedData is not changing") in check() (RunWatchdog.py:80) <- checkValues() (RunWatchdog.py:206) <- checkList() (RunWatchdog.py:192) <- checkComp() (RunWatchdog.py:410)
DAQRun [2008-05-23 03:55:10.266275] ** Run watchdog reports stagnant components:
    eventBuilder->dispatch backEnd.NumEventsSent not changing from 21712201
DAQRun [2008-05-23 03:55:10.492880] Caught error in system, going to ERROR state...
And from stringHub-69.log 
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.583] DOM Message exception: sent type = GET_DATA wrapped exception is java.nio.channels.ClosedByInterruptException
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.584]    at icecube.daq.domapp.DOMApp.sendMessage(DOMApp.java:382)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.584]    at icecube.daq.domapp.DOMApp.sendMessage(DOMApp.java:329)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.584]    at icecube.daq.domapp.DOMApp.getData(DOMApp.java:123)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.584]    at icecube.daq.domapp.DataCollector.runcore(DataCollector.java:844)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.584]    at icecube.daq.domapp.DataCollector.run(DataCollector.java:662)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.585] Caused by: java.nio.channels.ClosedByInterruptException
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.585]    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:212)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.586]    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:177)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.586]    at icecube.daq.domapp.DOMIO.recv(DOMIO.java:72)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.586]    at icecube.daq.domapp.DOMApp.sendMessage(DOMApp.java:370)
stringHub STDERR-22B ERROR [2008-05-23 03:55:13.587]    ... 4 more
stringHub icecube.daq.domapp.DataCollector-22B ERROR [2008-05-23 03:55:13.597] Intercepted error in DataCollector runcore: DOM Message exception: sent type = GET_DATA wrapped exception is java.nio.channels.ClosedByInterruptException
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.773] DOM Message exception: sent type = GET_DATA wrapped exception is java.nio.channels.ClosedByInterruptException
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.774]    at icecube.daq.domapp.DOMApp.sendMessage(DOMApp.java:382)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.774]    at icecube.daq.domapp.DOMApp.sendMessage(DOMApp.java:329)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.775]    at icecube.daq.domapp.DOMApp.getData(DOMApp.java:123)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.775]    at icecube.daq.domapp.DataCollector.runcore(DataCollector.java:844)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.776]    at icecube.daq.domapp.DataCollector.run(DataCollector.java:662)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.776] Caused by: java.nio.channels.ClosedByInterruptException
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.776]    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:212)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.776]    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:177)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.777]    at icecube.daq.domapp.DOMIO.recv(DOMIO.java:72)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.777]    at icecube.daq.domapp.DOMApp.sendMessage(DOMApp.java:370)
stringHub STDERR-22A ERROR [2008-05-23 03:55:13.777]    ... 4 more
stringHub icecube.daq.domapp.DataCollector-22A ERROR [2008-05-23 03:55:13.777] Intercepted error in DataCollector runcore: DOM Message exception: sent type = GET_DATA wrapped exception is java.nio.channels.ClosedByInterruptException

Ethan Dicks | 31 May 2008 00:26 GMT | Ice Cube/Runs | | permalink