Search below for 'logbook_sop' for help on usage.
Sections
- AMANDA - 417
- Computing - 183
- Hardware - 103
- Misc - 2
- Runs - 57
- Supernova - 39
- TWR - 7
- Temperature - 5
- Training - 6
- IceCube - 1932
- AURA - 14
- Commissioning - 81
- DAQ - 15
- DOMs - 66
- Deployment - 12
- HoseReel - 5
- ICL - 179
- IceTop - 114
- Inclinometer - 9
- Infrastructure - 128
- Misc - 220
- PBS - 2
- PNF - 17
- Power - 15
- Runs - 718
- RCAs - 297
- SPATS - 3
- SPOTS - 14
- SPS - 285
- StandardCandle - 2
- Testing - 8
- Training - 5
- MAPO - 124
- Misc - 41
- PowerOutages - 1
- SPADE - 1140
- SPASE - 6
Search
Archives
| November 2009 | ||||||
|---|---|---|---|---|---|---|
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| 8 | 9 | 10 | 11 | 12 | 13 | 14 |
| 15 | 16 | 17 | 18 | 19 | 20 | 21 |
| 22 | 23 | 24 | 25 | 26 | 27 | 28 |
| 29 | 30 | |||||
- November 2009 (61)
- October 2009 (52)
- September 2009 (59)
- August 2009 (57)
- July 2009 (60)
- June 2009 (61)
- May 2009 (66)
- April 2009 (52)
- March 2009 (81)
- February 2009 (30)
- January 2009 (85)
- December 2008 (118)
- November 2008 (136)
- October 2008 (143)
- September 2008 (116)
- August 2008 (219)
- July 2008 (228)
- June 2008 (269)
- May 2008 (236)
- April 2008 (160)
- March 2008 (145)
- February 2008 (141)
- January 2008 (131)
- December 2007 (104)
- November 2007 (103)
- October 2007 (8)
- September 2007 (9)
- August 2007 (19)
- July 2007 (10)
- June 2007 (23)
- May 2007 (15)
- April 2007 (34)
- March 2007 (83)
- February 2007 (17)
- January 2007 (3)
- December 2006 (14)
- November 2006 (42)
- October 2006 (40)
- September 2006 (37)
- August 2006 (30)
- July 2006 (57)
- June 2006 (27)
- May 2006 (40)
- April 2006 (50)
- March 2006 (56)
- February 2006 (38)
- January 2006 (93)
- December 2005 (4)
Feb 24, 2009
RCA for run 112983: TotalDispatchedData not changing
By: Erik Time: 20090224 0818Z 0818Z logbook entry started, dash.log shows:DAQRun [2009-02-24 01:55:15.193359] Configuring run set... DAQRun [2009-02-24 01:55:59.365794] Creating moni output file /mnt/data/pdaq/log/daqrun112983/inIceTrigger-0.moni (remote is 10.1.2.64:35587) DAQRun [2009-02-24 01:55:59.382647] Creating moni output file /mnt/data/pdaq/log/daqrun112983/iceTopTrigger-0.moni (remote is 10.1.2.64:35583) DAQRun [2009-02-24 01:55:59.397839] Creating moni output file /mnt/data/pdaq/log/daqrun112983/globalTrigger-0.moni (remote is 10.1.2.64:35595) DAQRun [2009-02-24 01:55:59.413012] Creating moni output file /mnt/data/pdaq/log/daqrun112983/amandaTrigger-0.moni (remote is 10.1.2.64:35601) DAQRun [2009-02-24 01:55:59.428948] Creating moni output file /mnt/data/pdaq/log/daqrun112983/eventBuilder-0.moni (remote is 10.1.2.15:34511) DAQRun [2009-02-24 01:55:59.450521] Creating moni output file /mnt/data/pdaq/log/daqrun112983/secondaryBuilders-0.moni (remote is 10.1.2.207:34644) DAQRun [2009-02-24 01:55:59.478970] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-0.moni (remote is 10.1.2.201:47242) DAQRun [2009-02-24 01:55:59.551748] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-21.moni (remote is 10.1.3.121:53705) DAQRun [2009-02-24 01:55:59.853145] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-202.moni (remote is 10.1.3.192:41069) DAQRun [2009-02-24 01:56:00.037060] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-201.moni (remote is 10.1.3.191:41641) DAQRun [2009-02-24 01:56:00.236514] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-29.moni (remote is 10.1.3.129:34315) DAQRun [2009-02-24 01:56:00.535784] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-30.moni (remote is 10.1.3.130:34904) DAQRun [2009-02-24 01:56:00.842981] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-38.moni (remote is 10.1.3.138:33588) DAQRun [2009-02-24 01:56:01.177439] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-39.moni (remote is 10.1.3.139:33534) DAQRun [2009-02-24 01:56:01.471185] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-40.moni (remote is 10.1.3.140:33789) DAQRun [2009-02-24 01:56:01.788329] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-44.moni (remote is 10.1.3.144:33536) DAQRun [2009-02-24 01:56:02.063020] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-45.moni (remote is 10.1.3.145:55785) DAQRun [2009-02-24 01:56:02.346036] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-46.moni (remote is 10.1.3.146:53276) DAQRun [2009-02-24 01:56:02.620561] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-47.moni (remote is 10.1.3.147:33829) DAQRun [2009-02-24 01:56:02.896005] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-48.moni (remote is 10.1.3.148:33837) DAQRun [2009-02-24 01:56:03.213619] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-49.moni (remote is 10.1.3.149:33276) DAQRun [2009-02-24 01:56:03.521775] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-50.moni (remote is 10.1.3.150:33300) DAQRun [2009-02-24 01:56:03.824278] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-52.moni (remote is 10.1.3.152:33516) DAQRun [2009-02-24 01:56:04.138339] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-53.moni (remote is 10.1.3.153:40968) DAQRun [2009-02-24 01:56:04.432540] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-54.moni (remote is 10.1.3.154:33389) DAQRun [2009-02-24 01:56:04.730946] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-55.moni (remote is 10.1.3.155:52652) DAQRun [2009-02-24 01:56:05.033757] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-56.moni (remote is 10.1.3.156:53575) DAQRun [2009-02-24 01:56:05.316759] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-57.moni (remote is 10.1.3.157:33568) DAQRun [2009-02-24 01:56:05.611169] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-58.moni (remote is 10.1.3.158:41750) DAQRun [2009-02-24 01:56:05.905230] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-59.moni (remote is 10.1.3.159:50117) DAQRun [2009-02-24 01:56:06.185077] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-60.moni (remote is 10.1.3.160:50188) DAQRun [2009-02-24 01:56:06.490351] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-61.moni (remote is 10.1.3.161:49785) DAQRun [2009-02-24 01:56:06.788902] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-62.moni (remote is 10.1.3.162:50045) DAQRun [2009-02-24 01:56:07.085695] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-63.moni (remote is 10.1.3.163:40118) DAQRun [2009-02-24 01:56:07.379656] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-64.moni (remote is 10.1.3.164:49601) DAQRun [2009-02-24 01:56:07.677894] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-65.moni (remote is 10.1.3.165:54047) DAQRun [2009-02-24 01:56:07.986921] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-66.moni (remote is 10.1.3.166:40055) DAQRun [2009-02-24 01:56:08.260928] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-67.moni (remote is 10.1.3.167:49878) DAQRun [2009-02-24 01:56:08.559465] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-68.moni (remote is 10.1.3.168:49862) DAQRun [2009-02-24 01:56:08.853245] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-69.moni (remote is 10.1.3.169:57782) DAQRun [2009-02-24 01:56:09.319269] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-70.moni (remote is 10.1.3.170:51019) DAQRun [2009-02-24 01:56:09.617778] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-71.moni (remote is 10.1.3.171:49831) DAQRun [2009-02-24 01:56:09.907746] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-72.moni (remote is 10.1.3.172:55942) DAQRun [2009-02-24 01:56:10.184693] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-73.moni (remote is 10.1.3.173:53486) DAQRun [2009-02-24 01:56:10.486331] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-74.moni (remote is 10.1.3.174:35662) DAQRun [2009-02-24 01:56:10.772115] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-75.moni (remote is 10.1.3.175:35143) DAQRun [2009-02-24 01:56:11.063967] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-76.moni (remote is 10.1.3.176:50159) DAQRun [2009-02-24 01:56:11.350827] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-77.moni (remote is 10.1.3.177:56537) DAQRun [2009-02-24 01:56:11.628630] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-78.moni (remote is 10.1.3.178:53734) DAQRun [2009-02-24 01:56:11.913866] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-205.moni (remote is 10.1.3.195:41270) DAQRun [2009-02-24 01:56:12.100204] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-206.moni (remote is 10.1.3.196:40597) DAQRun [2009-02-24 01:56:12.274491] Creating moni output file /mnt/data/pdaq/log/daqrun112983/stringHub-203.moni (remote is 10.1.3.193:40740) DAQRun [2009-02-24 01:56:28.698308] Started run 112983 on run set 1 DAQRun [2009-02-24 01:58:39.943684] ** Run watchdog reports starving components: stringHub->inIceTrigger stringHit.RecordsReceived not changing from [91775, 88723, 89012, 91461, 87093, 93057, 84038, 97554, 92313, 96934, 100823, 93432, 90067, 99162, 100112, 90856, 91250, 101440, 99475, 97378, 94823, 97295, 92710, 95373, 93031, 101629, 98521, 93825, 102324, 101773, 84004, 97773, 98165, 93652, 97110, 96531, 97100, 95473, 93534, 101681] DAQRun [2009-02-24 02:00:44.389743] ** Run watchdog reports starving components: stringHub->inIceTrigger stringHit.RecordsReceived not changing from [178944, 173830, 172545, 179670, 168849, 182261, 163531, 189189, 182187, 187788, 195556, 183168, 179328, 194084, 194589, 177005, 178877, 198305, 193482, 191331, 183974, 190738, 181953, 186027, 180743, 198955, 192066, 182544, 198458, 198632, 164761, 190318, 192435, 181847, 189393, 189090, 190775, 187160, 181882, 196147] DAQRun [2009-02-24 07:53:22.190185] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 52265637 secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 65794755 DAQRun [2009-02-24 07:53:31.349769] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 52265637 secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 65794755 DAQRun [2009-02-24 07:53:41.360532] #1: secondaryBuilders outputs: Exception("secondaryBuilders->dispatch snBuilder.TotalDispatchedData is not changing") in check() (RunWatchdog.py:92) <- __checkValues() (RunWatchdog.py:145) <- checkList() (RunWatchdog.py:221) <- __checkComp() (RunWatchdog.py:413) DAQRun [2009-02-24 07:53:51.362170] #1: secondaryBuilders outputs: Exception("secondaryBuilders->dispatch moniBuilder.TotalDispatchedData is not changing") in check() (RunWatchdog.py:92) <- __checkValues() (RunWatchdog.py:145) <- checkList() (RunWatchdog.py:221) <- __checkComp() (RunWatchdog.py:413) DAQRun [2009-02-24 08:04:33.100442] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 67852252 DAQRun [2009-02-24 08:04:42.727608] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 67852252 DAQRun [2009-02-24 08:04:53.153211] #1: secondaryBuilders outputs: Exception("secondaryBuilders->dispatch moniBuilder.TotalDispatchedData is not changing") in check() (RunWatchdog.py:92) <- __checkValues() (RunWatchdog.py:145) <- checkList() (RunWatchdog.py:221) <- __checkComp() (RunWatchdog.py:413) DAQRun [2009-02-24 08:05:02.777184] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 53969271 secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 67852252 DAQRun [2009-02-24 08:05:12.723423] #1: secondaryBuilders outputs: Exception("secondaryBuilders->dispatch snBuilder.TotalDispatchedData is not changing") in check() (RunWatchdog.py:92) <- __checkValues() (RunWatchdog.py:145) <- checkList() (RunWatchdog.py:221) <- __checkComp() (RunWatchdog.py:413) DAQRun [2009-02-24 08:05:23.739190] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 53969271 secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 67852252 DAQRun [2009-02-24 08:05:32.911814] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 53969271 secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 67852252 DAQRun [2009-02-24 08:05:42.970280] ** Run watchdog reports stagnant components: secondaryBuilders->dispatch snBuilder.TotalDispatchedData not changing from 53969271 secondaryBuilders->dispatch moniBuilder.TotalDispatchedData not changing from 67852252 DAQRun [2009-02-24 08:05:43.089082] Caught error in system, going to ERROR state... DAQRun [2009-02-24 08:06:37.090177] Recovering from failed run 112983... DAQRun [2009-02-24 08:06:37.107948] Stopping component logging DAQRun [2009-02-24 08:06:49.766957] RPC Call stats: rpc_show_components: 11 entries, min=0.1557 max=0.2458, avg=0.2060, rms=0.0232 rpc_runset_subrun: 1 entries, min=0.8433 max=0.8433, avg=0.8433, rms=0.0000 rpc_runset_make: 11 entries, min=6.6610 max=6.7939, avg=6.7504, rms=0.0389 rpc_runset_list: 11 entries, min=0.1866 max=0.2432, avg=0.2239, rms=0.0146 rpc_runset_configure: 11 entries, min=43.9263 max=46.3537, avg=44.7019, rms=0.6699 rpc_runset_events: 186 entries, min=0.0039 max=3.0052, avg=0.0348, rms=0.2224 rpc_runset_start_run: 11 entries, min=0.8690 max=5.1924, avg=2.4044, rms=1.2829 rpc_log_to: 11 entries, min=0.0025 max=0.0043, avg=0.0030, rms=0.0006 rpc_runset_break: 10 entries, min=0.0024 max=0.0037, avg=0.0031, rms=0.0003 rpc_runset_bothlog_to: 11 entries, min=0.9418 max=1.1093, avg=1.0350, rms=0.0440 rpc_runset_stop_run: 10 entries, min=3.0506 max=4.7223, avg=3.7475, rms=0.4269 DAQRun [2009-02-24 08:06:49.767206] Run terminated WITH ERROR.Nothing obviously wrong in secondaryBuilders-0.log Hope this RCA is helpful. Erik.
Erik Verhagen | 24 Feb 2009 02:41 GMT | Ice Cube/Runs/RCAs | |
