Home
South Pole Logbook

Search below for 'logbook_sop' for help on usage.

Sections

Search

Archives

November 2009
Sun Mon Tue Wed Thu Fri Sat
         

RSS Feed

Powered by Blosxom


Sep 29, 2008

Physical DNF checks and HR readings done

See more ...


Edgar Nielsen | 29 Sep 2008 16:45 GMT | Ice Cube/Misc | | permalink

Stopped run 111704 to fixed dropped DOM 65-18 "Bid_Poker"

<--! more --> As reported by DOMHubMonitor, 65-18 "Bid_Poker" dropped.
20A 5018 Q_6 R1B0658D05 COMM                               000000000000           88 mA  86 mA  90V
... and was successfully revived with pCycle
20A 5018 Q_6 R1B0658D05 COMM 65-18  Bid_Poker              8ca33eaaaad0 TP6P1399  79 mA  77 mA  90V
Run 111704 was started manually...
pdaq@sps-expcont[~] 19:44:31 (0)$ ./starti3am
issuing anvil control daq; anvil control twr...
Sub-system 'daq' are now under control of anvil.
Sub-system 'twr' are now under control of anvil.


 the following commands will now be executed:

# /usr/local/pdaq/pDAQ_current/dash/DAQConfig.py /usr/local/pdaq/pDAQ_current/config/sps-IC40-icetop-calibration-trigger-V047.xml &

/usr/local/pdaq/offline-db-updateconfig /usr/local/pdaq/pDAQ_current/config/sps-IC40-icetop-calibration-trigger-V047.xml &

/usr/local/pdaq/anvil/anvil start -j MergeEvents -p PhysicsFiltering -d sps-IC40-icetop-calibration-trigger-V047 -r PhysicsTrig -n 40000 -l 28800 -v sam

Do you with to continue? (yes/no) yes
starting up pDAQ the correct way...

ERROR: DB interface says 'ErrAlreadyExists'
2008-09-29 19:46:20: Started 'sam' service with PID=19638
2008-09-29 19:46:20: Parameters for this run set:
2008-09-29 19:46:20:           Run Mode: PhysicsTrig
2008-09-29 19:46:20:          DAQ Label: sps-IC40-icetop-calibration-trigger-V047
2008-09-29 19:46:20:          JeB Label: MergeEvents (1)
2008-09-29 19:46:20:          PnF Label: PhysicsFiltering (1)
2008-09-29 19:46:20:         Run Length: 28800
2008-09-29 19:46:20:     Number of Runs: 40000
2008-09-29 19:46:20: Started 'run' service
2008-09-29 19:46:20: Parameters for this run:
2008-09-29 19:46:20:         Run Number: 111705
2008-09-29 19:46:20:           Run Mode: PhysicsTrig
2008-09-29 19:46:20:          DAQ Label: sps-IC40-icetop-calibration-trigger-V047
2008-09-29 19:46:20:          JeB Label: MergeEvents (1)
2008-09-29 19:46:20:          PnF Label: PhysicsFiltering (1)
2008-09-29 19:46:20: Starting run 111705
2008-09-29 19:46:32: Starting 'twr' sub-system
2008-09-29 19:46:32: Starting 'daq' sub-system
2008-09-29 19:47:31: Started run 111705
2008-09-29 19:49:32: Sub-system 'daq' is in 'Error' state, not 'Starting'
2008-09-29 19:49:32: Successive daq failure number 1
2008-09-29 19:49:32: Stopping run 111705
2008-09-29 19:49:37: Stopping 'twr' sub-system
2008-09-29 19:49:37: Recovering 'daq' sub-system
2008-09-29 19:49:52: Stopped run 111705
2008-09-29 19:49:54: Stopped 'run' service
... but quickly died due to what looks like a backend time-out...
DAQRun [2008-09-29 19:46:47.135731] Configuring run set...
DAQRun [2008-09-29 19:47:30.897120] Started run 111705 on run set 1
DAQRun [2008-09-29 19:49:31.111732] Exception in monitoring: timeout("timed out") in readline() (socket.py:323) <- _read_status() (httplib.py:231) <- begin() (httplib.py:273) <- getresponse() (httplib.py:778) <- getreply() (httplib.py:1049) <- request() (xmlrpclib.py:1064) <- __request() (xmlrpclib.py:1316) <- __call__() (xmlrpclib.py:1029) <- getSingleBeanField() (DAQMoni.py:119) <- getEventCounts() (DAQRun.py:679) <- updateEventCounts() (DAQRun.py:277) <- check_all() (DAQRun.py:715)
DAQRun [2008-09-29 19:49:31.111871] Caught error in system, going to ERROR state.
... and restarted on its own...
2008-09-29 19:49:54: Started 'run' service
2008-09-29 19:49:54: Parameters for this run:
2008-09-29 19:49:54:         Run Number: 111706
2008-09-29 19:49:54:           Run Mode: PhysicsTrig
2008-09-29 19:49:54:          DAQ Label: sps-IC40-icetop-calibration-trigger-V047
2008-09-29 19:49:54:          JeB Label: MergeEvents (1)
2008-09-29 19:49:54:          PnF Label: PhysicsFiltering (1)
2008-09-29 19:49:54: Starting run 111706
2008-09-29 19:50:06: Starting 'twr' sub-system
2008-09-29 19:50:06: Starting 'daq' sub-system
2008-09-29 19:51:07: Started run 111706
... and it finally looks like things are running well...
DAQRun [2008-09-29 19:50:22.836028] Configuring run set...
DAQRun [2008-09-29 19:51:06.854574] Started run 111706 on run set 1
DAQRun [2008-09-29 19:51:07.473907]     0 physics events (0.00 Hz), 15194 moni events, 1683 SN events, 5 tcals
DAQRun [2008-09-29 19:51:37.405710]     45154 physics events (1401.78 Hz), 393528 moni events, 75992 SN events, 6632 tcals
DAQRun [2008-09-29 19:52:07.565637]     84575 physics events (1355.98 Hz), 504810 moni events, 148873 SN events, 77484 tcals
DAQRun [2008-09-29 19:52:37.748094]     123714 physics events (1336.66 Hz), 622974 moni events, 224496 SN events, 153495 tcals

Ethan Dicks | 29 Sep 2008 14:53 GMT | Ice Cube/Runs | | permalink

Sep 28, 2008

Acopian PS checkout

See more ...


Edgar Nielsen | 28 Sep 2008 17:02 GMT | Ice Cube/Ice Top | | permalink

NCM for sps-fpslave13 RAM

See more ...


Edgar Nielsen | 28 Sep 2008 16:17 GMT | Ice Cube/Misc | | permalink

Work on sps-fpslave13

See more ...


Edgar Nielsen | 28 Sep 2008 16:15 GMT | Ice Cube/SPS | | permalink

Turning watchdog cronjob back on

See more ...


Edgar Nielsen | 28 Sep 2008 12:53 GMT | Ice Cube/Infrastructure | | permalink

Sep 27, 2008

Cleaning B2

See more ...


Edgar Nielsen | 27 Sep 2008 20:49 GMT | Misc | | permalink

Daily checks

See more ...


Edgar Nielsen | 27 Sep 2008 18:47 GMT | MAPO | | permalink

Daily checks

See more ...


Edgar Nielsen | 27 Sep 2008 18:43 GMT | Ice Cube/ICL | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 27 Sep 2008 18:25 GMT | Ice Cube/Misc | | permalink

Power cycle iridium modem

See more ...


Edgar Nielsen | 27 Sep 2008 16:52 GMT | Ice Cube/Infrastructure | | permalink

Sep 26, 2008

Visual DNF checks

See more ...


Edgar Nielsen | 26 Sep 2008 15:55 GMT | Ice Cube/Misc | | permalink

Sep 25, 2008

apache on sps-supercore hosed

See more ...


Edgar Nielsen | 25 Sep 2008 23:58 GMT | Ice Cube/SPS | | permalink

Daily checks

See more ...


Edgar Nielsen | 25 Sep 2008 23:39 GMT | MAPO | | permalink

RAM errors, sps-fpslave13.sps

See more ...


Edgar Nielsen | 25 Sep 2008 22:10 GMT | Ice Cube/SPS | | permalink

Daily checks

See more ...


Edgar Nielsen | 25 Sep 2008 22:04 GMT | Ice Cube/ICL | | permalink

Shutdown of ic-tape and ic-tape2

See more ...


Edgar Nielsen | 25 Sep 2008 20:01 GMT | Ice Cube/SPS | | permalink

Changing HSM tapes

See more ...


Edgar Nielsen | 25 Sep 2008 20:00 GMT | SPADE | | permalink

Sep 24, 2008

Daily checks

See more ...


Edgar Nielsen | 24 Sep 2008 18:34 GMT | Ice Cube/ICL | | permalink

Daily checks

See more ...


Edgar Nielsen | 24 Sep 2008 18:31 GMT | MAPO | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 24 Sep 2008 18:23 GMT | Ice Cube/Misc | | permalink

Post Mortem on Power Outage

See more ...


Edgar Nielsen | 24 Sep 2008 17:54 GMT | Ice Cube/Power | | permalink

Sep 23, 2008

sps-i3moni failed drive

See more ...


Edgar Nielsen | 23 Sep 2008 22:24 GMT | Ice Cube/SPS | | permalink

RCA for run 111683: Feta:dropped_dom

See more ...


Edgar Nielsen | 23 Sep 2008 14:44 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111684: Trichopathophobia:dropped_dom/Papyrophobia:ClosedByInterrupt

See more ...


Edgar Nielsen | 23 Sep 2008 14:44 GMT | Ice Cube/Runs/RCAs | | permalink

Daily checks

See more ...


Edgar Nielsen | 23 Sep 2008 14:16 GMT | Ice Cube/ICL | | permalink

Fixing Feta

See more ...


Edgar Nielsen | 23 Sep 2008 14:13 GMT | Ice Cube/DOMs | | permalink

RCA for run 111675: TWR/power_outage

See more ...


Edgar Nielsen | 23 Sep 2008 03:34 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111676: TWR/power_outage

See more ...


Edgar Nielsen | 23 Sep 2008 03:34 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111677: TWR/power_outage

See more ...


Edgar Nielsen | 23 Sep 2008 03:34 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111679: TWR/power_outage

See more ...


Edgar Nielsen | 23 Sep 2008 03:34 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111678: TWR/power_outage

See more ...


Edgar Nielsen | 23 Sep 2008 03:34 GMT | Ice Cube/Runs/RCAs | | permalink

Sep 22, 2008

Physical DNF checks

See more ...


Edgar Nielsen | 22 Sep 2008 23:50 GMT | Ice Cube/Misc | | permalink

2nd floor computer lab vent closed

See more ...


Edgar Nielsen | 22 Sep 2008 22:39 GMT | Ice Cube/ICL | | permalink

Power outage

See more ...


Edgar Nielsen | 22 Sep 2008 21:14 GMT | Ice Cube/Power | | permalink

Thermostat from about 31 to 45F

See more ...


Edgar Nielsen | 22 Sep 2008 19:52 GMT | MAPO | | permalink

Daily checks

See more ...


Edgar Nielsen | 22 Sep 2008 04:44 GMT | MAPO | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 22 Sep 2008 04:29 GMT | Ice Cube/Misc | | permalink

Sep 21, 2008

Daily checks

See more ...


Edgar Nielsen | 21 Sep 2008 22:29 GMT | Ice Cube/ICL | | permalink

Moving pool chemicals to the station

See more ...


Edgar Nielsen | 21 Sep 2008 22:29 GMT | Ice Cube/ICL | | permalink

USAP weekly safety report for the ICL

See more ...


Edgar Nielsen | 21 Sep 2008 22:28 GMT | Ice Cube/ICL | | permalink

Swept and mopped 2nd floor of the ICL

See more ...


Edgar Nielsen | 21 Sep 2008 22:27 GMT | Ice Cube/ICL | | permalink

Sep 20, 2008

Stopping run 111668 to cycle DOM 73-11 "Pandavas" and 63-41 "Feta"

See more ...


Ethan Dicks | 20 Sep 2008 16:36 GMT | Ice Cube/Runs | | permalink

Sep 18, 2008

Took down run 111658 to fixed dropped DOM 60-62 "Miletus" on sps-ithub06

See more ...


Ethan Dicks | 18 Sep 2008 22:59 GMT | Ice Cube/Runs | | permalink

Installing dataprobe

See more ...


Edgar Nielsen | 18 Sep 2008 18:26 GMT | Ice Cube/Infrastructure | | permalink

Tapeset 2008a re-mounted on ic-hsm

See more ...


Ethan Dicks | 18 Sep 2008 03:24 GMT | Ice Cube/SPS | | permalink

Sep 17, 2008

ic-hsm problems again

See more ...


Edgar Nielsen | 17 Sep 2008 17:41 GMT | Ice Cube/SPS | | permalink

Stopping run 111653 to cycle DOM 73-11 "Pandavas"

See more ...


Ethan Dicks | 17 Sep 2008 15:45 GMT | Ice Cube/Runs | | permalink

SPADE tape tape_2008_510_RAW_copy_1 full. Inserted new tape.


Ethan Dicks | 17 Sep 2008 09:46 GMT | SPADE | | permalink

SPADE tape tape_2008_509_RAW_copy_1 full. Inserted new tape.


Ethan Dicks | 17 Sep 2008 08:56 GMT | SPADE | | permalink

Sep 15, 2008

Stopping run 111646 to cycle DOM 50-64 "Maniaphobia" on sps-ithub03

See more ...


Ethan Dicks | 15 Sep 2008 21:29 GMT | Ice Cube/Runs | | permalink

Sep 14, 2008

Weekly snow removal

See more ...


Edgar Nielsen | 14 Sep 2008 20:57 GMT | Ice Cube/ICL | | permalink

Incline-o-mite!

See more ...


Edgar Nielsen | 14 Sep 2008 16:57 GMT | Ice Cube/Inclinometer | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 14 Sep 2008 15:51 GMT | Ice Cube/Misc | | permalink

Daily checks

See more ...


Edgar Nielsen | 14 Sep 2008 15:50 GMT | Ice Cube/ICL | | permalink

Daily checks

See more ...


Edgar Nielsen | 14 Sep 2008 15:50 GMT | MAPO | | permalink

RCA for run 111640: xmlrpc-timeout_DAQMoni.py

See more ...


Edgar Nielsen | 14 Sep 2008 13:09 GMT | Ice Cube/Runs/RCAs | | permalink

Sep 13, 2008

Removed 2nd floor window coverings

See more ...


Edgar Nielsen | 13 Sep 2008 23:02 GMT | Ice Cube/ICL | | permalink

Fixing Pandavas

See more ...


Edgar Nielsen | 13 Sep 2008 19:20 GMT | Ice Cube/Runs | | permalink

RCA for run 111636: Pandavas:dropped_dom

See more ...


Edgar Nielsen | 13 Sep 2008 19:10 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111631: Eric_Coplin:ClosedByInterrupt

See more ...


Edgar Nielsen | 13 Sep 2008 07:19 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111634: Eric_Coplin:ClosedByInterrupt

See more ...


Edgar Nielsen | 13 Sep 2008 07:19 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111632: eventBuilder.stagnant.NumEventsSent

See more ...


Edgar Nielsen | 13 Sep 2008 07:16 GMT | Ice Cube/Runs/RCAs | | permalink

Oxidizer on the back deck

See more ...


Edgar Nielsen | 13 Sep 2008 01:48 GMT | Ice Cube/ICL | | permalink

Mopping 2nd floor of ICL

See more ...


Edgar Nielsen | 13 Sep 2008 00:10 GMT | Ice Cube/ICL | | permalink

Sep 12, 2008

Daily checks

See more ...


Edgar Nielsen | 12 Sep 2008 22:54 GMT | Ice Cube/ICL | | permalink

Shutdown ic-tape and ic-tape2

See more ...


Edgar Nielsen | 12 Sep 2008 17:23 GMT | Ice Cube/SPS | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 12 Sep 2008 17:11 GMT | Ice Cube/Misc | | permalink

Daily checks

See more ...


Edgar Nielsen | 12 Sep 2008 17:10 GMT | MAPO | | permalink

RCA for run 111628: Serology:ClosedByInterrupt

See more ...


Edgar Nielsen | 12 Sep 2008 03:53 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111625: Pandavas/dropped_dom

See more ...


Edgar Nielsen | 12 Sep 2008 02:22 GMT | Ice Cube/Runs/RCAs | | permalink

Quickie hub spare "inventory"

See more ...


Edgar Nielsen | 12 Sep 2008 00:16 GMT | Ice Cube/Misc | | permalink

Sep 11, 2008

Daily checks

See more ...


Edgar Nielsen | 11 Sep 2008 23:39 GMT | Ice Cube/ICL | | permalink

Took off MAPO window coverings

See more ...


Edgar Nielsen | 11 Sep 2008 23:18 GMT | MAPO | | permalink

MAPO temperatures

See more ...


Edgar Nielsen | 11 Sep 2008 23:17 GMT | MAPO | | permalink

RCA for run 111622: xmlrpc-timeout_DAQMoni.py

See more ...


Edgar Nielsen | 11 Sep 2008 00:24 GMT | Ice Cube/Runs/RCAs | | permalink

RCA for run 111620: xmlrpc-timeout_DAQMoni.py

See more ...


Edgar Nielsen | 11 Sep 2008 00:23 GMT | Ice Cube/Runs/RCAs | | permalink

Sep 10, 2008

Daily checks

See more ...


Edgar Nielsen | 10 Sep 2008 18:44 GMT | MAPO | | permalink

Setting up dataprobe unit

See more ...


Edgar Nielsen | 10 Sep 2008 18:42 GMT | Ice Cube/Infrastructure | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 10 Sep 2008 16:44 GMT | Ice Cube/Misc | | permalink

daily checks

See more ...


Edgar Nielsen | 10 Sep 2008 16:43 GMT | Ice Cube/ICL | | permalink

ICL iridium modem power cycled

See more ...


Edgar Nielsen | 10 Sep 2008 16:43 GMT | Ice Cube/Infrastructure | | permalink

Sep 09, 2008

Weekly sweep of the ICL

See more ...


Edgar Nielsen | 09 Sep 2008 22:50 GMT | Ice Cube/ICL | | permalink

Fluke recorders at ICL

See more ...


Edgar Nielsen | 09 Sep 2008 21:29 GMT | Ice Cube/Power | | permalink

Scott Tent

See more ...


Edgar Nielsen | 09 Sep 2008 21:25 GMT | Misc | | permalink

SPADE tape tape_2008_498_RAW_copy_1 full. Inserted new tape.


Ethan Dicks | 09 Sep 2008 15:36 GMT | SPADE | | permalink

RCA for run 111614: Eric_Coplin:ClosedByInterrupt.

See more ...


Edgar Nielsen | 09 Sep 2008 12:51 GMT | Ice Cube/Runs/RCAs | | permalink

Made ic-tape2:/dev/nst0 available to SPADE

See more ...


Edgar Nielsen | 09 Sep 2008 12:39 GMT | SPADE | | permalink

ic-tape2:/dev/nst1 tape change

See more ...


Edgar Nielsen | 09 Sep 2008 11:55 GMT | SPADE | | permalink

sps-sattx2 reboot

See more ...


Edgar Nielsen | 09 Sep 2008 11:41 GMT | Ice Cube/SPS | | permalink

More on ic-hsm

See more ...


Edgar Nielsen | 09 Sep 2008 05:14 GMT | Ice Cube/SPS | | permalink

ic-tape2:/dev/nst3 tape change

See more ...


Edgar Nielsen | 09 Sep 2008 03:23 GMT | SPADE | | permalink

Daily checks

See more ...


Edgar Nielsen | 09 Sep 2008 00:36 GMT | MAPO | | permalink

Sep 08, 2008

Physical DNF checks

See more ...


Edgar Nielsen | 08 Sep 2008 23:51 GMT | Ice Cube/Misc | | permalink

ic-hsm problems

See more ...


Edgar Nielsen | 08 Sep 2008 17:30 GMT | Ice Cube/SPS | | permalink

Upgrading WxGoose firmwares @ ICL

See more ...


Edgar Nielsen | 08 Sep 2008 16:37 GMT | Ice Cube/Infrastructure | | permalink

Daily checks

See more ...


Edgar Nielsen | 08 Sep 2008 15:57 GMT | Ice Cube/ICL | | permalink

Sep 07, 2008

Returning to normal runs

See more ...


Edgar Nielsen | 07 Sep 2008 23:49 GMT | Ice Cube/Runs | | permalink

Visual DNF checks

See more ...


Edgar Nielsen | 07 Sep 2008 23:01 GMT | Ice Cube/Misc | | permalink

Daily checks

See more ...


Edgar Nielsen | 07 Sep 2008 23:01 GMT | Ice Cube/ICL | | permalink

Daily checks

See more ...


Edgar Nielsen | 07 Sep 2008 23:00 GMT | MAPO | | permalink

Started run 111609 without IceTop for special runs

See more ...


Ethan Dicks | 07 Sep 2008 08:10 GMT | Ice Cube/Runs | | permalink

Partitioning the detector to In-Ice and IceTop for special IceTop runs

See more ...


Ethan Dicks | 07 Sep 2008 07:55 GMT | Ice Cube/Runs | | permalink

Sep 05, 2008

sps-sattx disk corruption

See more ...


Edgar Nielsen | 05 Sep 2008 18:23 GMT | Ice Cube/SPS | | permalink

Sep 04, 2008

Upgraded firmware on goose4 (goose-b2-00)

See more ...


Edgar Nielsen | 04 Sep 2008 22:29 GMT | Ice Cube/Infrastructure | | permalink

sps-sattx3 down

See more ...


Edgar Nielsen | 04 Sep 2008 22:19 GMT | Ice Cube/SPS | | permalink

Killed old pDAQ processes on sps-build64

See more ...


Edgar Nielsen | 04 Sep 2008 16:50 GMT | Ice Cube/SPS | | permalink

Replaced defective DIMM in sps-2ndbuild

See more ...


Ethan Dicks | 04 Sep 2008 12:45 GMT | Ice Cube/SPS | | permalink

Stopping run 111597 to cycle DOM 63-41 "Feta"

See more ...


Ethan Dicks | 04 Sep 2008 12:28 GMT | Ice Cube/Runs | | permalink

Restarting normal runs with run 111593

See more ...


Ethan Dicks | 04 Sep 2008 08:31 GMT | Ice Cube/Runs | | permalink

Sep 03, 2008

sps-ichub70 DevManBE memory leak

See more ...


Edgar Nielsen | 03 Sep 2008 18:23 GMT | Ice Cube/Infrastructure | | permalink

False fire alarm

See more ...


Edgar Nielsen | 03 Sep 2008 11:30 GMT | Misc | | permalink

Sep 02, 2008

SPADE tape tape_2008_506_RAW_copy_1 full. Inserted new tape.


Ethan Dicks | 02 Sep 2008 03:53 GMT | SPADE | | permalink

SPADE tape tape_2008_505_RAW_copy_1 full. Inserted new tape.


Ethan Dicks | 02 Sep 2008 00:45 GMT | SPADE | | permalink

Sep 01, 2008

ic-tape2:/dev/nst1 tape change

See more ...


Edgar Nielsen | 01 Sep 2008 02:10 GMT | SPADE | | permalink

ic-tape2:/dev/nst3 tape change

See more ...


Edgar Nielsen | 01 Sep 2008 02:10 GMT | SPADE | | permalink