The solenoid was found to be warm on Monday morning. M9Q3 had a bad water 
leak over the weekend. The water header serving Q3 was turned off and the 
leak confirmed. Q3 was isolated and the header was turned back on. My 
supposition is that while the main header was turned off, the diff. pump 
under the solenoid tripped. When I looked at it this morning the 
compressor was off and the vesssel vacuum was very bad. The PLC connection 
to EPICs is down, so I can not restart the vacuum system yet. Mike Leross 
is looking at the problem. 
There was no trip signal to the main control room. It looks as though the 
control room signal was bypassed downstairs. 
The power supply fault light was on on the QRL panel, but I okayed Bob 
Hilton locking out the power supply in order that Q3's water leak can be 
repired. Shutting of the main breaker may have turned on this light. The 
other possibility is that the power supply was not ramped down and shut 
off properly.

see bitmap for temps/pressures after presurrizing coldbox. Preparing to 
start flow to turbines, then will start flow to solenoid in parallel warm 
return.

Solenoid tripped on a power bump at 18:40 Tuesday
-- Donald Arseneau

From about 08:00 to 15:00

See Doug's previous entry for quench recovery details.
First the system had been set to // cold return mode because the solenoid 
temps seemed low enough. Temps started to rise instead of cooling so system 
was eventually set back to // warm return mode. With the system in // warm 
return mode the solenoid was still warming instead of cooling. Solenoid 
temps were rising to almost 50K and T1 shield heater was about 90%. T1 
inlet temp was below 75K and falling even with shield heat set so high. 
CV155 kept closed and // warm return flows were full flow. T1 brake valve 
seemed way too open at about 0.8 (compared to previous runs).

15:00 to 19:00

Tried opening CV155, didn't seem to help much. Temps still trending 
upwards. Decided to reduce flow to solenoid to see if the cold box would 
cool down. Started reducing the // warm return outlet valves. TI121 began 
to drop slowly. On graph plotting solenoid temps they started to trend 
downwards. Weird. A reduced flow seemed to produce more cooling ? Don't 
understand this but will continue. Flow temps returning from solenoid were 
about the same temp as TI121, about 35 K. Switching to // cold return mode 
to try to reduce TI121. Keeping CV155 as open as possible, just keeping 
CV204 from closing completely. After switching to // cold return TI121 
starts to slowly fall, T1 inlet temp rises over time so start to drop 
shield heater setting. Solenoid temps now trending downwards. Cracked open 
CV142 to get some flow going through powerleads. Raised compressor 
discharge press to about 11.2 barA. Shield heater down over time to about 
10%. Reduced CV155 down to the point where temps almost stop falling then 
open cv155 just enogh to keep temps falling.

19:30 Solenoid temps down to about 28K. Letting system continue cooling 
overnight.

Guessing the cause of the turbine overspeed quench was caused by 
contamination entering the spinning turbine, causing rapid speed changes or 
possibly the continued saga of the water damaged Jaquet turbine speed 
tachometer giving random false trips.

The compressor was restarted and the solenoid is cooling (sort of) in 
parallel cold return.

Solenoid quenched at 6:45.  All temperature monitors were flat until trip.  
Hearing compressor stop was first warning.  Lights are:
Turbine T1 overspeed,  Compresor alarm, cryostat quench.

--Donald Arseneau

See screen grab

This is a strip tool  plot of the cooldown

See attached file for a screen grab of temps.

I recieved a call from ops at 22:00 tonight that the solenoid had alarmed. 
I came in and found a cold box fault, with the compressor tripped off. I 
was unable to restart the compressor. I cleared the cryostat quench and 
compressor trips. The system looked as though it was ready to start. I 
pushed the compressor "ON" button, but as soon as the main contactor 
closed, the compressor tripped again. After a few trys and a bit of 
searching, I tried resetting the compressor over-pressure switch. That 
allowed me to restart the compressor. Once the high pressure comes up I'll 
start flow to the turbines.

The solenoid was pumped out Monday and yesterday after the quads just 
downstream of the solenoid were put back in place. The quad just 
downstream of the solenoid was remove to replace two coils on it. 
The cold box vacuum had tripped off some time in the last little while. 
Started it pumping on Monday also. The compressor was started Tuesday 
afternoon and flow was started to the solenoid. The purifier was 
regenerated and put on line in it's total buffer tank flow mode. The cold 
box was pressurized and the turbines were briefly spun up to check the 
operation of the new turbine speed indicator. It seems to be working 
properly. 
Flow to the turbines was started this morning and all went well. The speed 
indicator works properly. 
M9BIG1 started ot 6.6 x 10-6 before flow was started. One hour later it 
reads 4.8 x 10-6

The turbine 2 speed indicator control box STA 120 has been replaced with a 
newer style Jaquet controller. This is similar to the ones used in the 
ISAC helium liquifier. See attached photos for set up parameters. These 
were taken from the old controller, Sulzer documentation, and the ISAC 2 
controllers.

The following relays have been replaced in the Sulzer control panel:
K341, K379, K132, K324A, K314, K125, K127, K133, K252, K227, K233, K238, 
K239, K245, K246.
These were the one that had indications of sever rusting inside the case.
There are five more spare relays in the gray cupboard.

The cold box was isolated and pumped out after it warmed up. It has been 
refilled with helium.
Turbine 2 speed indicator failed at some time during the last run. After 
the turbine inlet vavle was closed the turbine continued to read full 
speed. I am removing the controller to look at it. The main power has been 
turned of for this procedure.

I loked at the power lead temps this morning. LHe_in 10.4-12.0  PL-Neg 8.0-
9.0  PL-Pos 9.8-10.4. Flows down to 160mm. Called Ops and got Danica to 
cycle CV142. Ripples and flow improved on the second try.

Ripples got worse overnight.LHe in 10.3-11.7, PL Neg 8.0-8.7. Flows <200mm.
cycled CV142. Flows 240mm. LHe in 10.3-11.3  PL Neg 7.9-8.1

buffer tank pressure is up to 6.5 psi.

Came down to check on system. CV211 was fully open, trying to charge the 
system. Ripples on LHe in and PL neg. CV141 fully closed. I closed CV147 to 
0.45 to try and move CV141 open. Checked the buffer tank pressure, it was 
almost at zero. Lowering the liquid level to 40%.

Got page very early this morning. Couldn't come in til 08:30. System was 
in idle running mode and someone had reset the QRL panel so my guess is 
that the solenoid tripped on powerlead overtemp. HIC142 was not fully open 
and I suspect this is why powerleads heated up. Now in // cold return mode.
Solenoid temps down to 25K and slowly going down. Have HIC142 slightly open 
to keep powerleads cool. Strongly recommend that when running HIC142 should 
be fully open to try to reduce the probability of powerlead warmup.