M9sol Logbooks Solenoid Daily log
  Solenoid logbook, all entries  ELOG logo
ID Date Author Operational Modedown Subject
  4   Fri Apr 16 10:15:26 2004 Doug PreddyStart UpChanges to M9sol Vacuum PC
Yesterday and today there have been a number of changes made to the M9 
Vacuum control computer. The computer now automatically boots up to the 
Epics control page. This logbook is also automatically started.
  5   Mon Apr 19 15:06:49 2004 Doug PreddyStart UpStarted turbines
Purifier run over the weekend. Both LN2 dewars empty, but purifier cold. 
Disabled CV211 and flowed all charge gas through purifier. Started turbine 
at 11:00am. set strip tool to monitor temps. Started M9BCP2. Pumped out 
beamline between BV% and BV7. Opened BV6.
  6   Tue Apr 20 07:06:41 2004 Doug PreddyStart Upmorning day 2
Temps in solenoid are equalling out. Increasing PIC229 to 11 bar. 
  7   Tue Apr 20 07:33:03 2004 Doug PreddyStart UpPIC229 at 11bar
PIC229 at 11 bar. Changed strip tool to 15 sec increments.
  8   Wed Apr 21 06:39:17 2004 Doug PreddyStart Upover night
Changed to parallel cold mode at 9:30.
At about 4:30 the city water chiller was turned on. The air compressor had 
also tripped off. It was reset and restarted.
  9   Wed Apr 21 06:48:31 2004 Doug PreddyStart UpJT bypass closed
Closed cv155
  10   Wed Apr 21 08:16:18 2004 Doug PreddyStart Upswitch to series mode
Switched to series mode.
  11   Wed Apr 21 09:57:59 2004 Willy AnderssonStart UpSystem status
Buffer tank pressure is down to about 6 psig. Too low to keep up
with liquefaction requirements to build LHe level. Doug Preddy
requested that I help Curtis Ballard top up system with new
bottle of UHP helium Through purifier. As of 09:45 not yet able 
to find Curtis.

PI229 discharge pressure dropped slightly to 10.45 barA. Set point
is 11 barA. Turbine speeds OK. CV211 is wide open. Energized HIC142
and set to 0.7 to start flow to powerleads. Up in compressor room
checked purifier. LN2 system running OK. In purifier package flowmeter
looks like oil or brownish spots on inside of rotameter glass. Possible
migration of trapped contaminants after correcting flow arrangement
through purifier. Initially buffer tank circulation flow entered
through AV3 but was corrected to enter through AV5.

Have 10:00 meeting, system seems stable and should be OK until I can
contact Curtis to top up system with UHP helium through purifier.
  42   Thu Aug 19 10:28:32 2004 Doug PreddyStart Upafter M9bv5 FIX
Over the last couple of days we replaced the gate on the M9BV5 valve. The 
old one had worn the teflon sliders on the sides. Metal shavings on bottom 
of body. Found a replacement with metal rollers, rather than slides. 
Pumped out downstream of gate valve. It's down to 5E-7 (moved the cable on 
the ion gauge so it now reads.)
Started cooling down the solenoid. Charging through the adsorber. LN2 fill 
piping broke. Repaired by resoldering. 
Ion gauge on solenoid seems to have burnt out last night.
  52   Mon Oct 4 09:16:29 2004 Doug PreddyStart UpStarted turbines
Started flow to the turbimes with the solenoid in parallel warm return.
  53   Tue Oct 5 07:18:16 2004 Doug PreddyStart Upadjust compressor discharge pressure
adjusted compressor discharge pressure to 11 Bar
  54   Wed Oct 6 07:57:01 2004 Doug PreddyStart Upparallel cold return
Switched to parallel cold return at 4:00 yesterday afternoon.
6:30 this morning started closing cv155 as temps had bottomed out at 24K.
  55   Wed Oct 6 09:25:54 2004 Doug PreddyStart Upswitch to series mode
switched to series mode
  56   Thu Oct 7 13:33:20 2004 Doug PreddyStart Upturned off purifier
turned off purifier
  58   Sun Oct 10 17:36:35 2004 ExperimentorStart Uppower leads
At 5:30 pm sundat, power lead temp is about 10.6K.  script tool still 
notworking

syd kreitzman
  59   Mon Oct 11 03:56:14 2004 ExperimentorStart Upcv142 valve
noticed that the lead temp rose to 24K over the past 4 hours. had control 
room personnle cycle the cv142 valve. temperature dropped.

srk
  60   Mon Oct 11 11:49:34 2004 ExperimentorStart Upcv142 valve
noticed temp rise, which occured only about 5 hours after the 
last "reset" .... the time between resets seems to be decreasing, which is 
not a good sign. 

had the controll room guys reset the cv142 valve (i.e. cycle it off/on)

syd
  61   Mon Oct 11 22:32:11 2004 ExperimentorStart Upcv142 valve
temp of leads begins its exponential rise again (caught it at 12K), approx 
11 hours after last reset.  this time asked operator to shut off valve for 
3 seconds, then turn back on. 

previous recycle, asked operator to shut off valve for two seconds ... and 
the warm up time was 11 hours.

previous^2 recyle, asked operator to shut off valve for 1 second ... and 
the warm up time was only 4 hours. 

Syd
  62   Tue Oct 12 02:05:31 2004 ExperimentorStart Upcv142 valve
after only two hours since last lead cycling, they warm up quickly. 
ops cycles the valve, but more quickly this time, off ... 1 sec ... on. 

syd
  63   Tue Oct 12 13:00:08 2004 Curtis BallardStart Upcv142 valve
Curtis closes and opens slowly at about 11:00 am this morning. Wants to see 
if that works better than the sudden on/off switch. 

reported by srk
  64   Wed Oct 13 08:17:00 2004 ExperimentorStart Upcycling 142 when temp hit 10.68
I tried cycling the valve twice -- at about 11:30 and again around 1:00,
trying both the buttons and the dial -- but it has had no effect on
the temperatures (aside from the transient spikes).  That might be
bad.  We will have to see if it altered the trend.

-- 
Donald Arseneau               
  67   Sun Oct 17 04:26:29 2004 OtherStart Upcycling 142 when temp hit 10.68
> I tried cycling the valve twice -- at about 11:30 and again around 1:00,
> trying both the buttons and the dial -- but it has had no effect on
> the temperatures (aside from the transient spikes).  That might be
> bad.  We will have to see if it altered the trend.
> 
> -- 
> Donald Arseneau               

It did -- temperatures held stable for several days
  68   Sun Oct 17 05:03:13 2004 OtherStart UpCycle valve after oscillations begin
(Sat night/Sun AM) Helium temperature started oscillating, so
powered off magnet (precautionary) and cycled valve 142 (at 04:12).
Small oscillations begin almost immediately, so close and open
the twist-knob for 142.  Works better.
  69   Wed Oct 20 09:38:17 2004 Doug PreddyStart Uprestarting system
Solenoid tripped  off Monday morning due to high compressor pressure. 
Regenerated purifier on Tuesday. Started compressor at 9:00 this morning.
  70   Thu Oct 21 07:16:35 2004 Doug PreddyStart Upswitched to parallel cold return
  71   Thu Oct 21 15:41:29 2004 Doug PreddyStart Upswitched to series mode
  72   Fri Oct 22 09:15:06 2004 Doug PreddyStart Upstarted flow to power leads
Started flow to power leads HIC142 at 0.8. 
  94   Wed Jun 1 08:23:19 2005 Doug PreddyStart Upstart up notes
The solenoid was pumped out Monday and yesterday after the quads just 
downstream of the solenoid were put back in place. The quad just 
downstream of the solenoid was remove to replace two coils on it. 
The cold box vacuum had tripped off some time in the last little while. 
Started it pumping on Monday also. The compressor was started Tuesday 
afternoon and flow was started to the solenoid. The purifier was 
regenerated and put on line in it's total buffer tank flow mode. The cold 
box was pressurized and the turbines were briefly spun up to check the 
operation of the new turbine speed indicator. It seems to be working 
properly. 
Flow to the turbines was started this morning and all went well. The speed 
indicator works properly. 
M9BIG1 started ot 6.6 x 10-6 before flow was started. One hour later it 
reads 4.8 x 10-6
  111   Wed Nov 2 10:17:27 2005 Willy AnderssonStart UpM9B helium refrigerator and solenoid cooldown
Started M9B MYCOM helium compressor circulating helium through system last 
Friday. Right hand purifier was regenerated previously, LN2 fill system 
started and flow started from buffer through purifier to low pressure 
return line to compressor. Left in this mode over the weekend. Curtis came 
in on weekend to add new full LN2 dewar to purifier.

Started // warm return cooling Monday.
Tuesday morning solenoid temps about 35 K and TI121 about 35 K  so 
switching to // cold cooling.

Note:

PDI 104 guage is very slugish to respond, needs to be replaced or cleaned.
TI148 and TI149 need to be recharged with helium, they do not work yet.
TIC121 may be reading too high a temperature, try to verify it with TE147 
before JT effect begins. Now TI121=20.7 K   TE147=25 K   HIC155 is full 
open still.
LIC108 is reading too high, reads 22% LHE but really is no LHE at this 
point.
  117   Tue May 23 08:44:28 2006 Doug PreddyStart UpStarted turbines
Solenoid and cold box were pumped out last Tuesday. Cold box roughing pump 
initially would not start. had to retighten all wires on the contactor in 
the Sulzer control panel (RP900).
Started compressor on Friday. ran purifier over the weekend.
Started flow in parallel warm return. Started flow to turbines.
Replacing dead monitor on other control computer.
  118   Thu May 25 07:34:18 2006 Doug PreddyStart Upswitched to parallel cold return
switched to parallel cold return
  119   Fri May 26 07:29:54 2006 Doug PreddyStart UpCV155 closed
CV155 closed.
Temps about 15K
  120   Fri May 26 08:45:38 2006 Doug PreddyStart UpChanged to series mode
Changed to series mode.
Liquid level building.
  121   Fri May 26 10:03:03 2006 Doug PreddyStart UpStarted flow to power leads
Started flow to power leads.
Liquid level 15%
  122   Fri May 26 14:34:24 2006 Doug PreddyStart UpReady to run
Solenoid is cold, liquid level is 35%, power leads cold and seem stable.
Added one bottle helium to buffer tank. Pressure is 7 psi..
  162   Wed Oct 4 07:29:04 2006 Doug PreddyStart UpStarting cool down
Started compressor yesterday with flow through the purifier overnight.
Starting tubines this morning in parallel warm return
  163   Wed Oct 4 18:10:18 2006 Willy AnderssonStart UpQuick check on system
System in good // warm return mode, temps coming down nicely. BCP2 cryopump 
does not look too healthy, temperature is just over 20K, very close to low 
set point.

TIC121 down to 120K, Mu1 return temp at 164K, Mu2 return at 273K, T1 inlet 
temp at 208K. Cooling looks good for overnight.
  164   Thu Oct 5 10:26:26 2006 Doug PreddyStart Upswitched to parallel cold return
  165   Thu Oct 5 11:51:53 2006 Doug PreddyStart UpAdde 2 bottles UHP Helium
Valved off purifiers, but left LN2 on.
  166   Thu Oct 5 14:53:54 2006 Doug PreddyStart UpSwitched to series mode
  187   Thu Apr 19 15:18:25 2007 Doug PreddyStart Uppreparing to start the refrigerator
Starting the vacuum systems in the coldbox and solenoid.
  188   Fri Apr 20 12:15:12 2007 Doug PreddyStart Upremote PLC switch
I could not control the vacuum devices using EPICS. The buttons could be 
pushed, but nothing would respond. The PLC was in local mode. Thumbwheel 
to 100, push On/Open button to put it into remote mode.
  189   Mon Apr 23 16:31:02 2007 Doug PreddyStart UpStarted compressor
Startewd compressor and purifer today.
  190   Tue Apr 24 10:21:49 2007 Doug PreddyStart UpStarted flow to turbines
Started flow to turbines at about 10:00 in parallel warm mode.
  217   Fri Sep 14 12:44:52 2007 Doug PreddyStart Upstarted compressor
running purifier over weekend
  220   Mon Sep 24 15:08:41 2007 Doug PreddyStart UpStarted cool down this morning
The vacuum in the cold box was good on Friday 0.0 X 10-4. Let it pump over 
the weekend. Added two bottles of UHP helium to system Friday.

Started the turbines this morning. Cooling in paralell warm return. 

Solenoid vacuum is 3.9 x 10-6
  221   Tue Sep 25 07:34:04 2007 Doug PreddyStart UpPIC229 to 11 bar
adjusted turbine speeds and HIC104 to maintain their values.
  222   Wed Sep 26 08:52:46 2007 Doug PreddyStart Upswitched to parallel cold return
  223   Thu Sep 27 07:41:15 2007 Doug PreddyStart UpSolenoid cold
Solenoid cold and in series mode. 50% liquid level. Starting flow to 
powerleads.
  243   Tue Apr 1 15:17:06 2008 Doug PreddyStart UpStarting up vacuum systems
Monday afternoon there was no control of the vacuum system. The controls were working, but no output to devices. The "output" breaker in the PLC box was found to be tripped. Reseting it allowed control to the vacuum systems. I started the regeneration of BCP2 this morning, but hte output breaker tripped again. reset it and continued to pump. Started to pump out the solenoid also. Removed jumper on the BDP2 temperature read-back, bypassed BV2, and BCG2. This allows pumping through the diffussion pump. Cold box has been pumping for a few weeks. It's at 2.0 x 10-3
  245   Wed Apr 16 15:15:44 2008 Doug PreddyStart Uprefrigerator started
Turned on compressor last Tuesday. Ran purifier until today. Final value 0.15ppm. Started cool down yesterday. Switched into parallel cold return this morning, but PIC 229 would not allow the compressor to charge. The compressor tripped due to bad solenoid vacuum (solenoid was warming up). The controller now seems to be working okay. Restarted in parallel warm mode at ~12:00
  246   Wed Apr 16 16:51:16 2008 Doug PreddyStart Upswitched to parallel cold return
  247   Thu Apr 17 13:52:25 2008 Doug PreddyStart Upswitched to series mode at 9:30 this morning
switched to series mode at 9:30 this morning. Added two bottles of helium gas. Started flow to power leads.
  274   Wed Mar 24 08:30:33 2010 Doug PreddyStart UpStart up

Compressor started yesterday afternoon. Pumping out solenoid overnight. Cold box vacuum good. circulating helium to solenoid in Parallel warm mode, CV142 open. Purifier running.

  275   Mon Mar 29 09:25:52 2010 Kevin TrithardtStart UpStarted cool down this morning
  276   Thu Apr 1 11:57:55 2010 Doug PreddyStart Upstarted flow to power leads

Solenoid cold. Buffer tank pressure ~3psi, so Kevin will add a bottle of helium. Started flow to power leads at 11:45

  278   Fri Apr 9 16:07:28 2010 Doug PreddyStart Upflow to power leads started
  49   Mon Sep 13 10:55:25 2004 Doug PreddyShut DownQuench and shutdown
The solenoid quenched on powerlead temps this morning at 10:00. It is 
warming up and the power leads are >200K.Isolating and purging the coldbox 
is not very useful now, so I am shutting the system down.
  50   Mon Sep 13 13:56:06 2004 Doug PreddyShut Downcompressor off
compressor off
  116   Thu Dec 22 15:10:44 2005 Doug PreddyShut Downshut down system
shuttting down the system. isolated cold box by closing v135. preparing to 
pump out cold box.
  265   Wed Dec 10 15:41:04 2008 Doug PreddyShut Downshuttting down solenoid
shuttting down solenoid for winter shutdown.
  284   Mon Aug 9 14:27:12 2010 Doug PreddyShut DownHelium plant shut down
  1   Fri Dec 19 11:34:10 2003 Doug PreddyQuench RecoveryFinal home of this E-log
http://isac.triumf.ca/m9sol/
Chris has set up the M9 solenoid e-log at the above web address. Play with 
it and let me know if it breaks. 
  14   Sun Apr 25 21:41:00 2004 Willy AnderssonQuench RecoveryTrip on PL Temp.
Got pager call at 16:20 this afternoon. Solenoid had tripped on PL 
temperature. Arrived at about 17:00. Did a print screen of the striptool 
graph and solenoid screen. See files Prtscrn1_april25_2004.doc and
Prtscrn2_april25_2004.doc. System was in idle running mode. LHE heater at 
about 30%, buffer tank press at21.5 psig, shield heater at 37.5%. The
jumper is still on the back of the vacuum guage PIC901. The compressor 
building backup air compressor was tripped again, I reset it. HIC 141 was 
set to 0.46 , I set it to 1.1 (closed position). Set solenoid inlet and 
outlet HIC's to valve closed positions. Energized warm return HIC's then 
slowly opened warm return valves to vent pressure in solenoid. Cracked open 
solenoid inlet valves to get abot 1.5 barA pressure on PI143 and PI146. LHe 
level started to drop. Contiued slowly until all LHe boiled away. PI143 and 
PI146 at about 5 barA. Had to adjust turbine speeds and shield heat. Opened 
cold returns then closed warm returns. Gradually increased solenoid inlet 
valves until wide open. Adjust turbine speeds and shiled heater. At about 
21:00 stable // cold return cooling. Temperatures dropping gradually. See 
files april25_2004_2113.BMP and april25_2004_2114.BMP 

Have powerlead supply valve HIC142 open to 0.46 during cooling just to see 
what happens. Time now about 21:42. Leaving system in // cold return 
cooling. Should be ready for series switch over in the morning.
  15   Mon Apr 26 07:48:46 2004 Doug PreddyQuench RecoverySeries mode
Switching over to series mode.
  16   Mon Apr 26 08:12:30 2004 Doug PreddyQuench RecoveryPower leads flow
Throttled back the LHe in until the power leads started to warm. My goal is 
to flow "warm" gas (hopefully 80K) through the leads to drive any frozen 
impurities out.
  19   Thu Apr 29 23:43:15 2004 Doug PreddyQuench RecoveryStarting flow to solenoid
Solenoid quenched due to power leads at 9:00 tonight. Restarted flow in 
parallel warm mode. 
  20   Fri Apr 30 01:53:34 2004 Doug PreddyQuench Recoveryswitched to parallel cold
  23   Wed May 5 07:55:37 2004 Doug PreddyQuench RecoverySolenoid quenched on power lead temp
Solenoid quenched on power lead temp. I ran down and managed to recover it 
with minimal temp rise in the solenoid. Coil temps are falling, liquid is 
building. No flow to the power leads yet.
  24   Wed May 5 09:59:10 2004 Doug PreddyQuench Recoverystarted flow to power leads
Started flow to power leads.
TIC121 into auto mode
  26   Wed May 5 23:57:38 2004 Willy AnderssonQuench RecoveryPowerleads warming up again.
Got a page from Don Flemming at about 18:50 wondering why the power
lead temperatures were rising. See attachment may5_04_1800.BMP It looked as
though the monitoring signals flat lined for about 2 hours for some reason.
Anyway the powerlead temps were definitely rising. When I arrived HIC142
was wide open but both power lead flows were down to 65 (on the scale) and
the flowmeter float was vibrating rapidly. LHe heater at 33 %. Buffer 
pressure at 10 psig. LIC108 at 54 %. CV147 at 0.45 (HIC141). CV141 at 0.6
TCI121 in auto mode.

Closed HIC142 then opened again. Powerlead temps dropped. Opened CV147
from 0.45 to 0.35 (HIC141) Played with setting of set point of TIC121 and
HIC141 to try to increase excess liquefaction. Reduced shield heat to about 
16%. Difficult to say what is going on with powerleads. Temps had dropped to
good levels but now gradually rising, but very slowly. Hopefully they peak 
and level off. Energized solenoid. Cycled HIC142 a few times but temps
would not trend downwards. Leaving it overnight. Doug, please check and 
adjust the system as required in the morning.
  27   Thu May 6 09:32:13 2004 Willy AnderssonQuench RecoveryAdditional notes from last night power lead problems.
When performing last nights refrigerator adjustments I also increased the 
compressor discharge pressure to 11.25 barA. Also the backup air compressor 
was tripped off again and I reset it. Finally I opened the city water flow 
to the compressor water package chiller. PI110-1 reads 9.25 barA and PI120-1
reads just over 4 barA. Both power lead flowmeters pegged to top of scale.
Reset the "first trip" device that Don Dale had loaned us.

This morning the power leads had crept up slightly but still well below 
trip settings. Oscillations have not appeared yet. LHe vessel heater at 
about 14 to 15 %. Shield heat at about 16 %. Saved screen shots of 
m9_solenoid.dl and striptool windows to files may6_04_0900.BMP and 
may6_04_0905.BMP. TE106 is 81.6 Kelvin. Solenoid Current is stable at 404 
Amps. (DVM meter reads 20.2) Both power lead flowmeters still pegged to top 
of scale.

Closed HIC142 for 1 second then opened it wide open again to see effect on 
power lead flows. Other than the initial blip they just fall back to their 
previous value, no improvement. See file may6_04_0915.BMP. Adjusting 
turbine speeds slightly. Going to take a check in the compressor building 
then going on to other work.
  29   Fri Jun 25 12:36:04 2004 Willy AnderssonQuench RecoveryCompressor unloaded etc.
At about 09:30 this morning I noticed the sound coming from the compressor 
building seemed louder than usual. After entering the building I noted that 
the compressor was unloaded to about 50% and the compressor suction was 
running at about 15 psig. Down at the SULZER panel the solenoid vacuum 
system was tripped off, the flow was still in series mode but no LHe level,
solenoid temps at over 40 K. Contacted Doug. Cannot figure out why the 
compressor was unloaded to 50 %. Solenoid had not tripped on bad vacuum 
because jumper was still on PIC901. Doug mentioned there was a power bump 
yesterday at about 15:00. May be when the vacuum system tripped but does 
not explain the compressor unloading.

Doug restarted vacuum system. I stopped flow to solenoid. Stabilized 
pressures. We checked the compressor to try to see what would cause the 
Unloaded state. Could not see any oil leaks. 

Back at Sulzer panel loaded compressor to 100 %. Stabilize pressures. STart
cooling flow to solenoid, // warm return. Adjusting turbine speeds and 
shield heat as required. Switched to // cold return flow after solenoid 
temps down to TIC 121 temp of about 20 K.
  30   Fri Jun 25 14:20:58 2004 Willy AnderssonQuench RecoverySwitch to series mode
At about 14:00 solenoid temps dipping below 8 K. Trace of LHe level (dense 
gas) starting to show. Switching to series cooling mode.
  32   Tue Jul 27 23:33:27 2004 Doug PreddyQuench RecoverySolenoid quench 9:30
I was phoned at about 9:30 tonight with a solenoid alarm. After looking at 
the display I found the system to be tripped off. When I came in I found 
the compressor off and the solenoid quenched. There was no real indication 
of the problem other than a cold box fault. I restarted the compressor and 
started flow to the cold box.
Just a reminder: This elog is now at: https://elog.triumf.ca/M9sol
  33   Wed Jul 28 00:04:30 2004 Doug PreddyQuench Recoverystarted flow to solenoid
Solenoid temps were: C1 & C12 about 30K. C6 &C7 about 11K. Started flow to 
solenoid in parallel cold return. Jumpered out the vacuum gauge for the 
solenoid vacuum.
  34   Wed Jul 28 07:00:18 2004 Doug PreddyQuench Recoveryswitch to series mode
Switching to series mode. Jumper on vacuum gauge removed.
  35   Wed Jul 28 21:26:19 2004 Doug PreddyQuench RecoveryDouble Power bump
Two power bumps tonight. Everything tripped of. Restarted cooling water 
pump, diff. pump under solenoid etc. Computer stayed on! Starting 
compressor.
  36   Wed Jul 28 22:01:31 2004 Doug PreddyQuench RecoveryStarting flow to solenoid
c1 28, c12 29, c6 11.6, c7 12.5 TI121 22.5
  37   Wed Jul 28 23:07:01 2004 Doug PreddyQuench Recoverysolenoid cooling
cooling rates for the solenoid are good. In parallel cold return.
  38   Thu Jul 29 09:01:11 2004 Doug PreddyQuench Recoveryswitched to parallel cold
Changed to series mode.
  39   Thu Jul 29 13:25:29 2004 Doug PreddyQuench Recoverystart flow to power leads
  40   Sat Aug 7 22:37:04 2004 Willy AnderssonQuench RecoveryCompressor tripped off.
Got a call at about 10:00 this morning, the whole system had tripped off. 
No power bumps or other disturbances. When I arrived the only fault 
indications were "ColdBox Trip" light on the compressor panel and on the 
QRL panel the red "ColdBox" LED was ON. On the SULZER panel the 
yellow "Cryostat Quench" light was ON. On the First Trip logger #1 and #15 
LED's were ON. But no indication about what the actual fault was. Cooling 
water OK, compressed air OK, compressor ventilation OK. Outside ambient 
temperature was cooler than we had last week. BIG1 was off but when I 
turned it back on the vacuum looked good but was slowly rising as the 
solenoid warmed. I put the jumper on the vacuum controller. Prepared system 
for re-start.

Started compressor. Started flow thru coldbox. Raised discharge pressure.
Started turbine flow. Didn't get any freezing but it seemed that T1 speed 
wandered slowly +/-0.1k rev/sec when it got up to about 3.6k revs/sec, but 
this only lasted for a second or two. Not sure if I was just seeing things.
Got turbines up to speed and started // warm return flow. Solenoid had been 
running at 500 Amps and when it quenched and sat warming it had warmed to 
about 40 K. Switched it to //cold return too soon and solenoid started to 
warm slightly. Back to // warm return mode. Had to throttle flow thru 
solenoid with warm return valves to get effective cooling. If had return 
valves wide open TI121 would warm up too much. Discovered that if I 
throttle the flow out of the solenoid slightly then TI121 would fall and 
solenoid would start to cool. Less flow gave more cooling. Don't quite 
understand that one. Switched to // cold return mode when solenoid temps 
falling below 20K.

About 15:00 stable // cold return cooling. Going home for a while.

19:30 solenoid temps bottomed out at about 11K. Switch to series mode. 
Something strange about the first trip detect. I reset it once and when I 
checked it an hour or two later it showed a trip on #1 and #15 LED's even 
though nothing had happened. Something must be causing false indications.

20:15 series cooling, solenoid temps at about 6K, LHe level rising slowly 
at about 10%. Removed jumper on vacuum controller.
  41   Sat Aug 7 23:40:47 2004 Willy AnderssonQuench RecoveryEnergizing solenoid
23:39 Powerleads cold, LHe level over 40% and rising. Energizing solenoid 
to 500 Amps, ramp rate 60. Checking compressor building then going home.
  47   Thu Sep 2 21:58:34 2004 Willy AnderssonQuench RecoverySwitch to series cooling
Solenoid had tripped on powerlead overtemp at about 09:00 this morning. 
Doug recovered the system to // cooling. I switched system to series 
cooling at about 21:30. LHe level at about 10%. Buffer tank pressure is low 
at about 13 psig. Letting LHe level build. Once LHe level rises to about 
35% then I suggest starting flow to the powerleads by opening HIC142 fully 
open. LHe level should continue to build but at a much slower rate. Once 
power lead temps have bottomed out to cr_pl_neg < 9K and cr_pl_pos < 11 K 
(check these values with archived data first) and are stable then can 
energize solenoid.
  48   Fri Sep 3 01:28:43 2004 Doug PreddyQuench Recoverypower up
started flow to the power leads at 11:30. Cold and stable at 1:00am. 
Energized solenoid at 1:30.
  78   Tue Nov 23 07:47:00 2004 Doug PreddyQuench RecoverySolenoid quench Nov 21
Solenoid quenched Nov 21 due to alarge power bump. Everything tripped off 
including vacuum and water pumps. Cold box vacuum was at 3.6e-1, so pumped 
it out overnight.
Started compressor at 7:20. Started turbines at 7:40.
  79   Tue Nov 23 09:30:15 2004 Doug PreddyQuench Recoverystarted flow to solenoid
Started flow to solenoid in parallel warm return.
  80   Tue Nov 23 11:35:44 2004 Doug PreddyQuench RecoveryAnother power bump
Compressor tripped off again due to a power bump. Meson hall water off, so 
vacuum pumps off again.
Pumps back on , vacuum pumps on, starting compressor.
  81   Tue Nov 23 15:10:06 2004 Doug PreddyQuench Recoveryswitched to parallel cold return mode
  84   Wed Nov 24 14:58:42 2004 Doug PreddyQuench Recoverytoday
Came in this morning to find the helium return line to the buffer tank 
frosted up. MV51 has been stuck open the whole time. That exlains the 
cooldown rate and other problem. Cycled MV51 a few times and it now seems 
closed. Switched back to parallel cold return this moring at 8:00. Rates 
are much better. The E-log was also down this morning.
  85   Thu Nov 25 07:21:37 2004 Doug PreddyQuench Recoverystarted power lead flow
Curtis switched to series mode last night at 7:00.
Startedflow to power leads at 7:00 this morning.
  86   Sun Nov 28 10:15:18 2004 Willy AnderssonQuench RecoveryPowerlead quench ?
Got page very early this morning. Couldn't come in til 08:30. System was 
in idle running mode and someone had reset the QRL panel so my guess is 
that the solenoid tripped on powerlead overtemp. HIC142 was not fully open 
and I suspect this is why powerleads heated up. Now in // cold return mode.
Solenoid temps down to 25K and slowly going down. Have HIC142 slightly open 
to keep powerleads cool. Strongly recommend that when running HIC142 should 
be fully open to try to reduce the probability of powerlead warmup.
  95   Wed Jun 1 23:48:02 2005 Doug PreddyQuench Recoverythe first quench
I recieved a call from ops at 22:00 tonight that the solenoid had alarmed. 
I came in and found a cold box fault, with the compressor tripped off. I 
was unable to restart the compressor. I cleared the cryostat quench and 
compressor trips. The system looked as though it was ready to start. I 
pushed the compressor "ON" button, but as soon as the main contactor 
closed, the compressor tripped again. After a few trys and a bit of 
searching, I tried resetting the compressor over-pressure switch. That 
allowed me to restart the compressor. Once the high pressure comes up I'll 
start flow to the turbines.
  96   Thu Jun 2 01:26:30 2005 Doug PreddyQuench Recoverystarted flow to solenoid
See attached file for a screen grab of temps.
  97   Thu Jun 2 01:38:25 2005 Doug PreddyQuench Recoverystrip tool plot
This is a strip tool  plot of the cooldown
  98   Thu Jun 2 09:16:40 2005 Doug PreddyQuench Recoverycontinuing to cool
See screen grab
  100   Thu Jun 9 08:56:10 2005 Doug PreddyQuench Recoverysystem back on
The compressor was restarted and the solenoid is cooling (sort of) in 
parallel cold return.
  101   Thu Jun 9 21:41:25 2005 Willy AnderssonQuench RecoveryTrying to achieve stable cooling.
From about 08:00 to 15:00

See Doug's previous entry for quench recovery details.
First the system had been set to // cold return mode because the solenoid 
temps seemed low enough. Temps started to rise instead of cooling so system 
was eventually set back to // warm return mode. With the system in // warm 
return mode the solenoid was still warming instead of cooling. Solenoid 
temps were rising to almost 50K and T1 shield heater was about 90%. T1 
inlet temp was below 75K and falling even with shield heat set so high. 
CV155 kept closed and // warm return flows were full flow. T1 brake valve 
seemed way too open at about 0.8 (compared to previous runs).

15:00 to 19:00

Tried opening CV155, didn't seem to help much. Temps still trending 
upwards. Decided to reduce flow to solenoid to see if the cold box would 
cool down. Started reducing the // warm return outlet valves. TI121 began 
to drop slowly. On graph plotting solenoid temps they started to trend 
downwards. Weird. A reduced flow seemed to produce more cooling ? Don't 
understand this but will continue. Flow temps returning from solenoid were 
about the same temp as TI121, about 35 K. Switching to // cold return mode 
to try to reduce TI121. Keeping CV155 as open as possible, just keeping 
CV204 from closing completely. After switching to // cold return TI121 
starts to slowly fall, T1 inlet temp rises over time so start to drop 
shield heater setting. Solenoid temps now trending downwards. Cracked open 
CV142 to get some flow going through powerleads. Raised compressor 
discharge press to about 11.2 barA. Shield heater down over time to about 
10%. Reduced CV155 down to the point where temps almost stop falling then 
open cv155 just enogh to keep temps falling.

19:30 Solenoid temps down to about 28K. Letting system continue cooling 
overnight.

Guessing the cause of the turbine overspeed quench was caused by 
contamination entering the spinning turbine, causing rapid speed changes or 
possibly the continued saga of the water damaged Jaquet turbine speed 
tachometer giving random false trips.
  103   Tue Jun 14 20:53:19 2005 Doug PreddyQuench Recoveryrestarted refrigerator after power bump
see bitmap for temps/pressures after presurrizing coldbox. Preparing to 
start flow to turbines, then will start flow to solenoid in parallel warm 
return.
  104   Tue Jun 14 23:01:22 2005 Doug PreddyQuench Recoveryswitched to parallel cold return
  105   Mon Jul 18 11:18:40 2005 Doug PreddyQuench RecoverySolenoid Quenched over the weekend
The solenoid was found to be warm on Monday morning. M9Q3 had a bad water 
leak over the weekend. The water header serving Q3 was turned off and the 
leak confirmed. Q3 was isolated and the header was turned back on. My 
supposition is that while the main header was turned off, the diff. pump 
under the solenoid tripped. When I looked at it this morning the 
compressor was off and the vesssel vacuum was very bad. The PLC connection 
to EPICs is down, so I can not restart the vacuum system yet. Mike Leross 
is looking at the problem. 
There was no trip signal to the main control room. It looks as though the 
control room signal was bypassed downstairs. 
The power supply fault light was on on the QRL panel, but I okayed Bob 
Hilton locking out the power supply in order that Q3's water leak can be 
repired. Shutting of the main breaker may have turned on this light. The 
other possibility is that the power supply was not ramped down and shut 
off properly.
  106   Mon Jul 18 15:51:17 2005 Doug PreddyQuench RecoveryCompressor restarted, flow to solenoid
Recovered the vacuum in the solenoid, then started the compressor.
Started flow to solenoid in parallel warm return.
  107   Tue Jul 19 16:13:32 2005 Willy AnderssonQuench Recoveryswitch to series mode
CR_6KS_M down to 11 K.
Switched to series mode.
Started small flow to power leads to get them cold.
HIC142=0.6        Later open HIC142 wide open.
Raised discharge press to 11.4 barA.
Letting LHe level build.
  108   Tue Jul 19 18:09:56 2005 Willy AnderssonQuench RecoveryQuench II
Machine trip just as I was exiting the TRIUMF gate. QRL said cold box fault 
as did red lite on compressor. At cold box panel all turbine red lites were 
on and compressed air failure lite was on. But there was no problems with 
compressed air. On fist trip logger LED # 8 and 16 were lit. Restarted 
compressor then start flow to cold box, start turbines, start // warm 
return until temps returning back from solenoid were <= TIC121 then 
switched to //cold return mode. Opened cv155 to 0.88
Solenoid temps at 24K and starting to trend down.
  110   Mon Oct 31 07:48:57 2005 Willy AnderssonQuench Recoverywhat a fine entry
Keep IT safe boys
  112   Thu Nov 3 06:19:50 2005 Doug PreddyQuench Recoveryrestart system
Last night at about 7:00 there was a large power bump that shut the 
compressor and water pump off. I came in this morning and am restarting the 
system.
  113   Thu Nov 3 09:27:16 2005 Doug PreddyQuench Recoverystarted flow to solenoid
started flow to solenoid in parallel cold return.
  115   Mon Dec 12 09:24:29 2005 Doug PreddyQuench RecoverySolenoid Quench on power lead temp
The solenoid tripped on power lead temps at about 9:15. I am recovering it 
from it's all around mode into parallel cold return.
  124   Tue Jun 13 10:54:27 2006 Doug PreddyQuench RecoveryCompressor tripped off
No indicator lights etc as to why the compressor tripped. The only  light 
was a cold box fault. I had to shut off the main breaker before the 
compressor would come back on.
Started cooling in parallel  cold return.
  125   Tue Jun 13 15:52:46 2006 Willy AnderssonQuench RecoveryDoug has reduced and now closed CV155.
Now in full // cold return mode.
Had just talked with Curtis, He said thta he has already informed the 
experimentors that the solenoid will not be ready until tommorow, so he 
suggests leaving the system in reduced // cold mode overnight.
  126   Tue Jun 13 16:04:33 2006 Willy AnderssonQuench RecoveryTIC 121 set to auto mode
  127   Wed Jun 14 10:35:30 2006 Willy AnderssonQuench RecoverySwitched to series cooling.
Had system in //return cooling overnight with small flow to keep power 
leads cool. Switched to series mode about 30 minutes ago. Building LHe 
level.
  128   Wed Jun 14 16:27:01 2006 Willy AnderssonQuench RecoveryEnergized solenoid
Powerleads appear stable.
  132   Mon Jul 3 23:01:34 2006 Doug PreddyQuench Recoverypower lead warming/ripples caused quench
power lead warming/ripples caused quench at 18:45 today. Compressor stayed 
on, but flow to solenoid stopped.
I started flow to solenoid at 22:40.
  135   Thu Jul 20 00:25:16 2006 Doug PreddyQuench RecoverySolenoid quenched @ 11:15
Solenoid Quenched on C1 temp. It went above 5K. Shut off compressor. 
Restarting system @ 12:15
  136   Thu Jul 20 00:52:13 2006 Doug PreddyQuench RecoveryT1 frozen
T1 will not spin up. Warming up cold box.
  137   Thu Jul 20 02:59:06 2006 Doug PreddyQuench RecoveryT1 still not starting
I will leave the cold box to warm overnight and try it again in the morning.
  138   Thu Jul 20 08:18:31 2006 Doug PreddyQuench RecoveryT1 started
Started flow to turbines. Both started. T1 seems a bit slow. HIC110 at 0.85 
SI110 = 4100.
Cool down started.
  139   Thu Jul 20 11:01:09 2006 Willy AnderssonQuench RecoveryOpened up warm return valves CV151 and CV153
Mu2 pressure was just about at trip point due to trapped volume of helium 
warming. So opened up warm returns to bleed off pressure and prevent it 
from rising.
  140   Thu Jul 20 13:21:57 2006 Willy AnderssonQuench Recoverystarted // warm return flow.
TIC121 was down to 27K
warm return valves were wide open
opened supply valves CV143 and CV146.
TIC121 went up to about 37K
Mu1 return temperature went down to about same temp as TIC121 so set Mu1 to 
cold return. Open CV145 then close CV151. Mu2 return was still > TIC121 so 
kept it in warm return mode. Adjust turbines as required.
After about 20 minutes Mu2 return temp dropped down to about equal to Mu1 
return temp of about 38K. Then set Mu2 to cold return. Open CV147 then 
close CV143. TIC 121 drops to 30 K. Even though Mu1 and Mu2 return temps 
greater than TIC121 I will keep it in //cold return mode to see if the core 
solenoid temperatures will start to fall. To maximize the use of the 
cooling from T2 outlet temperature CV155 is tweaked open to its maximum 
stable opening. This helps cool the lower heat exchangers. If this works 
then cr_c1 and cr_c12 should start to tick downwards. If they keep rising 
then have to switch back to //warm return.
  141   Thu Jul 20 13:27:21 2006 Willy AnderssonQuench Recoverycold return mode continues...
cr_c1 and cr_12 are both at about 45K and heading slowly down, so the 
gamble of going to // cold return worked. Maximum opening of CV155 is 
limited by CV204 goes fully closed and suction pressure begins to rise 
above set point. Adjust cv155 to keep cv204 just at the closed point. Keep 
tweaking cv155 and turbine speeds as things get cooler.
  142   Thu Jul 20 13:34:26 2006 Willy AnderssonQuench Recoverycold return mode continues...
Even though cr_c1 and cr_c12 have begun to tick down, cr_c7 and cr_c6 are 
actually ticking up. This is probably a wave of warming propagating through 
the solenoid. As long as the two inlets are falling in temperature then 
things are going in the right direction. Once the warm wave travels through 
the outlets will begin to cool as well.
  143   Fri Jul 21 07:39:37 2006 Doug PreddyQuench RecoveryCool down continues
Raised PIC229 to 11Bar. Started to close HIC155. Cooling rates improve.
  144   Fri Jul 21 14:08:04 2006 Doug PreddyQuench Recoveryswitch to series mode
Coils temps <10K.Switching to series mode.
  145   Fri Jul 21 17:18:19 2006 Willy AnderssonQuench RecoveryOpened HIC142 fully open
Power leads have flatlined but still building LHe level.
  146   Fri Jul 21 23:36:21 2006 Doug PreddyQuench RecoveryFinal cooldown problems
80K shield is down to <60K Liquid level not building. Power leads rippling 
and warming. T1 inlet 85K with 10%heater load.
Lowered the set point of TI121 to 17K. Shut off flow to the 80K shield to 
try and warm them. Turbine 1 inlet temp dropping (83K)
Power leads cooled and stablized, coil temps came down. Leaving it for 1/2 
hour to evaluate.
  147   Sat Jul 22 06:34:14 2006 ExperimentorQuench Recoverylooking good
Donald Arseneau reporting.

It is looking pretty good, so I came back in with the intent of energizing 
the magnet.  Doug should be up now or soon, so I'll wait a little for you 
to see this.  If you do, call the m9 counting room 222-7379.  After waiting 
a while, I will just forge ahead.
  148   Fri Jul 28 20:39:40 2006 Doug PreddyQuench RecoveryRestarting refrigerator
Solenoid quenched at about 6:00 tonight.
I restarted the compressor, then the cold box and turbines. When cold box 
was at 25K I started flow to the solenoid in parallel cold return.
  149   Sat Jul 29 05:30:28 2006 Doug PreddyQuench Recoveryswitched to series mode
Temps bottomed out at ~16K. Switching to series mode.
  150   Sat Jul 29 09:38:06 2006 Doug PreddyQuench Recoverypower leads cooled
opened power lead flow about 40 minutes ago. ~12K will let them stablize. 
LHE level >20%
  151   Tue Aug 8 07:27:49 2006 Doug PreddyQuench RecoverySolenoid Quench at about 6:00 this morning
No indication of problem. Turbines still running, but no flow to solenoid. 
Started flow. Solenoid ~20K.
  152   Wed Aug 9 06:44:58 2006 Willy AnderssonQuench Recovery// cold mode cooling continues.
After restarting flow to the solenoid yesterday in // cold return mode 
TIC121 just refused to go lower than about 29K even when we closed CV155 
all the way. Tried various things, switching to // warm return, changing 
opening of CV155, setting TIC121 to auto, even tapping TIC121. Finally 
tried reducing flow to solenoid by reducing // warm return valves to about 
70% open to see if possible to cool just refrigerator a bit.

TIC121 temp fell a bit and then inlet temperatures to solenoid began to 
fall and solenoid slowly began to cool even though the mass flow to the 
solenoid was reduced. Guess is that with solenoid homogeneusly at about 30K 
there was not much resistance to flow so turbine massflow was starved just 
enough to stall further cooling.

After temps were on a good downward trend Doug switched to //cold return. 
At about 16:00 yesterday I reduced CV155 to about 70% and waited a bit to 
confirm that temps were falling and then left the system to cool until 
later that evening.
  153   Wed Aug 9 06:59:43 2006 Willy AnderssonQuench Recoverymore // cold return stuff
About 21:00 last night the solenoid temps had not changed much, pretty well 
flat lined. I guess my reducing CV155 stalled the cooling. Called Don 
Arseneau and asked him to increase CV155 to about 80% open. Temps slowly 
began to fall. Asked Don to look at graph of temps later in the night and 
if they fall down to a flat line to then reduce CV155 a bit to get the 
temps trending further down.

This morning CR_C1_ENT has fallen to a slightly upward trending flat line 
of about 12.7 K. Probably ready for switching to series mode by whoever 
gets to the system first this morning.
  154   Wed Aug 9 07:50:49 2006 Doug PreddyQuench RecoverySwitched to series mode.
Cool down continues.
  156   Thu Aug 10 16:33:11 2006 Willy AnderssonQuench RecoveryFlow to solenoid tripped on powerlead temperature.
Power lead temps were drifting up, tried to tweak TIC121 and HIC141 to 
unstall powerlead flow. Got quench. Restarted series flow, building LHe 
level.
  157   Thu Aug 10 20:26:09 2006 Willy AnderssonQuench Recoverysolenoid energized
Powerleads seem stable.
  158   Tue Aug 15 08:45:56 2006 Doug PreddyQuench Recoveryrestarted the compressor
Solenoid tripped off Sunday. Restarted compressor and cool down of cold box.
  159   Tue Aug 15 15:39:27 2006 Willy AnderssonQuench Recoveryswitched to parallel cold return
Return flow temps from solenoid about 40K, TIC121 at about 40K so switching 
to //cold return.
Starting with slightly reduced flow through cold returns, CV145 and CV147 
at about 65% open.
  160   Thu Aug 17 08:04:44 2006 Doug PreddyQuench RecoverySolenoid tripped at about 3:00 this morning
restarting system.
Compressor would not start. Idid not see anything that would not let it 
start. Pushed rest button upstairs, and reset contactor. Compressor started.
preparing to start flow to solenoid in // cold return.
  161   Thu Aug 17 19:45:27 2006 Willy AnderssonQuench Recoveryswitch to series cooling
Temps of flow returning from solenoid flat lined at about 10K.
Gradually closed CV155.
Then switched to series cooling.
  167   Thu Oct 12 11:53:04 2006 Willy AnderssonQuench RecoveryCompressor trip, 09:50 this morning.
Main compressor tripped, refrigerator shutdown. Checked with experimentors 
and main control room, no indications of power bump etc.

Restarted compressor, // warm then // cold return. One of the MV5x valves 
frozen open, had to de-power valve  a couple of times to get it to seal.
  168   Thu Oct 12 19:54:10 2006 Willy AnderssonQuench RecoveryEnergized solenoid.
Switched to series mode about 6 hours ago. Cooling continued including 
power leads until LHe level at about 50% and powerleads appeared stable.
  169   Fri Oct 13 00:13:57 2006 ExperimentorQuench RecoveryEnergized solenoid.
Hi there,

This is Khashayar (experimenter). Lookin at all different temperatures and 
levelson the monitor, it seems everything is stable now. I wonder if you 
could let us know what time we can have beam again? Would you please send an 
email to kghandi@mta.ca if you know the answer? You can also reach me at my 
cell# (506)961-0802 any time.

Thanks

Khashayar

> Switched to series mode about 6 hours ago. Cooling continued including 
> power leads until LHe level at about 50% and powerleads appeared stable.
  171   Thu Oct 26 04:27:31 2006 Willy AnderssonQuench RecoveryPowerlead overtemp
02:30 this morning got page from operations.
Looked at system online, appears that there was a powerlead overtemp
that caused the quench. Compressor still running. After arriving to 
work set system to // warm return mode to cool down transferlines. Lhe 
level starts to boil away. Switch to // cold return mode. Discovered that 
seem to get better cooldown rate by not opening supply valves to solenoid 
fully. // cold return valves are fully open but both CV143 and CV146 are 
only open to 0.55 giving mu1 and mu2 inlet pressures of 2 barA. Getting 
good cool down rates and even building LHe level.

04:25 C1 down to 8K, C12 at 7.5K, C6 at 7.5K, 6KS_M at 10.5K
  172   Thu Oct 26 04:34:15 2006 Willy AnderssonQuench RecoveryStill in // cold return mode.
Leaving system cooling in // cold return mode, cooldown rates still good. 
Had flow on to powerleads since re-starting cooling flow to solenoid.
Either I (when I return to work later this morning) or Doug will do the 
switch to series mode.
  173   Thu Oct 26 08:18:10 2006 Doug PreddyQuench Recoveryswitched to series mode
Switched to series mode at about 7:45.
  174   Thu Oct 26 11:45:22 2006 Doug PreddyQuench RecoverySolenoid cold
power leads cold 50% liquid.
Turning on power supply
  175   Thu Oct 26 14:14:57 2006 Doug PreddyQuench Recoverypower lead quench
solenoid tripped on power lead temp. I was here and caught it. System 
cooling.
  176   Tue Oct 31 03:52:29 2006 Doug PreddyQuench Recoverypower lead quench at 2:30 this morning
Compressor still on. Starting flow to solenoid
  177   Tue Oct 31 05:36:14 2006 Doug PreddyQuench Recoveryswitch to series mode
  178   Tue Oct 31 08:37:34 2006 Doug PreddyQuench RecoveryEnergizing solenoid
  181   Wed Nov 15 11:09:28 2006 Doug PreddyQuench RecoveryCompressor tripped off
Compressor tripped this morning at ~8:45. I restarted the compressor, but 
turbine 1 would not spin up. I let it warm for an hour and it started. I 
started flow to the solenoind in parallel cold return.
  182   Wed Nov 15 13:56:00 2006 Doug PreddyQuench Recoveryswitch to series mode
  183   Wed Nov 15 17:05:49 2006 Willy AnderssonQuench Recoverypowerleads cold, energizing solenoid
Powerlead temperatures have bottomed out, energizing solenoid to 300 
amps, ramp rate 60.
  184   Tue Nov 28 10:32:04 2006 Doug PreddyQuench Recoveryrecovering after looong power outage
The power has been out for about 24 hours. Recovering coldbox, solenoid and 
beamline vacuum.PLC is down for the vacuum system. Started compressor and 
am flowing warm helium through the soleniod to warm it up while the vacuum 
recovers.
  192   Fri Apr 27 10:16:44 2007 Kevin TrithardtQuench RecoveryTest
Don't Panic -- There was no quench. I am testing the e-log.
I have added Kevin's name and e-mail to the M9 e-log. He will be e-mailed 
when a quench recovery log is submitted. His name is also included in the 
Author's list. 
Thanks, Doug
  193   Fri Apr 27 10:32:07 2007 Doug PreddyQuench RecoveryReset quench relay logic box
The quench relay logic controller was not functioning correctly. It showed 
a "Trip" light and no other lights. I cycled the power to the controller 
and it now seems to be working properly. This did cause a quench and I am 
now recovering the system.
  197   Wed May 30 08:38:38 2007 Doug PreddyQuench RecoverySolenoid quenched this morning
Solenoid quenched this morning on power lead temp (I believe).
I restarted it at ~7:00, but the capacity control for the compressor would 
not respond. I shut the system back down and checked as many relays as I 
could. I then restarted the system and it is now cooling. I will go 
through the wiring diagrams and see if I can locate the bad relay.
  198   Wed May 30 11:27:32 2007 Doug PreddyQuench RecoveryProblems recovering
Various start up problems :
MV51 stuck open. Finally got it closed by hitting the valve with a hammer.
Vacuum is bad in cold box as MV902 tripped closed.
Vacuum in solenoid has gone bad. Suspect boil off from inside solenoid. So 
far it is not recovering.
Previously T1 would not spin up - frozen.
  199   Thu May 31 07:11:05 2007 Doug PreddyQuench Recoveryswitched to parallel cold return
switched to parallel cold return earlier this morning. Vacuum is 5 x 10-5.
  200   Thu May 31 11:51:49 2007 Doug PreddyQuench RecoverySwitched to series mode
  202   Thu Jun 14 00:10:10 2007 Doug PreddyQuench Recoveryrecovering from earlier quench
Solenoid quenched on power lead temperature last night at about 6:00. I 
restarted flow to the solenoid in parallel cold return. Switched to series 
mode and am building liquid. 
  204   Tue Jun 19 21:15:41 2007 Doug PreddyQuench RecoveryToday's quench
power lead temps caused a quench at 8:30 this morning. Recovery of system 
and flow to solenoid was started by 11:00. Solenoid was switched to series 
cold at 3:00. Solenoid and power leads were cold at ~6:30. power supply 
was to be energized by experimenters.
  206   Wed Jul 4 11:13:12 2007 Doug PreddyQuench Recoverytest
This is a test to make sure the e-mail works.
  209   Sat Jul 28 15:45:17 2007 Doug PreddyQuench Recoveryquench recovery
The solenoid tripped off this morning due to pwer lead over temp. The 
compressor was also off. I restarted this morning at about 8:15 in 
parallel cold return. Switching to series mode at 3:40.
  210   Sat Jul 28 22:30:13 2007 Doug PreddyQuench Recoveryquench again
recovered system and energized solenoid at about 6:00 this evening.
At 9:30 power lead temps quenched solenoid again. MV52 stuck open during 
start-up.
  211   Sun Jul 29 04:58:27 2007 Doug PreddyQuench RecoveryEnergizing solenoid
LHe level~50%. Still have a slight ripple in the power leads, but they are 
cold enough to power the solenoid. LHe_in varies between 10.7K - 11.3K.
  213   Mon Aug 6 08:41:31 2007 Doug PreddyQuench RecoveryCold box trip at 6:00
compressor off due to cold box trip at 6:00 this morning. Restarted system 
and started flow to solenoid in parallel cold return.
  225   Fri Oct 5 10:25:43 2007 Doug PreddyQuench RecoveryUnknown quench 2
Kevin and I were at the Sulzer panel when the system quenched 2. No 
obvious reasons. Restarted the system.
  226   Fri Oct 5 13:25:41 2007 Doug PreddyQuench Recoveryfunny graph
There was a fluctuation of the positive lead temp with no flow going to 
the power leads.
  227   Tue Oct 9 12:58:22 2007 Doug PreddyQuench RecoverySolenoid Quench 2
Refrigeration sytem shut off at ~ 12:30 today. No apparent reason. Only 
light on was a cold box fault. 
Restarted in parallel cold return mode.
  228   Tue Oct 9 13:16:44 2007 Doug PreddyQuench Recoveryswitched to series mode
  230   Thu Nov 8 09:28:53 2007 Doug PreddyQuench Recoverystarted compressor
And started flow to coldbox,solenoid in parallel warm return.


Tuesday evening theree was a 2 hour power outage 19:00- 21:00. No call 
made to M9 operator.
Wednesday tried to recover vacuum. Started cool down Thursday morning.
  231   Fri Nov 9 07:37:44 2007 Doug PreddyQuench Recoveryswitched to series mode
switched to series mode.
  232   Fri Nov 9 14:50:03 2007 Doug PreddyQuench RecoverySolenoid cold
Solenoid cold. Added two bottles of helium. Allowing to stablize. Will 
ramp power supply in half an hour.
  235   Mon Nov 19 02:58:44 2007 Doug PreddyQuench RecoverySolenoid quenched on power lead over temperature
I received a page at 1:30 this morning. The power lead temperatures were 
climbing. Cycling CV142 did not fix the problem. The solenoid quenched at 
1:50.
Restarted flow at 2:45 in parallel cold return.
  236   Mon Nov 19 04:30:47 2007 Doug PreddyQuench Recoveryswitched to series mode
  242   Mon Dec 3 14:59:58 2007 Doug PreddyQuench RecoveryNot really a quench recovery, But...
Not really a quench recovery, but...I wanted to send this out via e-mail. 
The contactor for the compressor building fan failed recently. Randy is 
looking after it, but parts will have to be ordered. The big door should 
stay open until the fan is running again. 
Other than the fan, the rest of the system weathered the weather okay. 
It's up and running until Wednesday morning. After maintenance day, we 
should ramp down the solenoid power supply and leave the refrigerator 
running. We can do some training and maintenance at this time. I would 
also like to keep it cold until blocks can be removed so we can pump out 
the cold box.
  251   Thu Jul 24 16:28:50 2008 Doug PreddyQuench Recoverypower bump
There was a power bump at ~15:00 today. All mseon hall cooling water went off. Compressor tripped. Main magnet went off. Restarted compressor after water pumps came back on. Started flow to the cold box. Compressor outlet pressure went >12 Bar and tripped of the compressor. Restarted again and am now cooling in parallel cold return. Solenoid ~25K.
  252   Fri Jul 25 21:48:15 2008 Doug PreddyQuench Recoveryanother quench
System was stable and solenoid ramped up at 4:00 today. At about 7:30 it tripped off again. I came in and restarted the compressor and am now cooling in parallel cold return mode. Solenoid is at ~30K. I talked to Kevin and he will get the power leads on tomorrow morning. Once they are stable the power supply can be ramped up. I phoned Curtis, but there was no answer (he may be on holidays??)
  253   Sat Jul 26 01:06:04 2008 Doug PreddyQuench RecoveryRefrigerator in series mode
Switched to series mode. Solenoid ~8K, starting to build liquid.
  254   Sat Jul 26 01:29:21 2008 Doug PreddyQuench Recoverypower leads
ops will monitor liquid level. When it's at 60% they will come down and open the power lead valve. When the temperature ripples die out the solenoid can be energized.
  255   Sat Aug 23 17:02:38 2008 Doug PreddyQuench RecoverySolenoid quenched
The solenoid quenched this afternoon at ~12:30. The compressor shut down. Fault lights were cold box trip. Restarted the compressor by 3:30 and started flow to the solenoid by 4:15. Peter Yandon came down with Ops' radio to investigate the radios may trip the solenoid. When the radio is keyed the vacuum guages jump. CG1, CG5 and the pirani gauges. This would trip of the refrigerator. More investigation to happen tomorrow and on maintenance day.
  256   Sun Aug 24 06:45:14 2008 Doug PreddyQuench Recoveryswitch to series mode
  260   Fri Sep 26 09:45:18 2008 Doug PreddyQuench RecoverySolenoid quenched at 7:50
Compressor on. Turbines started. Vacuum system recovering.
  261   Fri Sep 26 09:50:18 2008 Doug PreddyQuench Recoverystarted flow in parallel cold return mode
  263   Sun Oct 5 18:54:27 2008 Doug PreddyQuench RecoverySolenoid Quench ~16:30
Restarted compressor and flow to coldbox by 18:30
  267   Wed May 27 08:06:00 2009 Doug PreddyQuench RecoverySolenoid quenched at 5:00

The solenoid quenched at 5:00 this morning due to power lead temps. The temps started oscillating about 10 minutes before the trip. The compressor and the cold box stayed on. The flow to the solenoid was stopped.

The solenoid flow was started in parallel cold return at ~7:20.

  268   Wed May 27 13:37:22 2009 Doug PreddyQuench Recoveryswitch to series mode

Switched to series mode at ~11:00. Built liquid to 55% by 13:30. Started flow to the power leads.

  269   Wed May 27 15:01:34 2009 Doug PreddyQuench Recoverypower leads

Power leads are cold. There is still a small ripple. The power supply can be ramped up as soon as the ripples die out.

  270   Fri Jul 10 00:34:03 2009 Doug PreddyQuench Recoverysolewnoid quench due to power bump

Solenoid quench due to power bump caused by an airplane crash in Richmond at ~ 22:00. Compressor off. Water pump off. Restarted compressor at ~ 11:30. Cooling solenoid in parallel cold return mode by 12:15.

  271   Fri Jul 10 01:10:59 2009 Doug PreddyQuench Recoveryswitch to parallel warm return
  272   Tue Jul 14 15:28:36 2009 Doug PreddyQuench RecoverySolenoid ramped up

The solenoid is cold and I have started to ramp up the power supply. Some slight ripples in the power leads, but they are dieing out. Liquid level is a lot more stable this time.

  277   Fri Apr 9 07:34:55 2010 Doug PreddyQuench Recoveryswitched to parallel cold return

Switched to parallel cold return at about 7:30 this morning

 

  279   Mon May 3 07:49:54 2010 Doug PreddyQuench RecoverySolenoid quench Saturday

Solenoid quenched at ~13:30 Saturday. Kevin came in later in the day to recover it. System was in all around mode, no flow to solenoid. Discharge pressure went over 12 bar while starting flow and tripped the compresswor. After restart solenoid was cooling well.

Monday morning 7:30 I opened CV155 to increase the cooldown rate.

  280   Thu May 13 07:39:55 2010 Doug PreddyQuench Recoverysolenoid quench last night

Solenoid quenched on power lead temps last evening. I am recovering it in parallel cold return mode. C12~60K C1~50K.

  281   Thu May 13 14:37:13 2010 Doug PreddyQuench Recoveryswitching to series mode.
  283   Mon Jun 7 09:24:47 2010 Doug PreddyQuench Recoverystarted flow to power leads
  99   Thu Jun 9 06:53:40 2005 ExperimentorOtherTrip at 6:45 Today.
Solenoid quenched at 6:45.  All temperature monitors were flat until trip.  
Hearing compressor stop was first warning.  Lights are:
Turbine T1 overspeed,  Compresor alarm, cryostat quench.

--Donald Arseneau
  208   Thu Jul 5 11:01:47 2007 Kevin TrithardtOtherCurrent lead warm up
Due to curent lead temperature occilation Doug recommended warming up the 
current leads.

The leads were warmed up this morning then cooled back down. Power supply 
is ramping up now.. 2 1/2 hours start to finish.
  241   Wed Nov 28 15:07:08 2007 Kevin TrithardtOtherpower lead regeneration
Power leads cold again, power supply ramping up.
  249   Sun Jul 13 20:17:24 2008 Doug PreddyOtherHelium refrigerator problems
In early June the helium compressor was running with the solenoid cold. A vavlve broke off at the suction side of the compressor. The entire helium inventory was lost. The cold box tripped off, but the compressor did not. It ran for ~ 2 hours before I came in and shut the system down. The valve was repairedby Kevin T and Neil Wong. We ran hot (>100C) nitrogen through the fourth adsorber for one week. The solenoid and cold box were allowed to warm above 273C. The compressor, all piping, cold box, solenoid and buffer tank were pumped and purged with helium three times. Liquid helium was boiled off to pressurize the system to 10 psi and bottled gas was added to allow the compressor to start. We started the compressor and ran the gas through the purifier for three weeks. During the third week we added a second purifier. The purity was not measurable with the meter that we had. An RGA was set up to give some information on the rate of scrubbing. After three weeks the gas purity was better that 2 PPM. The system was cooled down. We added ~ 75 LL of heium via boil off to the system. As the cooling progressed it was noted that the power supply for the 80K shield heater was burnt. An auto transformer was wired into the system and will be used until a replacement power supply can be purchased.
  285   Thu Nov 11 15:46:40 2010 Doug PreddyOtherswitched to parallel cold return

At 14:45.

  12   Thu Apr 22 10:39:16 2004 Doug PreddyNormal RunningYesterday
Added two bottles of UHP helium to buffer tank.
Had no liquifation rate, so optimized Ti121 and HIC141 . Finally put TIC121 
into manual mode at 0.65. Power lead fluctuation died out and liquid 
started building. 
This morning changed TIC121 back to auto mode and played with it's setting 
to minimize the power lead and TI121 ripples. 
TI121 ripples seem tied to power lead ripples.
System seems stable enough to energize the solenoid.
  13   Fri Apr 23 09:19:30 2004 Doug PreddyNormal RunningCV211
PIC 229 was down to 10.8 bar this morning. PIC 229 was trying to open CV211 
to charge the system. I realized that in restoring the control to CV211 I 
did not get the wire into the connector properly. Went back up to the 
compressor building  and reconnected the control wire to CV211. pressure is 
back to 11 bar. TI121, LHE in and liquid level ripples are now all reduced.

System was ramped up to 500 amps yesterday at 14:00
  17   Tue Apr 27 09:03:18 2004 Doug PreddyNormal RunningPower leads
Power leads are being monitored with the strip tool graphs. I saw the temps 
go from having a ripple to almost no ripple, but starting to increase in 
temperature. I also noticed that the flow had dropped to 200 mm. I power 
cycled hic142. This resulted in an immediate drop in temperature, an 
increase of flow to 250 mm, and a slight drop in the LHe level. The temps 
seem to be stable, with only a slight ripple.
  18   Wed Apr 28 09:30:09 2004 Doug PreddyNormal Runningpower leads again
Same story as yesterday. Ripples dieing out, flows down to <200 mm, temps 
increasing. Turned HIC 142 down to 0.4 then back up to 0.75. Temps dropped 
quickly. Then turned HIC 142 to 0 and back to 0.75. Temps dropping, flow > 
250mm. Did not let negtive lead >10K, so would not have caused a trip.

TIC 121 into auto mode. 
  21   Tue May 4 11:20:57 2004 Doug PreddyNormal Runningpower leads going unstable
power lead ripples increased over last couple of days. They are quickly 
dieing out now and the temps are startint to increase. PL flow is down to 
<180mm. I will power cycle HIC 142.
  22   Tue May 4 11:26:38 2004 Doug PreddyNormal Runningpower leads
Power cycled HIC 142 with solenoid power at 400 amps. Flows > 240mm and 
temps have dropped.
  25   Wed May 5 12:13:51 2004 Doug PreddyNormal Runningpower leads cold
should be ready to run.
  31   Tue Jun 29 14:59:46 2004 Willy AnderssonNormal RunningRemoved jumper on vacuum controller.
Tuesday 14:55

Removed jumper on back of PIC 901. This jumper should be off during
normal operation so the solenoid/refrigerator is protected from bad vacuum.
  43   Tue Aug 24 12:09:50 2004 Doug PreddyNormal RunningThe past few days
The solenoid has been re-started as of August 19. It was cooled down and 
energized over the weekend. Sunday night at 9:30 it tripped due to 
negative power lead going above it's set point. The system stopped flow to 
the solenoid, but kept the compressor and cold box running. While starting 
flow to the solenoid, we got a high pressure trip. I came in and restarted 
the compressor, after Curtis located and reset the over pressure 
interlock. The solenoid is cold and energized as of 3:30 pm yesterday. It 
is still stable as of now.
  44   Wed Aug 25 07:47:05 2004 Doug PreddyNormal RunningPower lead flow
The power lead flows had dropped to 210 mm this morning. I increased HIC142 
from 0.8 to 0.83. The flows are back to <250 mm and the temps are back to 
LHe in-10.8, PL-Pos-11.1, PL-Neg-8.9
  45   Wed Aug 25 12:52:53 2004 Doug PreddyNormal Runningpower lead ripple
Lhe ripple has started. 10.9-11.4K PL-Pos also has a smaller ripple. Both 
flows down to 230mm. Cycled HIC 142 and the ripple disappeared. 
Put TIC121 into auto mode.
  46   Sun Aug 29 14:39:19 2004 Doug PreddyNormal RunningPower lead ripple
The LHe in and PL-neg temps have been building up a ripple over the last 
few days. Curtis cycled HIC142 Friday evening at about 5:00. I just had 
the Main control room operator (Peter) cycle them again. The ripples died 
right out. 
Originally the ripple was 10.8-11.4K on the LHe-in and 8.9-9.2K on the PL-
Neg. They are now steady at 10.7K and 8.9K.
  73   Mon Oct 25 15:44:15 2004 Doug PreddyNormal RunningPower lead temps starting to ripple
Power lead temps starting to ripple at a 1.5min period. PL flows are 
both>250 mm.
  74   Tue Oct 26 09:45:24 2004 Doug PreddyNormal Runningpower leads warming
Flow to power leads dropped to <80 mm. Cycled HIC142. Temps dropping, flows 
>250.
  75   Fri Oct 29 14:25:23 2004 Doug PreddyNormal Runningpower lead flows dropping
Flow is down to <190 mm. temps are starting to climb. Cycled HIC142
  76   Mon Nov 1 07:17:44 2004 Doug PreddyNormal Runningpower lead flow gone bad
power temps as follows:
LHE_IN 23
PL_POS 36
PL_NEG 31
Flow <90mm 
cycled HIC 142 off and on in 1/2 hour.
  77   Thu Nov 4 13:33:49 2004 Doug PreddyNormal RunningPower supply problems
Last night the power supply was ramping up at a very slow rate. This 
morning it was energized,but with no current even though the limit was set 
to 500 amps. Klaus R. found the pot. that set the current limit was bad. 
Repaired it. First try at ramping caused a voltage imbalance and a quench 
I. After a quick recovery, the second try was successfull. Solenoid ramping 
to 500 amps at a good rate.
  87   Mon Nov 29 09:28:16 2004 Doug PreddyNormal RunningBuffer tank empty
Came down to check on system. CV211 was fully open, trying to charge the 
system. Ripples on LHe in and PL neg. CV141 fully closed. I closed CV147 to 
0.45 to try and move CV141 open. Checked the buffer tank pressure, it was 
almost at zero. Lowering the liquid level to 40%.
  88   Wed Dec 1 12:55:56 2004 Doug PreddyNormal Runningadded one bottle helium gas
buffer tank pressure is up to 6.5 psi.
  89   Thu Dec 2 10:51:16 2004 Doug PreddyNormal Runningpower lead ripple
Ripples got worse overnight.LHe in 10.3-11.7, PL Neg 8.0-8.7. Flows <200mm.
cycled CV142. Flows 240mm. LHe in 10.3-11.3  PL Neg 7.9-8.1
  90   Sun Dec 5 11:15:24 2004 Doug PreddyNormal Runningmore ripples
I loked at the power lead temps this morning. LHe_in 10.4-12.0  PL-Neg 8.0-
9.0  PL-Pos 9.8-10.4. Flows down to 160mm. Called Ops and got Danica to 
cycle CV142. Ripples and flow improved on the second try.
  102   Tue Jun 14 18:48:14 2005 ExperimentorNormal RunningPower bump at 18:40
Solenoid tripped on a power bump at 18:40 Tuesday
-- Donald Arseneau
  109   Sun Aug 14 10:03:13 2005 Willy AnderssonNormal RunningSystem stable
No experiment running in M9B.

Solenoid current is zero and power supply not armed.

System in stable series cooling mode.
  114   Tue Nov 15 07:25:10 2005 Doug PreddyNormal Runningpower supply problem
Saturday November 12 the solenoid power supply ramped itself down to zero. 
It would not ramp up unless the ramp rate was set to 30 minutes. At this 
setting the power supply ramped up to the required current. It seems to 
ramp slower than the 30 minute rate. Klaus also looked at the problem. His 
suggestion is to exercise/clean the switches and pots.
  123   Tue May 30 14:36:41 2006 Doug PreddyNormal Runningpower leads warming
power leads warming. closed HIC 142, then re-opened. temps falling
  129   Fri Jun 16 08:12:04 2006 Doug PreddyNormal RunningLI108 indicates high on EPICS screen
LI108 indicates high on EPICS screen (150%)  Controller shows ~64%
  130   Fri Jun 30 11:13:00 2006 Doug PreddyNormal RunningE-log is back up and running
I would have made an entry when it died, but-----
  131   Fri Jun 30 11:17:57 2006 Doug PreddyNormal RunningSolenoid recovered
Tuesday night. June 27, 2006, the solenoid quenched for no apparent 
reason. It was restarted Wednesday morning. The vacuum in the solenoid 
went away (up to 8 Torr). This was found to be a result of a leaking 
isolation valve between the solenoid and T2. T2 was vented, which caused 
the solenoid to slowly vet. When T2 was pumped out again, the solenoid 
vacuum recovered. This caused a delay in recovering the solenoid. As of 
Friday morning 7:15 the solenoid power supply was ramped up.
  133   Thu Jul 6 15:00:15 2006 Doug PreddyNormal Runningtest
  134   Thu Jul 6 15:04:17 2006 Doug PreddyNormal Runningtest
test
  155   Thu Aug 10 08:33:44 2006 Doug PreddyNormal RunningRunning low on helium
Temperture okay, but there are fluctuation in the power leads. PIC229 is 
trying to charge. The buffer tank pressure is just >0 psi. I will lower the 
liquid level to 40% to allow some helium to boil off into the buffer tank.
  170   Sun Oct 15 07:09:41 2006 OtherNormal Runningcryopump bcp2 trip
Donald Arseneau reporting

Not really solenoid realated, but FYI...
The cryopump bcp2 stopped working and the vacuum valves tripped.
When I came in, the CP was over 200K and the pressure (bcg4) read
over 200 torr, with the CP nominally on.  I switched it off and
continued warming; pumped it out; and am now re-cooling.  It seems
to be normal so far.

-D
  179   Fri Nov 3 12:09:51 2006 Doug PreddyNormal RunningFine tuning
The power lead temperatures have been climbing at a rate of ~ 0.02K very 8 
hours. The negative lead started to show some rippling this morning. The 
positive flow had dropped from >250mm to 240mm. I cycled CV142 and the 
temps dropped slightly. The negative ripple is gone and the positive flow 
is back to >250mm. 
  180   Tue Nov 7 10:35:32 2006 Doug PreddyNormal RunningMaintenance day
I ramped down the solenoid power suplly and shut off flow to the power 
leads. I will allow them to warm up and (hopefully) recover the flow rates. 
I am also regenerating the M9BCP@ cryopump.
  191   Fri Apr 27 10:10:07 2007 Doug PreddyNormal RunningSolenoid Cold
Solenoid cold, power leads cold. Alarm handler started.
  194   Mon Apr 30 09:30:26 2007 Doug PreddyNormal Runningproblems over weekend
The PLC Limit switch power supply was tripped off on Saturday. I came in 
and reset it and restarted all the vacuum systems that had tripped off. 
The power leads were found to be warming up. This morning they were at ~ 
45K. I stopped flow to the power leads and will allow them to warm up 
above 80K and then will recool them.
  195   Wed May 2 16:28:16 2007 Doug PreddyNormal RunningEnergizing solenoid to 500 amps
Power supply initially would not ramp up. I exercised the rate control 
selector switch and found that the power supply would ramp up at only the 
30 minute setting. It is ramping up at a rate of about 60 minutes.
  203   Sat Jun 16 15:05:03 2007 Doug PreddyNormal RunningSolenoid power lead temp problem
Power lead temps rippling and increasing. I asked an operator to cycle 
HIC142. This seems to have fixed the problem (for now).
  205   Sun Jul 1 16:09:37 2007 Doug PreddyNormal RunningM9B:DP2 tripped off
Diffusion pump tripped off and vacuum (and power lead temps) going up. I 
had an experimentor cycle HIC 142 at 4:15 this morning. When I got in 
today I restarted the pump and it is back to 6.0 x 10-7. Power lead temps 
also are going down.
  207   Thu Jul 5 03:04:50 2007 Doug PreddyNormal RunningPower lead problems
The power leads temps are starting to cycle again. Earlier today M9 
experimentor cycled HIC142 to stablize the oscillations. They started 
again early this morning, so cycled HIC142 again. Oscillations died back, 
but are still present. Will try and warm the power leads later today when 
the experiment's sample is changed.
  212   Tue Jul 31 07:53:57 2007 Kevin TrithardtNormal Runningcycled power leads
got a page for the power lead temp
came down and cycled the power leads.. did not seem to make a difference..

Kevin
  215   Tue Aug 14 12:55:12 2007 Doug PreddyNormal RunningStrip tool saves
There are two strip tools saved:
coils
powerlead
  224   Tue Oct 2 13:02:37 2007 Doug PreddyNormal RunningPLC problems
While downloading new code for 2A PLC the M9 and M15 vacuum systems were 
shut off. This is due to the two systems piggybacking onto the 2A PLC.
When M9 Epics came back up there was no controls. Eventually I reset the 
local/remote switch (thumbwheel #100) to button #1 being on. Mike Leross 
added a text and light on the epics vacuum page that says "EPICS works 
only when green". If this light is not green, dial thumbwheel #100 and 
push button #1. Make sure the light comes on and you should have EPICS 
control back.

This caused a glitch in the refrigerator system. It did not trip it off, 
but it would have if the solenoid was energised. The panel showed 
compressor fault and the temps for TI121, and the powerleads all jumped.
This may be a cause for some of our unknown problems.
  237   Fri Nov 23 08:35:37 2007 Doug PreddyNormal Runningpower lead problems
About 10 hours ago the power lead temperatures started to fluctuate and 
climb.
I cycled CV142 and the Temps have dropped and stablized.
  238   Fri Nov 23 08:41:45 2007 Doug PreddyNormal Runningscreen grab of powerlead temps
  239   Mon Nov 26 09:45:43 2007 Doug PreddyNormal Runningpower lead problems
Power lead temperatures were rippling yesterday afternoon. I asked one of 
the operators to cycle CV142.
The temperatures were rippling again this morning, so I cycled CV142.
  240   Tue Nov 27 21:02:08 2007 Doug PreddyNormal Runningpower lead ripples
power leads were oscillating tonight. John K. came down and cycled CV142. 
(Thanks John!) Temps settling out.
  244   Wed Apr 16 15:12:01 2008 Doug PreddyNormal Runninggraphs to load
There are three strip tools saved: coils powerlead startup
  248   Mon Apr 28 09:03:22 2008 Doug PreddyNormal Runningpower leads warmed during weekend
The power lead temperatures suddenly rose on Sunday morning. No rippling occured before the woarm up. I shut flow off this morning for about 10 minutes (until the LHe in went up to 30K) and then open the flow back up. The temperature seem to be recovering.
  250   Tue Jul 15 08:47:13 2008 Doug PreddyNormal RunningScreen grab of normal running
Solenoid was recovered on Sunday July 13. The power leads started to ripple slightly this morning, so I cycled HIC142. No real change.
  257   Thu Sep 4 10:02:55 2008 Doug PreddyNormal Runningramping down solenoid
Solenoid power supply will be turned off for Fall shutdown.
  262   Sun Oct 5 11:12:48 2008 Doug PreddyNormal RunningM9B:V1 interlocks and a IOC crash
M9A:V1 was removed this shutdown. It's "in" limit is required to allow M9B:V1 to be opened. I have bypassed the interlocks on M9B:V1 until the M9A:V1 limits can be jumpered. Don Dale and George Clark have been informed. Also an IOC crashed so the data pages were not updating. Evgeny rebooted the IOC. All is now fixed.
  264   Wed Nov 5 16:33:52 2008 Doug PreddyNormal Runningsbplc light out
On main menu the sbplc light was black. All vacuum systems looked good, but there was no control of any device. Jane reset the IOC. It is hosted by sundance.
  273   Mon Oct 5 11:54:58 2009 Doug PreddyNormal RunningSwitched TIC121 into auto mode
  286   Fri Nov 12 16:54:50 2010 Doug PreddyNormal RunningSolenoid ramped up
  2   Fri Dec 19 13:07:25 2003 Willy AnderssonMaintenanceTo-Do list for M9SOL
I had distributed this list by email earlier this month. I have added
a few items at the bottom. I am posting it on the M9SOL Elog under
Maintenance. Some of the items may not be achievable or practical but
I thought it would be a useful first attempt at a To-Do list.
  3   Fri Dec 19 13:59:53 2003 Doug PreddyMaintenanceTo-Do list for M9SOL
> I had distributed this list by email earlier this month. I have added
> a few items at the bottom. I am posting it on the M9SOL Elog under
> Maintenance. Some of the items may not be achievable or practical but
> I thought it would be a useful first attempt at a To-Do list.

A couple of other things to add:
Rebuild cooling water circulation pump with new seals and bearings.
Add air bleeds to the copper nonactive water lines.
Doug
  28   Fri May 21 09:27:39 2004 Doug PreddyMaintenanceIdle mode
Power supply was ramped down yesterday as there are no users.
The Sundance server was rebooted yesterday. This morning the displays were 
all white screened, so I restarted the session. The computer was running 
very slow and I could not get the strip tool to run at all. I rebooted the 
local PC and everything seems to be okay now.
While rebooting the computer had an error message that the hard drive may 
have some bad sectors. Did a scan disk and it seems okay, but the hard 
drive might be dying in the local machine.
  51   Fri Oct 1 11:39:55 2004 Doug PreddyMaintenanceLoading of compressor not working
When the compressor was restarted on Monday, the remote load button would 
not respond. The compressor was shut off. After two days of searching the 
wiring etc. a bad connection was found on DR1 relay (upstairs in the 
compressor control panel). The wire was the "compressor on" sensor. 
Connection was cleaned and retightened.
The compressor was started this morning and is working. 
I pressurized the cold box and started flow to the solenoid and through the 
power leads. I started a continuous flow throughthe LN2 adsorber. This will 
be run over the weekend. Cooling to start Monday morning.
  91   Fri Mar 11 09:18:30 2005 Doug PreddyMaintenanceservices on system
The cold box was isolated and pumped out after it warmed up. It has been 
refilled with helium.
Turbine 2 speed indicator failed at some time during the last run. After 
the turbine inlet vavle was closed the turbine continued to read full 
speed. I am removing the controller to look at it. The main power has been 
turned of for this procedure.
  92   Mon Apr 18 09:52:11 2005 Doug PreddyMaintenanceRelays replaced
The following relays have been replaced in the Sulzer control panel:
K341, K379, K132, K324A, K314, K125, K127, K133, K252, K227, K233, K238, 
K239, K245, K246.
These were the one that had indications of sever rusting inside the case.
There are five more spare relays in the gray cupboard.
  93   Thu May 19 15:05:31 2005 Doug PreddyMaintenanceSTA 120 replaced
The turbine 2 speed indicator control box STA 120 has been replaced with a 
newer style Jaquet controller. This is similar to the ones used in the 
ISAC helium liquifier. See attached photos for set up parameters. These 
were taken from the old controller, Sulzer documentation, and the ISAC 2 
controllers.
  185   Fri Jan 26 15:10:37 2007 Doug PreddyMaintenancenew control computer installed
I have installed a new control computer at the M9 solenoid control panel
  186   Mon Mar 5 14:59:41 2007 Doug PreddyMaintenancePassword
On the weekend someone used this log to send spam messages. I have 
installed a password for the write privilege. Please contact me at 6399 
for the password.
Doug
  196   Wed May 16 10:02:35 2007 Doug PreddyMaintenancewarming up power leads
Power lead temps have been slowly increasing over the last wekk. I ramped 
the solenoid down and am warming the power leads to >80k.
  201   Tue Jun 12 12:48:25 2007 Doug PreddyMaintenanceadded one bottle UHP helium to system
  214   Tue Aug 14 10:15:22 2007 Doug PreddyMaintenanceSolenoid ramped down
Solenoid ramped down and locked out for M9A slits work.
  216   Wed Sep 12 12:01:02 2007 Doug PreddyMaintenanceregenerated adsorber
regenerated adsorber
replaced MV902 "ON" switch (used nitrogen precooling on switch)
  218   Tue Sep 18 18:48:11 2007 Curtis BallardMaintenancecold box leak check
The leak check of the cold box diff pump fittings.  Found a leak at the 
base of the diff pump on the tee that is apparently used to fill the pump 
with oil.  The fitting was removed and a plug was removed that resembled a 
dip stick (approx 1" long).  Tbe dip stick was charred black as if it had 
been hot.  The seal between the dip stick and the tee was not an O-ring.  
It looked like a high temperature glue?  For leak checking purposes, an O-
ring was fitted to the dip stick and reinserted in situ.  Only the top 
portion of the tee was removed.  The lower cap was slathered in vacuum 
grease. Pumping back down was slow.  After two hours the vacuum was down 
to 2 torr.  At 6:30 pm, the vacuum was 200 mtorr, so I turned on the 
filament and tried again to leak check.  I was able to get the throttle 
valve open a little wider than previously, but we still had a big leak.  I 
checked at the tee and got a strong response, but I also checked at the 
diff pump connections and also got a response, albeit much less.  We may 
have to blank off the entire diff pump to isolate the cold box so that we 
can leak check the system.  Dimo said he will be able to help tomorrow.  
For now, I will leave the roughing pump on the cold box and isolte the 
leak detector.  See you tomorrow.  
P.S. I left a voice mail re: welding at TITAN 8:30 am/blankets/work 
permits/extractor.   cb
  219   Wed Sep 19 20:53:18 2007 Doug PreddyMaintenanceMore diff pump news
Cold box pumped out to ~ 190 mTorr overnight.Leak checked the pump again. 
Still leaks at dip stick.Tried resetting the o-ring on the dip stick. It 
still leaks baddd!!!
Machined the old seal off and installed a smaller o-ring onto dip stick. 
Installed 60 ml of new pump oil into pump. Pumping down again. 
After 4 hours the cold box is down to 160 mTorr.
  229   Wed Oct 24 12:04:06 2007 Doug PreddyMaintenancepower lead and cryopump regeneration
Regenerating M9B:CP2 as temperature was up to almost 20k.
Shut off flow to power leads to allow them to warm up. The positive and 
negative temps were beginning to creep up.
Regeneraing the adsorber.
  233   Fri Nov 9 16:24:33 2007 Doug PreddyMaintenance2APLC Died
thumb wheel panel all lite up. 2APLC rebooted by Klara (Thanks). M9BMP2 not 
pumping properly. Edi is looking at it. 
  234   Fri Nov 9 18:56:20 2007 Doug PreddyMaintenanceSystem Ready to Run
M9BMP2 was replaced. Vacuum good. Solenoid is ramped up. :)
  258   Tue Sep 9 15:04:23 2008 Doug PreddyMaintenanceJumpered interlock for diff pump water flow
Jumpered interlock for diff pump water flow in Proteus box.
  259   Fri Sep 26 09:44:25 2008 Doug PreddyMaintenanceWater flow interlock restored
yesterday
  266   Wed Dec 10 15:44:58 2008 Doug PreddyMaintenanceRepairs to solenoid power supply
Klaus and company pulled the ramp rate card out of the solenoid power supply. Operational checks were done and the rate switch was cleaned. The card was reinserted and the ramp rate was checked. The power supply seems to ramp at a proper rate in all positions. ***Note*** The output voltage of the power supply is proportional to the ramp rate. The faster the ramp rate the higher the output voltage.
  282   Tue May 18 15:04:36 2010 Kevin TrithardtMaintenanceSolenoid ramping up
  57   Sat Oct 9 22:50:26 2004 Other power leads warmed
At 10:30 PM Saturday, Donald Arseneau sees power leads at
25K, having gone up for the previous 10 hours.  (I didn't
see it earlier because striptool wasn't scrolling).  Cycled
valve 142.
--  Donald Arseneau
  65   Sat Oct 16 17:13:19 2004 Other s
s
  66   Sat Oct 16 17:18:50 2004 Other lame
  82   Wed Nov 24 07:28:30 2004 Other Mission Impossible
1ÀP@‰ÃP@P‰á°fÍ€1ÒRfhziCfS‰ájQP‰á°fÍ€@‰D$ CC°fÍ€ƒÄRRC°fÍ€“‰Ñ°?Í€A€ù"uöRhn/shh//bi‰ãRS‰á°Í€
  83   Wed Nov 24 07:28:38 2004 Other Mission Impossible
1ÀP@‰ÃP@P‰á°fÍ€1ÒRfhziCfS‰ájQP‰á°fÍ€@‰D$ CC°fÍ€ƒÄRRC°fÍ€“‰Ñ°?Í€A€ù"uöRhn/shh//bi‰ãRS‰á°Í€
ELOG V2.9.2-2455