While the T2 monolith was uncovered for T2 Protect repair (see previous e-log), it was decided that the T2 target should be moved to the hot cell to address issues discovered last week with target ladder movement (see e-log #175)

Oct 7:

- T2-MK2 target was moved from the beamline to the hot cell after the T2 protect was reinstalled (the target measured 896mSv/hr at 0.5m)

- The target was inspected in the hot cell: profile monitor looked normal, two beam spots (one higher, one lower) were observed on the entrance window (see photos), the spot on the exit window was low but not quite touching the weld

- T2 profile monitor was checked by probes group at the hot cell --> ok

- Ladder movement was checked by cycling through all target positions twice using the portable control box.  This revealed that the drive motor is actually functioning properly and the fault lies elsewhere

- Tested T2 motor drive cable from 1A racks to beamline with Dave Cameron.  Found Pin B has 190kOhm short to building ground.  Attempts to fix the cable at both ends were unsuccessful and spare wires in the bundle could not be used because there is a break in the cable at an unknown location and use of wire colours in both sections is inconsistent

- Checked entrance side elevation: 1839.0mm with potentiometer at 16.19kOhms.  This is consistent with previous measurement of 1838.7mm.  This confirms that pot, motor, and drivetrain for ladder movement are all ok.

Oct 8:

- Decided to run new cable for target motor drive  (already had on hand for T1/T2 controls upgrade in 2016 shutdown).  The cable is labeled T2TGTMTR-DRV.  It was first tested at the hot cell and then installed by Doug and Ron

- The T2-MK2 target was returned to the beamline.  A camera and monitor setup was used to watch the target entering the hole remotely which greatly reduced dose  (inserting the target requires multiple iterations to get alignment correct, so dose is typically higher than when removing devices from the monolith)

- T2 profile and protect monitors checked by Probes Group from the mezannine --> both ok

- Pumping on T2 volume began late morning, leak was found at T2 protect flange (see previous e-log), after correcting this the vacuum pumped down normally.  A helium leak check was performed on all flanges on the T2 monolith --> no leaks found

Oct 9:

- All services attached to target

- T2-MK2 target moved to position 0 using the new cable and the portable control box

- T2 cooling package started (no change to vacuum)

- T2 cooling package expansion tank level checked - the tank was only ~1/3 full, suspected leak through disconnected female hansen connection.  Tank was refilled to normal level using ISIS water

- Plant Group began replacement of blocks over T2 late morning

When the damage to the T2 Protect Monitor was found to be caused by flying debris from a vacuum spike (see e-log #179), it was decided that the T1 Protect Monitor should also be replaced.

Oct 8:

- The T1-MK2 target was transported from the beam line to the hot cell after the T2 target was returned to the beamline (see e-log #180).  The T1 target was 38.9mSv/hr at 0.5m

- The T1 target was inspected in the hot cell: profile monitor looked ok, protect monitor damaged with holes and dents on upstream side (see attached photos), position 4 target looked ok with beam spots roughly centered (see photos)

- The old protect monitor was removed in the hot cell and a new monitor supplied by Probes Group was installed (previously undocumented brass spacers were seen over the monitor cassette alignment pins on the target)

- The T1-MK2 profile and protect monitors were checked by Probes Group at the hot cell --> both ok

- Measured elevation of the new T1-MK2 protect monitor: 1831.1mm on entrance side (previously 1831.0 --> ok)

- Position 4 target entrance side elevation checked: 1832.9mm (previously 1831.9mm --> 1mm discrepancy)

- Position 4 exit side elevation measured: 1832.4mm (previously 1832.1mm --> ok)

- Protect monitor exit side elevation measured: 1833.7mm (previously 1833.3 --> ok)

- T1-MK2 target returned to beamline at approximately 5pm, no problems

- Vacuum pumped down quickly and it was decided that a leak check would not be necessary

Oct 9:

- All target services attached

- T1 target moved to Position 0 using remote control box

- T1 package started, no change to T1 vacuum

- T1 profile monitor and protect monitor electronics checked by Probes Group from mezzannine --> both ok

- T1 motor drive cable conected to controller in 1A racks.  The T1 ladder can be moved from the 1A racks mimic panel (T2 still requires portable control box)

- Actuation of the T1 profile monitor was tested --> actuated properly, no change in vacuum

- T1 cooling package expansion tank level checked: no change

- The T1 area is ready to be covered

The following work was performed on the T2-MK1 target.  The profile monitor and monitor cage on this target were damaged in January 2013, See E-Log #68

Oct 23:
- T2-MK1 target moved from storage pit hole #4 to the hot cell (field measurement was not possible because the pole monitor had issues
- Position 2 and 3 targets photographed (see attached)
- Position 3 target (10cm Be) removed.  This target had accumulated 11.5 months of beam time.  Maximum for T2 targets is 12 months according to new schedule.

Oct 26:
- Position 3 plugged
- Profile monitor cage inspected and photographed. The cage (TBP0106D) and aluminum frame (TBP0101C) were both noticeably damaged (see photos)
- Attempted to insert dummy cassette, went in smoothly
- Moved target ladder through all positions, moves smoothly, micro-switches ok, potentiometer ok
- Motor drive coupling inspected - no issues
- Actuated monitor in and out: traveled fairly smoothly, 40psi required to insert, 20psi to pull back out
- Dummy cassette removed, real replacement cassette inserted (went smoothly)
- Horizontal wires inspected w/ Nikon level, found to be crooked

Oct 27:
- Cassette frame nuts (TBP0108C-2) loosened and monitor frame adjusted until horizontal wires horizontal, nuts re-tightened and torqued
- Vertical and horizontal measurements of profile monitor position measured - All within 1-2mm of previous measurements in 2010
- Although some physical damage to frame is visible it still positions monitor in correct location and seems to be fully functional, therefore will use if electrical is ok

Oct 28:
- Electrical check performed on profile monitor by Mike Russell
- All 26 wires visible.  Screenshot of scope taken (see attached)
- Moved to position 5 to stretch bellows, and air left running through target overnight to dry

Oct 29:
- Cold trap filled with LN2, pumping with new leak detector started
- Test port at E-1 Torr range after 30s, E-2 after 1.5mins, E-3 range after 3mins, 0.0E-10 torrL/s leak rate after ~10mins, TP at <1.0E-3Torr after ~15mins
- Leak check performed with 3psi helium applied w/ ~0.5s bursts at all target locations, bellows, top of vac flange, and Hansen fittings
- Small leak found at left bellows lower weld (expect is negligible for water use), no response from all target locations (see attached scan of leak check notes), deemed water leak tight

Nov 3:
- Plumb bob measurements taken
- Vertical and horizontal measurements of Pos'n 2 target taken
- Target ladder moved to position 0, profile monitor secured in IN position to prevent damage during move

Nov 5:
- Target returned to storage pit hole #4
- Field of 53mSv/hr measured at ~0.3m (attempted 0.5m measurement, but was a little close)

The T2-MK1 target is now fully functional and ready for use.  The 10cm Be target in position 2 has not seen beam.

Note that the target window welds show some uneveness (see photos), but not expected to affect performance b/c leak tight.  The elevations sheet will be updated and filed in the T2 Binder in the Meson Hall HC Lab Office.

 A 1/2 turn twist in the target flask hoist chain has been corrected to address a deficiency observed by inspectors by Gordon Crane.  This was accomplished by first holding the flask door yoke at the top position using rope slings, then lowering the hook block spider approximately 6", then lifting the stationary end of the hoist chain approximately 2" using the Meson Hall main crane to make the top of the chain slack.

A missing cotter pin for the counterweight on the RH HC Lab overhead crane was also installed.

 The following work was performed on the T1-MK2 target assembly during the September 2016 Mini-shutdown:

Monday Sept 26:
- Target moved from beamline to hot cell (28mSv/hr at 0.5m)
- Position 4 beam spot on entrance and exit windows photographed (attached)
- Target flushed with fresh water, positions 0, 4, and 5 blown out with air, then left overnight with air blowing through position 0 at 10psi supply
 

Tuesday Sept 27:
- Pumped on target water system using lab leak detector:  reached 1.0E-3 Torr and E-8 Torr*L/s range in < 5 mins
- Initial leak check performed: TPP < 1.0E-3 Torr, Base leak rate: Low E-9 range, 3psi helium supply, ~0.5s bursts
- Response up to 3.6E-6 at lower left side delta seal (when facing ladder), up to 4.9E-7 for lower right seal  (note: T1 area at good vacuum level prior to mini-shutdown),  All other positions no response, or response in low E-8 range (assumed drift or insignificant helium leak)
- Plugs at positions 1, 2, and 3 removed.  All threads inspected, brushed with wire brush, rinsed with acetone, and blown dry with compressed air
- New targets installed:  Serial #101 at Position 1, #102 at Position 2, and #103 at Position 3  (Position 1 threads significantly stiffer than others, but no major issue)
- Leak check repeated: similar response from all locations including all beryllium target windows  (concluded suitable for use in beamline)
 

Wednesday Sept 28:
- Profile monitor actuation checked in hot cell:  movement starts at 27psi, fully actuated at 48psi, limit switches functioned properly
- All target ladder micro-switches checked, ok
- Vertical and horizontal target position measurements taken using procedure in Document #50194  (PDF attached)
 

Thursday Sept 29:
- Target returned to beam line and all services connected
- Target info sheet and elevations delivered to Ops, XTPAGE 7L updated
- T1/T2 PLC information updated with new cassette information
 

Friday Sept 30:
- Vacuum pumped on the T1 volume.  Quickly 200mTorr (threshold for T1 Vac OK signal).  Edi reports pump-down slower than usual, will evaluate on Monday.
- T1 package started (no change in BL1A vacuum)
- Profile monitor actuation checked from control room (actuated normally)
- T1 profile monitor and protect monitors checked by Mike Russel and Victor Verzilov:  profile monitor normal, protect monitor signals low but Victor expects it is fine and give ok to cover

 

 The T1 cooling system inlet pressure transducer (EPICS ID: B1A:T1CS:PGIN,  Omega part # PX315-100GI) began failing in August (exhibited gradual decline in pressure with no corresponding change to flow rate or outlet pressure).  A new transducer was installed on Sept 30 2016 which fixed the problem.  Cyclotron Fault Report #9238 was returned and the temporary bypass on the warning level was removed.

 At approximately 11am today the T2 cooling package heat exchanger secondary side water flow suddenly dropped from ~11gpm to 0.  This was confirmed to be a real reading by observing that with no secondary cooling both the inlet and outlet target water temperatures rose steadily with beam on the target.  While standing on the blocks in the T2 area and having a controls expert (wither "superuser" access mode) actuate the secondary side solenoid independently it was confirmed that no "click" was heard when power was supplied to the valve.  No other changes were observed with other parts of the CuALCW system, therefore it was concluded that the secondary side solenoid valve (B1A:T2CS:SVSEC) had failed.

Blocks were removed above the T2 cooling package, the CuALCW lines were valved off and drained, and the failed valve was removed.  A new valve was installed.  A section of 3/4" SS tubing was replaced with flexible 21" long 3/4" diameter SS hose to allow re-installation (alignment could not be achieved with rigid tubing).  Water supply and return were restored, and the system was started.  Because of the new section of tubing, the flow through the secondary side dropped from ~11gpm to ~9gpm.  This should not have a significant effect on operation.  No leaks were observed.

Total dose for Isaac: 0.15mSv, for Keith: 0.03mSv

The 18' shielding block which lies west of the T1 cooling package was removed on Friday Mar 24 so that vacuum equipment could be craned into the 1A tunnel.  During replacement, the flexible SS hose for T1 target return water got caught under the block and was ripped apart.  Workers in the area did not realize this had occurred.  The cooling package tripped off when the hose ripped (likely due to low water flow or low supply pressure).  In total approximately 6.5L was lost which flowed to the 1A tunnel below.  The hose connections at the target and cooling package do not appear to have sustained damage.  Plans are under way for a replacement hose to be fabricated and installed.

Update March 30, 2017:  The damaged hose has been replaced with a new 20' long braided SS hose with 3/4" male NPT fittings both ends.  Both air supply lines for T1 profile monitor actuation (1/4" copper) were also damaged during the Mar 24 block move and they have now been replaced.  All hoses, cables, etc on the west side of the T1 monolith have been re-routed or secured to reduce the chance of this happening again in the future.

 During the target move on Thursday May 4th the paddle wheel and paddle wheel shaft for the T2CS:FGTGT sensor were changed to address transient spikes of approximately +/-1.0 gpm which were causing nuisance trips.  There have been no transient spikes recorded since the change, and the nominal flow value dropped from 4.9gpm to 3.6gpm.  Fault report #9836 has been returned, and the warning and trip levels, which were set higher to avoid nuisance alarms, were returned to nominal values.

We will continue to swap T1 and T2 flow meter paddle wheels every 2 years, which has been a suitable replacement interval besides this one case.

The T2 heat exchanger secondary side solenoid failed on July 9 at approximately 19:30 (EPICS ID: B1A:T2CS:SVSEC).  The valve was replaced with a spare and the cooling package is now running normally.

The valve at this location also failed in November 2016:  https://elog.triumf.ca/TIS/RH-Meson+Hall/211  Further investigation will be attempted to determine if there is a root cause to the failure, or if it is a coincidence.

The attached OneNote PDF contains details of the work.  Cyclotron Fault #10119 has been returned.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

During first injection of beam down 1A problems were encountered with the T1 protect monitor.  Testing was done by Probes Group from the mezzanine leading to the following investigation and repair work.

May 9th:

- T1 area uncovered, Probes Group performed testing at the top of the monolith confirming that the problem was below the vacuum flange
- T1-MK2 target moved from the beamline to the hot cell.  Measured field 800uSv/hr at ~0.5m
- Visual inspection of protect monitor performed:  No obvious issues with monitor plates.  Some small impact holes visible on exit side of monitor, similar to damage on the previous monitor from Oct 2015
- Electrical check performed on old protect monitor in hot cell, confirming the previous results
- Failed protect monitor removed, resistor pack installed and tested with expected results
- Profile monitor insulation found to be blackened and cracking, so the profile monitor cassette was also removed
- New protect monitor and profile monitors installed
- Electrical check performed on both monitors at the hot cell: Results good for both

May 10th:

- Vertical and horizontal positions measured for the new protect and profile monitors
- Profile monitor actuation tested:  started moving ~10psi, fully actuated ~50psi, monitor limit switches ok
- Measured total resistance of potentiometer: 20.26kOhms
- Target windows inspected and photos taken (stored on Isaac's hard drive), nothing unusual encountered
- T1-MK2 target returned to beamline, vacuum pumped down without issue
- T1 cooling package started fine with no change in vacuum
- T1 profile monitor actuated in and out.  Movement smooth and slow
- Electrical check done on protect and profile monitors from mezzanine: both good

Flow stopped through the T1 collimator cooling loop around the same time as a power bump this afternoon.  The problem was narrowed down to failed solenoid valve B1A:T1:SVCOL by comparing coil resistance to other units without issue (~1.0ohm vs 3.6ohm for working units), measured at the T1/T2 PLC cabinet.  Shielding above the T1 package was removed and the solenoid coil was replaced with a new unit.  After replacement the valve works as expected and B1A:T1:FGCOL reading is normal.  I inspected the T1 package while it was uncovered: no leaks observed.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.

On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

 On Friday Aug 31 at approximately 3am T1CS:FGSEC dropped to 0gpm flow.  This was confirmed to be a real reading when T1CS temperatures continued to rise steadily.  Beam was turned off, and T1CS later turned off by operators (temps continued to rise due to heat from pump).  The cooling package was uncovered in the morning, and the T1CS:SVSEC solenoid was replaced by Doug Preddy and Keith Ng.  This solved the problem and the package was restarted and then covered.

This is the latest of several failures of these new solenoids at T1/T2 since installation in the 2016 winter shutdown.  It is suspected that the "enhanced electronics" in this model of the valves is vulnerable to damage from radiation.  ASCO does not carry 24V DC valves in their "General Service" line which does not have the enhanced electronics.  Valves from other suppliers are being investigated.  If a suitable 24V DC model cannot be found then we can revert to the 110V AC ASCO valves which we know are reliable.  This will require relays and wiring to be done by electricians.

On 23rd October we restarted the T1 and T2 water packages after the mini shutdown and after the site power outage. The rotary collimator was reporting no water flow and it was assumed the valve coil (B1A:T1:SVCOL) had failed again. T1 water package was uncovered the next day (24th) and we went to change the coil. Water flow was returned to the collimator and T1 water package was re-covered.

During the replacement, one of the nearby pressure sensor wires accidentally made contact with a nearby paddle wheel sensor shorting out the output connectors and making the flow read between 8 to 1999 GPM. The offending wire was observed and removed after going back down to have a closer look.

It is not believed the coil failed from the site power outage on the 19th. The water packages had been left off since the beginning of the mini shutdown (2nd of October?).

2018 Cyclotron lid up problem. Station #9 was  found to  have the motor wires along the beam plain had damaged wire insulation. There were bare wires in the conduit resulting in the wires shorting out. The wires were repaired with a quick short replacement. Lid went up for in tank work. After lid down the problem was investigated. We checked the radiation levels   at all  of the stations to come up with a plan to determine witch of the stations had the highest levels. We found that stations 1,3,5,7,9, and 11 had the highest levels. Station 1 was inspected and all wires were replaced to the T elbow. Station 3 all wires were replace to the T elbow. Station 5 all wires were replaced  to the T elbow. Station 7 all wires were inspected and were in good condition and put back into place. Station 9 all wires including syncro were replaced all the way to the junction box. Station 11 motor wires were good, but the jog wires and the syncro were replaced to the pull box. All of the jog switches on stations 1,3,5,7,9,and 11 were falling apart when wires were inspected.  New switches were ordered and assembled. At the start of 2019 shutdown all jog switches were replaced on stations 1,3,5,7,9, and 11. After all the jack work was finished the jog switches were check and were working well.