The water in the South TNF Holding Tank was released today under work permit #C2018-08-28-3.  The tank contained approximately 100L from the T1 cooling package (drained Aug 16, 2018) and 350L from the 500MeV system (drained April 23, 2018 by Steve Sapriken).

On Wednesday August 15th the T1 cooling package tripped off, and could not be restarted.  On the EPICS screen the pump device went into an immediate time-out each time we attempted to turn it on.  An electrician was involved in the debugging, and issues were found with the 480V supply.  The breaker switch in the 480V supply panel was found to be damaged, and was replaced.  The relay in the contactor box for the pump motor thermal protection was also replaced.  These new parts did not solve the problem.  Resistance measurements between the phases of the pump motor revealed ~10ohms between phases A-B, but ~250ohms from B-C and A-C (measured from inside the contactor box).  The T2 pump and a spare replacement pump were also tested, and had ~10ohms between all phases.  It was therefore concluded that the pump motor, or possibly the wiring between the contactor and motor had failed.  The T1 cooling package was uncovered and drained by the end of the day.

On August 16th the old pump was disconnected and de-wired.  Damaged wire insulation on the pump motor was found, which likely caused a short, leading to failure of the pump motor, and cascading failures of the electrical system and possibly also the control PLC.  A spare pump (Chempump GB-3K-1S) was installed with custom cut gaskets.  The pump was re-wired by an electrician.  The cooling system was re-filled, and start-up was attempted, however there were remaining issues with the electrical system.  By the end of the day various testing by Controls Group and Electrical Group determined that the pump motor contactor also had failed and required replacing.

On August 17th the contactor and over-current protector were replaced by an electrician.  When the breaker was switched on the pump unexpectedly started immediately.  This was found to be caused by the control system drive signal module being faulty causing the output to be stuck on.  Failure of this module may have been caused by excessive current draw due to the damaged contactor.  The failed module was replaced, and the PLC and IOC were both restarted.  After this the system worked normally.  The pump was inspected by Maico Dalle Valle, no leaks were observed and operation seemed normal.  Shielding above the cooling package was replaced, and BL1A was restarted.  Cyclotron Fault #11527 was returned.

A new spare contactor and over-current protector have been ordered (Allied Electronics LC1D09BD and LRD12, Requisition #1037940).  These will be labeled and given to the Electrical Group.  A new spare T1 / T2 pump has also been ordered (Chempump GB-3K-1S, Requisition #1038041) which will be stored in the RH Meson Hall Hot Cell Lab Tool Port Boot Box Area.

A data sheet and quote for the replacement pump are attached.

On Saturday July 21st the T1 Collimator solenoid valve would not re-open after tripping closed due to a (planned) loss of vacuum.  Shielding was removed and the valve solenoid was replaced which fixed the problem.  The original valve body was left in place.

This is the second time the solenoid at this location failed (See E-Log #239), and the latest of several failures of these new 24VDC solenoids at the T1 and T2 cooling packages.  Clearly they are not reliable in this radiation environment, so I will investigate an alternative to be installed in the 2018 Fall Shutdown or 2019 Winter Shutdown.

Cyclotron E-Fault #11420 has been returned.

 The T1 cooling package outlet water pressure sensor (B1A:T1CS:PGOUT) was replaced this morning with a new unit.  Over the last few months the sensor readout had been steadily decreasing (from around 20psi to 4psi) with no corresponding decrease in inlet pressure, and no change in the circuit flowrate (this is how these pressure sensors typically fail).

After installing the new sensor the readback in EPICS returned to the expected value.  Although there are no alarms or trips associated with this signal, it is still useful as a diagnostic tool.

Flow stopped through the T1 collimator cooling loop around the same time as a power bump this afternoon.  The problem was narrowed down to failed solenoid valve B1A:T1:SVCOL by comparing coil resistance to other units without issue (~1.0ohm vs 3.6ohm for working units), measured at the T1/T2 PLC cabinet.  Shielding above the T1 package was removed and the solenoid coil was replaced with a new unit.  After replacement the valve works as expected and B1A:T1:FGCOL reading is normal.  I inspected the T1 package while it was uncovered: no leaks observed.

During first injection of beam down 1A problems were encountered with the T1 protect monitor.  Testing was done by Probes Group from the mezzanine leading to the following investigation and repair work.

May 9th:

- T1 area uncovered, Probes Group performed testing at the top of the monolith confirming that the problem was below the vacuum flange
- T1-MK2 target moved from the beamline to the hot cell.  Measured field 800uSv/hr at ~0.5m
- Visual inspection of protect monitor performed:  No obvious issues with monitor plates.  Some small impact holes visible on exit side of monitor, similar to damage on the previous monitor from Oct 2015
- Electrical check performed on old protect monitor in hot cell, confirming the previous results
- Failed protect monitor removed, resistor pack installed and tested with expected results
- Profile monitor insulation found to be blackened and cracking, so the profile monitor cassette was also removed
- New protect monitor and profile monitors installed
- Electrical check performed on both monitors at the hot cell: Results good for both

May 10th:

- Vertical and horizontal positions measured for the new protect and profile monitors
- Profile monitor actuation tested:  started moving ~10psi, fully actuated ~50psi, monitor limit switches ok
- Measured total resistance of potentiometer: 20.26kOhms
- Target windows inspected and photos taken (stored on Isaac's hard drive), nothing unusual encountered
- T1-MK2 target returned to beamline, vacuum pumped down without issue
- T1 cooling package started fine with no change in vacuum
- T1 profile monitor actuated in and out.  Movement smooth and slow
- Electrical check done on protect and profile monitors from mezzanine: both good

The M20 beam blocker main shaft seals were replaced as per the T1/T2 target device maintenance schedule.  Both air cylinders were found to be leaking past the seals, and the blocker would not actuate in the hot cell even with 100psi (max site air) so they were also replaced.  With the new cylinders the blocker actuated smoothly at about 45psi.  It is recommended that the air cylinders be replaced every second service (once per 10 years).  The blocker was returned to the beam line and smooth actuation and limit switch functionality were confirmed.

The following final checks were performed on T1/T2 devices:

- Lower air amplifier started and tuned to 120psi output
- Air amplifier cycle time measured:  70 seconds for two cylinder turn-overs (1 complete cycle)
- T1 and T2 profile monitor actuation checked:  T1 reached in and out limits but speed unknown because target covered;  T2 traveled smoothly in and out and reached limits as expected
- T1 and T2 target ladders were moved to the various targets that could be used this running period (#1 for T1;  #1, 3, and 4 for T2) - No issues

The T1 and T2 targets are ready for the running period.  Shielding can now be replaced over T2.
 

 The following work was performed on the T1 and T2 cooling packages during the 2018 shutdown:

- T1 and T2 cooling packages both sampled and drained to South TNF holding tank
- Water released after testing and approval from RPG

- T1 high active and M15 filters changed (demin. filter to be changed next year)
- Replaced all T1 Proteus paddle wheels, shafts, and o-rings
- Replaced T1 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T1 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T1 package filled and started, no change to T1 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

- T2 high active filter changed (demin. filter to be changed next year)
- Replaced all T2 Proteus paddle wheels, shafts, and o-rings
- Replaced T2 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T2 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T2 resin can changed;  old resin can flushed with water then air in Warm Cell and left to decay
- T2 package filled and started, no change to T2 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

The cooling packages are functioning normally and are ready for the running period.

The following is a summary of MRO work done on the T2-MK2 target during the 2018 shutdown:

Jan 8:
- Target moved from beamline to hot cell;  130mSv/hr at 0.5m

Jan 10:
- Position 4 (12mm Be) beam spots photographed.  Spot is slightly high and left (from beam's eye view), also smaller and rounder (less eliptical) than usual - See E-Log #232

Jan 11:
- Flushed position 0,1,2,3,4 with water, then 4,2,1,0 with air
- Left with 10psi air running overnight in Position 0

Jan 12:
- Moved to position 4, 10psi air running through over the weekend to dry cooling lines

Jan 15:
- Pumped out water lines with leak detector
- Frist pump down blanked off tubing and new vac hose (no LN2): ~15 mins to get to 0.0E-10 TorrL/s
- Pumped out target w/ cold trap:  ~7 mins to get to 0.0E-10 TorrL/s (left pumping for 2 hours)

April 2:
- Removed plugs at position 3 and 5
- Pressure tested, leak tested, and recorded alignment dimensions for first batch of new 5cm targets (#301, 307, and 309) - Records stored with targets in RH lab office
- Installed new 5cm beryllium target #309 at Position 3, and target #307 at Position 5 
- Helium leak check on target water system:  Baseline 0.0E-10 TorrL/s leak rate, 1.0E-3Torr, half second helium applications @ 4psi
      Positions 1, 2, 3: no response
      Positions 4, 5: up to 5.0E-10 after delay (appears to be drift)
      Lower 'C' seals: no response
      Upper-left 'C' seal: up to 1.2E-9
      Upper-right 'C' seal: up to 2.4E-9
      Left bellows: up to 6.0E-9
      Right bellows: up to 3.0E-9
- Deemed water leak tight and ready for use (based on previous leak check results and operating experience)
- Tested all target position micro switches: all OK
- Tested profile monitor limits: both OK
- Tested profile monitor actuation: off down limit @ 5psi, on up limit at 35psi
- Tested potentiometer: slightly noisy signal from position 0 to 2 (no change from before)
- Measured Position 3 and 5 elevations

April 3:
- Measured Position 3 and 5 horizontal alignment wrt ladder
- Quick repeat of leak check:  no change

April 6:
- Target returned to beamline
- Vacuum pulled on T2 area (1ACG4 down to 5mTorr in ~2.5hrs, was 4mTorr before venting yesterday)
- T2 cooling package started with target at Position 0 - no change to T2 vacuum
- Target information sheet and elevations data delivered to Ops
- XTPAGE 7L information updated
 

 Five 5 gallon metal pails with active waste were moved from the Warm Cell to the Cyclotron East Vault Tunnel today for long term storage.  The pails were labeled with radioactive item tags and storage item numbers (1442 through 1446).  The pails contain used resin from T1, T2, and STF, as well as one pail with active metal parts.

 The RH Lab active sump level reached the high limit yesterday due to an overflow of the ultrasonic cleaner in the Rad Waste Area.  The sump was sampled by RPG, and after analysis the water was approved for release.  The water was pumped out today.  Local and MCR alarms functioned as expected.

 The T2-MK2 target was moved from the beamline to the hot cell on January 8th.  The field at 0.5m was 130mSv/hr.  Photographs of the beam spot of the Position 4 target are attached below.  This is a 12mm beryllium target (ID: "JUL.06#4).  This is the only target on the T2 ladder that has seen beam since it was installed on October 10, 2017.  A shielding plug was installed on the T2 flange.  A new 5cm target will be installed on the T2-MK2 ladder after fabrication is complete.  Regular testing and maintenance of the target assembly will also be performed before it is returned to the beamline.

The M20 beam blocker was also moved from the beamline to Hole #7 in the storage pit.  The field at 0.5m was 22mSv/hr.  A blank-off plate was installed on the M20BB flange (no o-ring).  Seal replacement and general inspection/testing will be performed before it is returned to the beamline.

Six 5 gallon pails were transferred this morning from a temporary storage bunker in the Warm Cell to the Rad Waste Pit for long term storage.  The pails contain various BL1A active waste from the East Hot Cell including dozens of irradiated beryllium and graphite targets.  The field from the pails ranged from 0.1 to 9mSv/hr at 0.5m.  The work was performed according to the clean-out procedure (Document-142965).  Photographs and records of the pail contents, field, weight, etc have been uploaded to Docushare Collection #20185 and hard paper copies were given to RPG.

All remaining waste in the east hot cell is too large to fit into 5 gallon pails, so it must be packaged in a 55 gallon drum.  Some long items will need to be cut in the cell before hand.

The field on the hot cell roof is now less than 5uSv/hr in working areas, and a maximum of 55uSv/hr at 0.5m above the east hot cell opening.  The field was previously > 50uSv/hr in some working areas, and > 200uSv/hr above the opening.

Before and after photos of the hot cell interior are included below:

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

The following work was performed on the T2-MK2 target assembly during the 2017 fall shutdown:

Oct 4, 2017:
- Target moved from beamline to hot cell, measured 661mSv/hr at 0.5m (move went smoothly, having camera views at N/S and E/W angles very helpful)
- Flask tray wet wiped after transfer: 1st wipe 600cpm on 44-2 monitor, 2nd wipe 900cpm (3rd swipe not taken because tray looked clean)
- Flushed target with clean water, then compressed air (positions 0 through 5)
- Male Hansen fittings on the target assembly swapped with new units (in process of replacing all self sealing fittings in T1 and T2 water system)
- Checked all micro-switch positions and potentiometer readback: switches ok, pot value jumpy around position 1 and 2 (known from before), otherwise fine
- Left target overnight at position 4 with 20psi supply air running through to dry water system

Oct 5, 2017:
- Pumped down target with cold trap (~2 hours)
- Baseline leak check performed: 6 mins to reach 5.0E-3 Torr (3.2E-9 Torr*L/s), 21 mins to reach 1.0E-3 Torr (1.4E-10 Torr*L/s)
- Helium spray at 3psi for 0.5s bursts, tested with ladder in position 4
- No leak at any target positions or plugs
- Small leak on left side bellows, or bellows seal to ladder: up to 8.0E-9 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Small leak on right side bellows, or bellows seal to ladder: up to 2.0E-8 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Beam spot photos taken, position 1 (only target used this running period) has fairly well centered beam spot (see below)
- Position 4 target removed ("JAN29'97"), was very difficult to remove because one water tube stuck in ladder (this target was installed on ladder in 1997!  First beam 2014)
- Used hammer in hot cell to remove which eventually broke the target water tubes from the wire EDM housing
- sealing surfaces cleaned with burgandy Scotchbrite on electric dremel tool
- Inspected sealing surfaces and ladder fitting threads using Nikon level: all appeared clean and undamaged
- Installed new 12mm beryllium target at position 4 ("Jul.06#4"), no issues with install
- Leak check repeated: reached 1.0E-3 Torr after 3 minutes (1.3E-8Torr*L/s)
- Down to 3.5E-9 Torr*L/s after 18 mins, began test
- No response at all target positions, checked left side bellows: same response as above
- Beam spot profile at Position 1 measured with Nikon level
- Vertical and horizontal positions of new Position 4 target measured
- New profile monitor cassette retainer pin installed (plate hole had to be opened up to 0.375" diameter), used vice grip pliers clamped onto clevis pin in order to install

Oct 6, 2017:
- Profile monitor actuation tested: start movement 5psi, fully IN at 35psi
- Profile monitor limit switches tested, ok
- Target transported back to beamline, no issues
- BL1A pumped down, Edi reported pump-down normal
- Electrical check performed on T2 profile monitor from mezzannine: Sheng-Li reported results normal
- Cooling system started: no change to BL1A vacuum
- Target info sheet and target elevations sheet delivered to Main Ops
- XTPAGE 7L information updated

 The #6 male Hansen fittings for the T1-MK1 and T2-MK1 targets (both currently stored in the target storage pit) were replaced today with new units.  Some of the target assembly Hansen fittings had shown signs of leaking and the internal seals are not easily replaced.

"120V" warning labels were put on the micro-switch IN and OUT limit sensors for the profile monitors on both targets.  These are the only micro-switches on the assemblies that still run at 120V after the PLC controls upgrade.

test 2

 The three failed T2 cooling system secondary side solenoid valves were tested yesterday to try to determine the problem.  The test steps are detailed below:

• New spare valve (NSV) produced audible "click" and valve was seen to physically open when 24V DC driving voltage was applied
• Failed valves of July 25, 2017 (J25), July 10, 2017 (J10), and November 2016 (N16) all did not click or open when driven
• The solenoid of NSV was removed from its valve and installed on each of the failed valves in turn.  The NSV solenoid succesfully opened each of J25, J10, and N16 (the valve bodies and pistons of the failed valves appear to be fine, and the solenoids all faulty)
• Each solenoid's coil resistance was measured: NSV = 187kOhm, J25 = 200kOhm, J10 = O/C, N16 = O/C

On the next maintenance day Tony Tateyama is planning on performing continuity tests on the T2CS:SVSEC control cable and comparing it to the control cable for identical solenoids in the system.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

The T2 heat exchanger secondary side solenoid failed on July 9 at approximately 19:30 (EPICS ID: B1A:T2CS:SVSEC).  The valve was replaced with a spare and the cooling package is now running normally.

The valve at this location also failed in November 2016:  https://elog.triumf.ca/TIS/RH-Meson+Hall/211  Further investigation will be attempted to determine if there is a root cause to the failure, or if it is a coincidence.

The attached OneNote PDF contains details of the work.  Cyclotron Fault #10119 has been returned.