During first injection of beam down 1A problems were encountered with the T1 protect monitor.  Testing was done by Probes Group from the mezzanine leading to the following investigation and repair work.

May 9th:

- T1 area uncovered, Probes Group performed testing at the top of the monolith confirming that the problem was below the vacuum flange
- T1-MK2 target moved from the beamline to the hot cell.  Measured field 800uSv/hr at ~0.5m
- Visual inspection of protect monitor performed:  No obvious issues with monitor plates.  Some small impact holes visible on exit side of monitor, similar to damage on the previous monitor from Oct 2015
- Electrical check performed on old protect monitor in hot cell, confirming the previous results
- Failed protect monitor removed, resistor pack installed and tested with expected results
- Profile monitor insulation found to be blackened and cracking, so the profile monitor cassette was also removed
- New protect monitor and profile monitors installed
- Electrical check performed on both monitors at the hot cell: Results good for both

May 10th:

- Vertical and horizontal positions measured for the new protect and profile monitors
- Profile monitor actuation tested:  started moving ~10psi, fully actuated ~50psi, monitor limit switches ok
- Measured total resistance of potentiometer: 20.26kOhms
- Target windows inspected and photos taken (stored on Isaac's hard drive), nothing unusual encountered
- T1-MK2 target returned to beamline, vacuum pumped down without issue
- T1 cooling package started fine with no change in vacuum
- T1 profile monitor actuated in and out.  Movement smooth and slow
- Electrical check done on protect and profile monitors from mezzanine: both good

The M20 beam blocker main shaft seals were replaced as per the T1/T2 target device maintenance schedule.  Both air cylinders were found to be leaking past the seals, and the blocker would not actuate in the hot cell even with 100psi (max site air) so they were also replaced.  With the new cylinders the blocker actuated smoothly at about 45psi.  It is recommended that the air cylinders be replaced every second service (once per 10 years).  The blocker was returned to the beam line and smooth actuation and limit switch functionality were confirmed.

The following final checks were performed on T1/T2 devices:

- Lower air amplifier started and tuned to 120psi output
- Air amplifier cycle time measured:  70 seconds for two cylinder turn-overs (1 complete cycle)
- T1 and T2 profile monitor actuation checked:  T1 reached in and out limits but speed unknown because target covered;  T2 traveled smoothly in and out and reached limits as expected
- T1 and T2 target ladders were moved to the various targets that could be used this running period (#1 for T1;  #1, 3, and 4 for T2) - No issues

The T1 and T2 targets are ready for the running period.  Shielding can now be replaced over T2.
 

 The following work was performed on the T1 and T2 cooling packages during the 2018 shutdown:

- T1 and T2 cooling packages both sampled and drained to South TNF holding tank
- Water released after testing and approval from RPG

- T1 high active and M15 filters changed (demin. filter to be changed next year)
- Replaced all T1 Proteus paddle wheels, shafts, and o-rings
- Replaced T1 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T1 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T1 package filled and started, no change to T1 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

- T2 high active filter changed (demin. filter to be changed next year)
- Replaced all T2 Proteus paddle wheels, shafts, and o-rings
- Replaced T2 male and female #6 SS Hansen fittings at the package/hose interface
- Replaced T2 #4 female Hansen fittings for the resin can (now all Hansen fittings on the high active loop have been replaced)
- T2 resin can changed;  old resin can flushed with water then air in Warm Cell and left to decay
- T2 package filled and started, no change to T2 area vacuum
- Demin flow tuned to ~1.0gpm
- Package inspected while running, no leaks observed

The cooling packages are functioning normally and are ready for the running period.

The following is a summary of MRO work done on the T2-MK2 target during the 2018 shutdown:

Jan 8:
- Target moved from beamline to hot cell;  130mSv/hr at 0.5m

Jan 10:
- Position 4 (12mm Be) beam spots photographed.  Spot is slightly high and left (from beam's eye view), also smaller and rounder (less eliptical) than usual - See E-Log #232

Jan 11:
- Flushed position 0,1,2,3,4 with water, then 4,2,1,0 with air
- Left with 10psi air running overnight in Position 0

Jan 12:
- Moved to position 4, 10psi air running through over the weekend to dry cooling lines

Jan 15:
- Pumped out water lines with leak detector
- Frist pump down blanked off tubing and new vac hose (no LN2): ~15 mins to get to 0.0E-10 TorrL/s
- Pumped out target w/ cold trap:  ~7 mins to get to 0.0E-10 TorrL/s (left pumping for 2 hours)

April 2:
- Removed plugs at position 3 and 5
- Pressure tested, leak tested, and recorded alignment dimensions for first batch of new 5cm targets (#301, 307, and 309) - Records stored with targets in RH lab office
- Installed new 5cm beryllium target #309 at Position 3, and target #307 at Position 5 
- Helium leak check on target water system:  Baseline 0.0E-10 TorrL/s leak rate, 1.0E-3Torr, half second helium applications @ 4psi
      Positions 1, 2, 3: no response
      Positions 4, 5: up to 5.0E-10 after delay (appears to be drift)
      Lower 'C' seals: no response
      Upper-left 'C' seal: up to 1.2E-9
      Upper-right 'C' seal: up to 2.4E-9
      Left bellows: up to 6.0E-9
      Right bellows: up to 3.0E-9
- Deemed water leak tight and ready for use (based on previous leak check results and operating experience)
- Tested all target position micro switches: all OK
- Tested profile monitor limits: both OK
- Tested profile monitor actuation: off down limit @ 5psi, on up limit at 35psi
- Tested potentiometer: slightly noisy signal from position 0 to 2 (no change from before)
- Measured Position 3 and 5 elevations

April 3:
- Measured Position 3 and 5 horizontal alignment wrt ladder
- Quick repeat of leak check:  no change

April 6:
- Target returned to beamline
- Vacuum pulled on T2 area (1ACG4 down to 5mTorr in ~2.5hrs, was 4mTorr before venting yesterday)
- T2 cooling package started with target at Position 0 - no change to T2 vacuum
- Target information sheet and elevations data delivered to Ops
- XTPAGE 7L information updated
 

 Five 5 gallon metal pails with active waste were moved from the Warm Cell to the Cyclotron East Vault Tunnel today for long term storage.  The pails were labeled with radioactive item tags and storage item numbers (1442 through 1446).  The pails contain used resin from T1, T2, and STF, as well as one pail with active metal parts.

 The RH Lab active sump level reached the high limit yesterday due to an overflow of the ultrasonic cleaner in the Rad Waste Area.  The sump was sampled by RPG, and after analysis the water was approved for release.  The water was pumped out today.  Local and MCR alarms functioned as expected.

 The T2-MK2 target was moved from the beamline to the hot cell on January 8th.  The field at 0.5m was 130mSv/hr.  Photographs of the beam spot of the Position 4 target are attached below.  This is a 12mm beryllium target (ID: "JUL.06#4).  This is the only target on the T2 ladder that has seen beam since it was installed on October 10, 2017.  A shielding plug was installed on the T2 flange.  A new 5cm target will be installed on the T2-MK2 ladder after fabrication is complete.  Regular testing and maintenance of the target assembly will also be performed before it is returned to the beamline.

The M20 beam blocker was also moved from the beamline to Hole #7 in the storage pit.  The field at 0.5m was 22mSv/hr.  A blank-off plate was installed on the M20BB flange (no o-ring).  Seal replacement and general inspection/testing will be performed before it is returned to the beamline.

Six 5 gallon pails were transferred this morning from a temporary storage bunker in the Warm Cell to the Rad Waste Pit for long term storage.  The pails contain various BL1A active waste from the East Hot Cell including dozens of irradiated beryllium and graphite targets.  The field from the pails ranged from 0.1 to 9mSv/hr at 0.5m.  The work was performed according to the clean-out procedure (Document-142965).  Photographs and records of the pail contents, field, weight, etc have been uploaded to Docushare Collection #20185 and hard paper copies were given to RPG.

All remaining waste in the east hot cell is too large to fit into 5 gallon pails, so it must be packaged in a 55 gallon drum.  Some long items will need to be cut in the cell before hand.

The field on the hot cell roof is now less than 5uSv/hr in working areas, and a maximum of 55uSv/hr at 0.5m above the east hot cell opening.  The field was previously > 50uSv/hr in some working areas, and > 200uSv/hr above the opening.

Before and after photos of the hot cell interior are included below:

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.

The following work was performed on the T2-MK2 target assembly during the 2017 fall shutdown:

Oct 4, 2017:
- Target moved from beamline to hot cell, measured 661mSv/hr at 0.5m (move went smoothly, having camera views at N/S and E/W angles very helpful)
- Flask tray wet wiped after transfer: 1st wipe 600cpm on 44-2 monitor, 2nd wipe 900cpm (3rd swipe not taken because tray looked clean)
- Flushed target with clean water, then compressed air (positions 0 through 5)
- Male Hansen fittings on the target assembly swapped with new units (in process of replacing all self sealing fittings in T1 and T2 water system)
- Checked all micro-switch positions and potentiometer readback: switches ok, pot value jumpy around position 1 and 2 (known from before), otherwise fine
- Left target overnight at position 4 with 20psi supply air running through to dry water system

Oct 5, 2017:
- Pumped down target with cold trap (~2 hours)
- Baseline leak check performed: 6 mins to reach 5.0E-3 Torr (3.2E-9 Torr*L/s), 21 mins to reach 1.0E-3 Torr (1.4E-10 Torr*L/s)
- Helium spray at 3psi for 0.5s bursts, tested with ladder in position 4
- No leak at any target positions or plugs
- Small leak on left side bellows, or bellows seal to ladder: up to 8.0E-9 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Small leak on right side bellows, or bellows seal to ladder: up to 2.0E-8 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Beam spot photos taken, position 1 (only target used this running period) has fairly well centered beam spot (see below)
- Position 4 target removed ("JAN29'97"), was very difficult to remove because one water tube stuck in ladder (this target was installed on ladder in 1997!  First beam 2014)
- Used hammer in hot cell to remove which eventually broke the target water tubes from the wire EDM housing
- sealing surfaces cleaned with burgandy Scotchbrite on electric dremel tool
- Inspected sealing surfaces and ladder fitting threads using Nikon level: all appeared clean and undamaged
- Installed new 12mm beryllium target at position 4 ("Jul.06#4"), no issues with install
- Leak check repeated: reached 1.0E-3 Torr after 3 minutes (1.3E-8Torr*L/s)
- Down to 3.5E-9 Torr*L/s after 18 mins, began test
- No response at all target positions, checked left side bellows: same response as above
- Beam spot profile at Position 1 measured with Nikon level
- Vertical and horizontal positions of new Position 4 target measured
- New profile monitor cassette retainer pin installed (plate hole had to be opened up to 0.375" diameter), used vice grip pliers clamped onto clevis pin in order to install

Oct 6, 2017:
- Profile monitor actuation tested: start movement 5psi, fully IN at 35psi
- Profile monitor limit switches tested, ok
- Target transported back to beamline, no issues
- BL1A pumped down, Edi reported pump-down normal
- Electrical check performed on T2 profile monitor from mezzannine: Sheng-Li reported results normal
- Cooling system started: no change to BL1A vacuum
- Target info sheet and target elevations sheet delivered to Main Ops
- XTPAGE 7L information updated

 The #6 male Hansen fittings for the T1-MK1 and T2-MK1 targets (both currently stored in the target storage pit) were replaced today with new units.  Some of the target assembly Hansen fittings had shown signs of leaking and the internal seals are not easily replaced.

"120V" warning labels were put on the micro-switch IN and OUT limit sensors for the profile monitors on both targets.  These are the only micro-switches on the assemblies that still run at 120V after the PLC controls upgrade.

test 2

 The three failed T2 cooling system secondary side solenoid valves were tested yesterday to try to determine the problem.  The test steps are detailed below:

• New spare valve (NSV) produced audible "click" and valve was seen to physically open when 24V DC driving voltage was applied
• Failed valves of July 25, 2017 (J25), July 10, 2017 (J10), and November 2016 (N16) all did not click or open when driven
• The solenoid of NSV was removed from its valve and installed on each of the failed valves in turn.  The NSV solenoid succesfully opened each of J25, J10, and N16 (the valve bodies and pistons of the failed valves appear to be fine, and the solenoids all faulty)
• Each solenoid's coil resistance was measured: NSV = 187kOhm, J25 = 200kOhm, J10 = O/C, N16 = O/C

On the next maintenance day Tony Tateyama is planning on performing continuity tests on the T2CS:SVSEC control cable and comparing it to the control cable for identical solenoids in the system.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

The T2 heat exchanger secondary side solenoid failed on July 9 at approximately 19:30 (EPICS ID: B1A:T2CS:SVSEC).  The valve was replaced with a spare and the cooling package is now running normally.

The valve at this location also failed in November 2016:  https://elog.triumf.ca/TIS/RH-Meson+Hall/211  Further investigation will be attempted to determine if there is a root cause to the failure, or if it is a coincidence.

The attached OneNote PDF contains details of the work.  Cyclotron Fault #10119 has been returned.

 On Thursday May 11th operators attempted to move the T2 profile monitor (T2-MK2) to the IN position but were unsuccessful.  The monitor would leave the OUT limit, but would not reach the IN limit.  This was repeated 2 times. The air amplifier was inspected and nothing out of the ordinary was noted.  The T1 profile monitor was tested and was working properly.  At the time it was suspected that the monitor cassette had become dislodged as occurred earlier this running period (see E-Log #221).

Today monitor actuation was attempted again and it moved to the IN position normally, this was done a total of 3 times..  An electronic check was performed by Mike Russell who reported that it looked the same as when he last checked, meaning the monitor cassette was definitely not unplugged. The monitor was then used with beam and Eric Chapman reported that monitor was functioning normally.  With the beam spot centered vertically on the profile monitor the protect monitor showed more beam on the lower plate, which is expected because the protect monitor center is 2-3mm higher than the profile monitor center (see E-Log #220).

It is unknown why it would not actuate properly last week, but seems to be fine now.  Cyclotron fault #9928 has been returned.

 During the target move on Thursday May 4th the paddle wheel and paddle wheel shaft for the T2CS:FGTGT sensor were changed to address transient spikes of approximately +/-1.0 gpm which were causing nuisance trips.  There have been no transient spikes recorded since the change, and the nominal flow value dropped from 4.9gpm to 3.6gpm.  Fault report #9836 has been returned, and the warning and trip levels, which were set higher to avoid nuisance alarms, were returned to nominal values.

We will continue to swap T1 and T2 flow meter paddle wheels every 2 years, which has been a suitable replacement interval besides this one case.

 On May 5th the T2-MK2 target was moved to the hot cell for the purpose of checking the beam spot on the Position 2 beryllium target to confirm the results of the T2 area realignment detailed in E-logs #218 and #220.  In preparation beam was put on this target overnight for approximately 1.5hrs at 10uA and 4 hours at 50uA (previously had not seen beam).  Also, there had been issues with the T2 profile monitor over the last couple days, so inspection and repeat of electrical check in the hot cell was planned.  The field in the T2 monolith work area was up to 1.3mSv/hr, and a maximum field of 230mSv/hr was measured 0.5m from the target during removal.

Unfortunately no beam spot was visible on either the entrance or exit windows at Position 2  (a longer exposure will be required next time a beam spot is to be inspected).  Detailed photos were taken, one of each side attached.  It was noticed that the T2 profile monitor cassette had become dislodged within its housing (it was sitting approximately 1" from the fully installed position - photo attached).  It is suspected that this happened when the target assembly was in the beamline - the first time the profile monitor was exercised back in April it moved quite quickly which may have caused the cassette to come loose (it is only held in with friction of the 9-pin connectors and the fit of the cassette body in the housing).  It was re-inserted in the hot cell and an electrical check was performed by Shengli: all good.  After the target was returned to the beam line the electrical check was repeated while the monitor was actuated in and out: once again all good, and the movement of the monitor was slow and smooth (video taken)

Prior to this job new cameras were installed in the hot cell opening aimed at the support ring hole (one aimed north, the other east).  This proved to be a very useful upgrade and made the installation process significantly easier and smoother.  Two cameras were also used during target installation at the monolith (only 1 was used in the past).  Having a second camera at a different angle made alignment easier (reducing time and dose), and should be adopted for future target moves.

Vacuum pumpdown was normal, the cooling system was restarted without change to the vacuum, and operators were able to use the T2 profile monitor during beam ramp-up overnight.

Dose for this job:  Isaac 0.17mSv, Keith 0.06, Ron 0.06

The following work was performed on the T2-MK2 target assembly and the T2 Protect assembly for maintenance purposes and to implement the alignment strategy detailed in E-Log #218:

March 30 (T2-MK2 target in hot cell):
- Profile monitor actuation checked:  moved in and out smoothly, fully in at ~25psi
- Ran through all target positions, motion smooth, micro-switches all ok, potentiometer value unsteady when moving between position 0-1, steady when stopped, no issues form positions 2-5 (this was also observed last shutdown and there were no issues with ladder moves in the 2016 running period using EPICS)
- Water system pumped out with cold trap for 2 hours (previously was purged with clean water, then air overnight)
- Leak check performed: 10 minutes to reach 5.0E-3 Torr TP pressure, 25 minutes to reach 1.0E-3 Torr; 3psi helium supply applied in 0.5s bursts;  No response at all target locations and all four aluminum delta seal locations.  At base of bellows on left side (when facing ladder) quick response up to 0.6E-8atm.cc/s, quick recovery to baseline (<10s); At base of bellows on right side (when facing ladder) quick response up to 1.2E-8atm.cc/s, quick recovery to baseline (<10s).  These small helium leaks were known (see E-Log #95) and have not caused issue in multiple years of running.
 

April 5:
- Position 1 and 2 potentiometer ratio changed to achieve new target alignment, look-up tables updated in PLC code (see E-Log #218, and attached new elevation measurements)
- Position 1 and 2 micro-switches adjusted to be centered at new positions
- Profile monitor inspected: no broken wires, but some not quite straight or parallel with others, also some wire insulation blackened.  Scott Kellogg recommended installing new profile cassette
- Old profile cassette removed in hot cell and new one installed
- Profile monitor elevation adjusted (see attached new elevation measurements)
 

April 6:
- Electrical check on profile monitor performed at HC by Dave Cameron: deemed OK
- T2-MK2 moved from the hot cell to the beam line (~200mSv/hr at 0.5m)
- T2 protect monitor moved from beam line to hot cell (~16mSv/hr at 0.5m)
 

April 7:
- Elevation of protect monitor measured to confirm old measurement: within 0.1mm
- Old protect monitor cassette removed
- New cassette supplied pre-adjusted by Scott Kellogg
- New cassette installed in HC, elevation measured on entrance side: 1832.8mm (intended elevation was 1833.2mm, from E-Log #218)
- Protect monitor electronics checked by Shengli Liu and Dave Cameron at the hot cell: all OK
 

April 10:
- T2 profile monitor cable plugged in which caused unexpected jump in vacuum: 20mTorr - 180mTorr
- Large nut at top of feed-through connector tightened and vacuum returned to baseline
- T2 target position 1 and 2 potentiometer values tested from EPICS system to confirm PLC look-up table changes ok
 

April 11:
- T2 Protect assembly returned to beamline (12mSv/hr at 0.5m).  T2 monolith position was monitored by Beamlines Group with their laser tracker during the move.  A maximum of 0.2mm movement was measured which is attributable to noise.
- Air amplifier started, leak found from one of the M20BB cylinders;  went away after reset of BB control system, BB exercised several times and leak could not be reproduced (no change to BL vacuum)
- T2 profile monitor tuned for slower actuation, smooth movement confirmed and no change to beamline vacuum (also done for T1 profile monitor)
- Electrical check on T2 protect monitor performed by Dave Cameron and Shengli Liu from racks on mezannine: all OK
- Electrical check on T2 profile monitor performed by Dave Cameron and Shengli Liu from racks on mezannine: initially very low signal, they discovered a previously unknown large resistor which was bypassed for the test:  approximately 1/2 of signals show response to HV blip, deemed OK to restore shielding
- Target info sheet and updated elevations measurements delivered to Operations (elevations sheet attached)

Note: the T1 target (T1-MK2) was not removed from the beamline this shutdown, so new elevation measurements have not been submitted and T1 profile and protect monitor checks have not been performed

 The following work has been completed on the T2 cooling package:

- T2 system drained to South TNF holding tank (completed in Jan 2017, water already released)
- Target water and demin. circuit Cuno filters changed
- Both female Hansen fittings at target end of flexible hose replaced with new SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion inside)
- Small leak on pipe thread fitting near target water supply solenoid repaired
- Reservoir o-rings replaced
- System filled and re-started, no leaks observed, no drop in water level over 48 hour run period
- Demin. circuit flow rate tuned to 1.0gpm