The T1 cooling system inlet pressure transducer (EPICS ID: B1A:T1CS:PGIN,  Omega part # PX315-100GI) began failing in August (exhibited gradual decline in pressure with no corresponding change to flow rate or outlet pressure).  A new transducer was installed on Sept 30 2016 which fixed the problem.  Cyclotron Fault Report #9238 was returned and the temporary bypass on the warning level was removed.

 An expansion tank high level warning has been added to the T1/T2 control system under Work Request #4011.  The purpose for this new warning is to alert staff of an unexpected water increase in the unlikely scenario that a leak develops in the heat exchanger between the two fluids and CuALCW on the secondary side enters the target water system.  The new thresholds for high level warn, low level warn, and low level trip have been set at 40cm, 30cm, and 25cm respectively.  The nominal water level is 35cm.  All thresholds were tested by adding water to the tank, or removing it.  All responded as expected.

 At approximately 11am today the T2 cooling package heat exchanger secondary side water flow suddenly dropped from ~11gpm to 0.  This was confirmed to be a real reading by observing that with no secondary cooling both the inlet and outlet target water temperatures rose steadily with beam on the target.  While standing on the blocks in the T2 area and having a controls expert (wither "superuser" access mode) actuate the secondary side solenoid independently it was confirmed that no "click" was heard when power was supplied to the valve.  No other changes were observed with other parts of the CuALCW system, therefore it was concluded that the secondary side solenoid valve (B1A:T2CS:SVSEC) had failed.

Blocks were removed above the T2 cooling package, the CuALCW lines were valved off and drained, and the failed valve was removed.  A new valve was installed.  A section of 3/4" SS tubing was replaced with flexible 21" long 3/4" diameter SS hose to allow re-installation (alignment could not be achieved with rigid tubing).  Water supply and return were restored, and the system was started.  Because of the new section of tubing, the flow through the secondary side dropped from ~11gpm to ~9gpm.  This should not have a significant effect on operation.  No leaks were observed.

Total dose for Isaac: 0.15mSv, for Keith: 0.03mSv

 The T2-MK2 target assembly was moved from the beam line to the hot cell today.  A maximum field of 203mSv/hr was measured at approximately 0.5m

A steel shielding plug has been installed in the T2 hole to reduce shine and allow the T2 volume to be pumped down.

 The beam spot on the 10cm beryllium target in Position 1 on T2-MK2 (Target ID: "OCT.06#3") was measured and photographed yesterday.  A fairly well centered spot was measured to be approximately 4.6mm wide x 5.2mm tall on the entrance window and 5.7mm wide x 7.6mm tall on the exit window using the Nikon micrometer level.  This target first saw beam on June 16, 2016.

A PDF scan of the beam spot measurements and photographs of the beam spots are attached.

The following work was completed on the T1 cooling package:
- water drained to BL1A holding tank
- all resevoir o-rings changed
- all cuno filters changed:
     M15 permanent magnet filter < 5uSv/hr at 0.5m, dark grey colour
     Target inlet filter 210uSv/hr at 0.5m, light brown colour
     Resin filter, <5uSv/hr, looks clean and new
- package re-filled and started
- demin circuit flow tuned to 1.0gpm
- #6 female Hansen fittings at target end of flexible hose swapped for previously used SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion on the inside)
 

The T2 cooling package was also drained to the 1A holding tank.
After approval from RPG the holding tank water was released.

Note: the appearance of the used cuno filters is consistent with previous years.

 On March 13th in the morning the Meson Hall Hot Cell Lab active sump reached the high level sensor.  The system responded as expected:  high level panel warning light illuminated, siren sounded, and MCR notified via "B1 RCR1 RM6 WARN".  A 1L sample was delivered to RPG for analysis.  After approval, the sump was pumped out over the following two days.  The procedure in Document #64834 was followed for sampling and pump-out.

The 18' shielding block which lies west of the T1 cooling package was removed on Friday Mar 24 so that vacuum equipment could be craned into the 1A tunnel.  During replacement, the flexible SS hose for T1 target return water got caught under the block and was ripped apart.  Workers in the area did not realize this had occurred.  The cooling package tripped off when the hose ripped (likely due to low water flow or low supply pressure).  In total approximately 6.5L was lost which flowed to the 1A tunnel below.  The hose connections at the target and cooling package do not appear to have sustained damage.  Plans are under way for a replacement hose to be fabricated and installed.

Update March 30, 2017:  The damaged hose has been replaced with a new 20' long braided SS hose with 3/4" male NPT fittings both ends.  Both air supply lines for T1 profile monitor actuation (1/4" copper) were also damaged during the Mar 24 block move and they have now been replaced.  All hoses, cables, etc on the west side of the T1 monolith have been re-routed or secured to reduce the chance of this happening again in the future.

 The following changes were made to the T1/T2 PLC control system:

- De-glitching system investigated and Tony discovered that it was programmed incorrectly:  the de-glitch time was observed only when a PV changed from bad state to good, not the opposite (meaning there was actually no de-glitching in place since system start-up in 2016!).  This was reversed and the following de-glitch times were programmed:  100ms for trip thresholds except 300ms for flow meter trips, and 2s de-glitch on all warning thresholds.  The new system was tested with the PLC in simulation mode and seems to be working correctly.  This should eliminate or significantly reduce nuisance warnings and trips from the system.

- PLC system code was modified so that the target version displays '???' when no target is installed instead of defaulting to 'MK1'  (all target positions registered as 'plugged' with no target installed, so there was no danger to the system, but this was still confusing to operators)

A strategy for alignment of the T2 protect monitor, T2 target, and T2 Collimator A was discussed today and decided upon by Yuri Bylinski, Isaac Earle, Yi-Nong Rao, and Roman Ruegg.  The purpose of the adjustment is to try to reduce the amount of beam lost as it passes through the T2 area (presumably being blocked by the collimator because it is not possible to steer the beam through all three devices in their current configuration).  Keeping in mind that changing the elevation of Collimator A would be prohibitively time consuming it was decided that the protect monitor and target both be adjusted upwards 3.0mm compared to where they were during the 2016 running period.  This will result in the following elevations (all w.r.t. the T2 target flange).  The T2 profile monitor elevation will also be adjusted to match the target elevation.  

T2 Protect Monitor:  1833.5mm (1833.2mm from protect monitor flange)

T2 Target: 1836.5mm 

T2 Collimator A:  1834.7mm  (1834.5mm from collimator flange;  This is the current collimator elevation, measurement taken by Tom Lyth in Feb 2009)

These values are similar to what were used in 2009 during which time TNF neutron flux rates were acceptable (>800), and centered beam spots were achieved.  During the 2016 running period the beam spot on target was also roughly centered.  At both times the target was approximately 3mm lower than the protect which seems to indicate the beam is travelling at a downwards angle, possibly not correctable with available tuning.  This is a departure from the strategy in the last few years when the target and protect monitor were adjusted downwards to bring them closer to the beam plane (they had risen due to movement of the T2 monolith).  This is now thought to have been a mistake because it made it difficult to steer beam through the collimator.

Beam will be started with T2 in Position 2 (new target, never seen beam), then after a few days the target assembly will be moved to the hot cell to check the beam spot to confirm alignment and make changes if the strategy needs adjusting.

A PDF copy of Yi-Nong's summary is also attached.

 The following work has been completed on the T2 cooling package:

- T2 system drained to South TNF holding tank (completed in Jan 2017, water already released)
- Target water and demin. circuit Cuno filters changed
- Both female Hansen fittings at target end of flexible hose replaced with new SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion inside)
- Small leak on pipe thread fitting near target water supply solenoid repaired
- Reservoir o-rings replaced
- System filled and re-started, no leaks observed, no drop in water level over 48 hour run period
- Demin. circuit flow rate tuned to 1.0gpm

The following work was performed on the T2-MK2 target assembly and the T2 Protect assembly for maintenance purposes and to implement the alignment strategy detailed in E-Log #218:

March 30 (T2-MK2 target in hot cell):
- Profile monitor actuation checked:  moved in and out smoothly, fully in at ~25psi
- Ran through all target positions, motion smooth, micro-switches all ok, potentiometer value unsteady when moving between position 0-1, steady when stopped, no issues form positions 2-5 (this was also observed last shutdown and there were no issues with ladder moves in the 2016 running period using EPICS)
- Water system pumped out with cold trap for 2 hours (previously was purged with clean water, then air overnight)
- Leak check performed: 10 minutes to reach 5.0E-3 Torr TP pressure, 25 minutes to reach 1.0E-3 Torr; 3psi helium supply applied in 0.5s bursts;  No response at all target locations and all four aluminum delta seal locations.  At base of bellows on left side (when facing ladder) quick response up to 0.6E-8atm.cc/s, quick recovery to baseline (<10s); At base of bellows on right side (when facing ladder) quick response up to 1.2E-8atm.cc/s, quick recovery to baseline (<10s).  These small helium leaks were known (see E-Log #95) and have not caused issue in multiple years of running.
 

April 5:
- Position 1 and 2 potentiometer ratio changed to achieve new target alignment, look-up tables updated in PLC code (see E-Log #218, and attached new elevation measurements)
- Position 1 and 2 micro-switches adjusted to be centered at new positions
- Profile monitor inspected: no broken wires, but some not quite straight or parallel with others, also some wire insulation blackened.  Scott Kellogg recommended installing new profile cassette
- Old profile cassette removed in hot cell and new one installed
- Profile monitor elevation adjusted (see attached new elevation measurements)
 

April 6:
- Electrical check on profile monitor performed at HC by Dave Cameron: deemed OK
- T2-MK2 moved from the hot cell to the beam line (~200mSv/hr at 0.5m)
- T2 protect monitor moved from beam line to hot cell (~16mSv/hr at 0.5m)
 

April 7:
- Elevation of protect monitor measured to confirm old measurement: within 0.1mm
- Old protect monitor cassette removed
- New cassette supplied pre-adjusted by Scott Kellogg
- New cassette installed in HC, elevation measured on entrance side: 1832.8mm (intended elevation was 1833.2mm, from E-Log #218)
- Protect monitor electronics checked by Shengli Liu and Dave Cameron at the hot cell: all OK
 

April 10:
- T2 profile monitor cable plugged in which caused unexpected jump in vacuum: 20mTorr - 180mTorr
- Large nut at top of feed-through connector tightened and vacuum returned to baseline
- T2 target position 1 and 2 potentiometer values tested from EPICS system to confirm PLC look-up table changes ok
 

April 11:
- T2 Protect assembly returned to beamline (12mSv/hr at 0.5m).  T2 monolith position was monitored by Beamlines Group with their laser tracker during the move.  A maximum of 0.2mm movement was measured which is attributable to noise.
- Air amplifier started, leak found from one of the M20BB cylinders;  went away after reset of BB control system, BB exercised several times and leak could not be reproduced (no change to BL vacuum)
- T2 profile monitor tuned for slower actuation, smooth movement confirmed and no change to beamline vacuum (also done for T1 profile monitor)
- Electrical check on T2 protect monitor performed by Dave Cameron and Shengli Liu from racks on mezannine: all OK
- Electrical check on T2 profile monitor performed by Dave Cameron and Shengli Liu from racks on mezannine: initially very low signal, they discovered a previously unknown large resistor which was bypassed for the test:  approximately 1/2 of signals show response to HV blip, deemed OK to restore shielding
- Target info sheet and updated elevations measurements delivered to Operations (elevations sheet attached)

Note: the T1 target (T1-MK2) was not removed from the beamline this shutdown, so new elevation measurements have not been submitted and T1 profile and protect monitor checks have not been performed

 On May 5th the T2-MK2 target was moved to the hot cell for the purpose of checking the beam spot on the Position 2 beryllium target to confirm the results of the T2 area realignment detailed in E-logs #218 and #220.  In preparation beam was put on this target overnight for approximately 1.5hrs at 10uA and 4 hours at 50uA (previously had not seen beam).  Also, there had been issues with the T2 profile monitor over the last couple days, so inspection and repeat of electrical check in the hot cell was planned.  The field in the T2 monolith work area was up to 1.3mSv/hr, and a maximum field of 230mSv/hr was measured 0.5m from the target during removal.

Unfortunately no beam spot was visible on either the entrance or exit windows at Position 2  (a longer exposure will be required next time a beam spot is to be inspected).  Detailed photos were taken, one of each side attached.  It was noticed that the T2 profile monitor cassette had become dislodged within its housing (it was sitting approximately 1" from the fully installed position - photo attached).  It is suspected that this happened when the target assembly was in the beamline - the first time the profile monitor was exercised back in April it moved quite quickly which may have caused the cassette to come loose (it is only held in with friction of the 9-pin connectors and the fit of the cassette body in the housing).  It was re-inserted in the hot cell and an electrical check was performed by Shengli: all good.  After the target was returned to the beam line the electrical check was repeated while the monitor was actuated in and out: once again all good, and the movement of the monitor was slow and smooth (video taken)

Prior to this job new cameras were installed in the hot cell opening aimed at the support ring hole (one aimed north, the other east).  This proved to be a very useful upgrade and made the installation process significantly easier and smoother.  Two cameras were also used during target installation at the monolith (only 1 was used in the past).  Having a second camera at a different angle made alignment easier (reducing time and dose), and should be adopted for future target moves.

Vacuum pumpdown was normal, the cooling system was restarted without change to the vacuum, and operators were able to use the T2 profile monitor during beam ramp-up overnight.

Dose for this job:  Isaac 0.17mSv, Keith 0.06, Ron 0.06

 During the target move on Thursday May 4th the paddle wheel and paddle wheel shaft for the T2CS:FGTGT sensor were changed to address transient spikes of approximately +/-1.0 gpm which were causing nuisance trips.  There have been no transient spikes recorded since the change, and the nominal flow value dropped from 4.9gpm to 3.6gpm.  Fault report #9836 has been returned, and the warning and trip levels, which were set higher to avoid nuisance alarms, were returned to nominal values.

We will continue to swap T1 and T2 flow meter paddle wheels every 2 years, which has been a suitable replacement interval besides this one case.

 On Thursday May 11th operators attempted to move the T2 profile monitor (T2-MK2) to the IN position but were unsuccessful.  The monitor would leave the OUT limit, but would not reach the IN limit.  This was repeated 2 times. The air amplifier was inspected and nothing out of the ordinary was noted.  The T1 profile monitor was tested and was working properly.  At the time it was suspected that the monitor cassette had become dislodged as occurred earlier this running period (see E-Log #221).

Today monitor actuation was attempted again and it moved to the IN position normally, this was done a total of 3 times..  An electronic check was performed by Mike Russell who reported that it looked the same as when he last checked, meaning the monitor cassette was definitely not unplugged. The monitor was then used with beam and Eric Chapman reported that monitor was functioning normally.  With the beam spot centered vertically on the profile monitor the protect monitor showed more beam on the lower plate, which is expected because the protect monitor center is 2-3mm higher than the profile monitor center (see E-Log #220).

It is unknown why it would not actuate properly last week, but seems to be fine now.  Cyclotron fault #9928 has been returned.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

 The three failed T2 cooling system secondary side solenoid valves were tested yesterday to try to determine the problem.  The test steps are detailed below:

• New spare valve (NSV) produced audible "click" and valve was seen to physically open when 24V DC driving voltage was applied
• Failed valves of July 25, 2017 (J25), July 10, 2017 (J10), and November 2016 (N16) all did not click or open when driven
• The solenoid of NSV was removed from its valve and installed on each of the failed valves in turn.  The NSV solenoid succesfully opened each of J25, J10, and N16 (the valve bodies and pistons of the failed valves appear to be fine, and the solenoids all faulty)
• Each solenoid's coil resistance was measured: NSV = 187kOhm, J25 = 200kOhm, J10 = O/C, N16 = O/C

On the next maintenance day Tony Tateyama is planning on performing continuity tests on the T2CS:SVSEC control cable and comparing it to the control cable for identical solenoids in the system.

 The #6 male Hansen fittings for the T1-MK1 and T2-MK1 targets (both currently stored in the target storage pit) were replaced today with new units.  Some of the target assembly Hansen fittings had shown signs of leaking and the internal seals are not easily replaced.

"120V" warning labels were put on the micro-switch IN and OUT limit sensors for the profile monitors on both targets.  These are the only micro-switches on the assemblies that still run at 120V after the PLC controls upgrade.

The following work was performed on the T2-MK2 target assembly during the 2017 fall shutdown:

Oct 4, 2017:
- Target moved from beamline to hot cell, measured 661mSv/hr at 0.5m (move went smoothly, having camera views at N/S and E/W angles very helpful)
- Flask tray wet wiped after transfer: 1st wipe 600cpm on 44-2 monitor, 2nd wipe 900cpm (3rd swipe not taken because tray looked clean)
- Flushed target with clean water, then compressed air (positions 0 through 5)
- Male Hansen fittings on the target assembly swapped with new units (in process of replacing all self sealing fittings in T1 and T2 water system)
- Checked all micro-switch positions and potentiometer readback: switches ok, pot value jumpy around position 1 and 2 (known from before), otherwise fine
- Left target overnight at position 4 with 20psi supply air running through to dry water system

Oct 5, 2017:
- Pumped down target with cold trap (~2 hours)
- Baseline leak check performed: 6 mins to reach 5.0E-3 Torr (3.2E-9 Torr*L/s), 21 mins to reach 1.0E-3 Torr (1.4E-10 Torr*L/s)
- Helium spray at 3psi for 0.5s bursts, tested with ladder in position 4
- No leak at any target positions or plugs
- Small leak on left side bellows, or bellows seal to ladder: up to 8.0E-9 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Small leak on right side bellows, or bellows seal to ladder: up to 2.0E-8 Torr*L/s response, then quick recovery  (known from previous leak checks, same magnitude)
- Beam spot photos taken, position 1 (only target used this running period) has fairly well centered beam spot (see below)
- Position 4 target removed ("JAN29'97"), was very difficult to remove because one water tube stuck in ladder (this target was installed on ladder in 1997!  First beam 2014)
- Used hammer in hot cell to remove which eventually broke the target water tubes from the wire EDM housing
- sealing surfaces cleaned with burgandy Scotchbrite on electric dremel tool
- Inspected sealing surfaces and ladder fitting threads using Nikon level: all appeared clean and undamaged
- Installed new 12mm beryllium target at position 4 ("Jul.06#4"), no issues with install
- Leak check repeated: reached 1.0E-3 Torr after 3 minutes (1.3E-8Torr*L/s)
- Down to 3.5E-9 Torr*L/s after 18 mins, began test
- No response at all target positions, checked left side bellows: same response as above
- Beam spot profile at Position 1 measured with Nikon level
- Vertical and horizontal positions of new Position 4 target measured
- New profile monitor cassette retainer pin installed (plate hole had to be opened up to 0.375" diameter), used vice grip pliers clamped onto clevis pin in order to install

Oct 6, 2017:
- Profile monitor actuation tested: start movement 5psi, fully IN at 35psi
- Profile monitor limit switches tested, ok
- Target transported back to beamline, no issues
- BL1A pumped down, Edi reported pump-down normal
- Electrical check performed on T2 profile monitor from mezzannine: Sheng-Li reported results normal
- Cooling system started: no change to BL1A vacuum
- Target info sheet and target elevations sheet delivered to Main Ops
- XTPAGE 7L information updated

 Around 10pm on Sunday Oct 22 beam was tripped multiple times due to low flow for the T1 collimator cooling circuit (B1A:T1:FGCOL).  Temperature of the collimator (TC7COL, TC8COL) was monitored with reduced beam current and no increase was observed leading to the conclusion that flow was OK and the flow gauge was faulty.  Normal operation continued until around 1:30am on Oct 26 when the collimator thermocouples both rose quickly, causing the beam to trip when they reached 60C.  At this point we concluded that there was no longer flow in the cooling circuit.  1A was defined off and shielding was removed down to the T1 cooling package.  The problem was found to be the solenoid valve which would open when actuated, but close in less than a second.  The flow meter was checked, appeared fine, and the paddle wheel was changed anyways.  The solenoid part of the valve was replaced (valve body not changed), which solved the problem.  Flow returned to 0.7gpm and the collimator thermocouples returned to normal value.  It is suspected that the valve was fluttering open and closed causing the decreased flow observed before, and then the valve finally failed closed at 1:30 on the 26th.