A strategy for alignment of the T2 protect monitor, T2 target, and T2 Collimator A was discussed today and decided upon by Yuri Bylinski, Isaac Earle, Yi-Nong Rao, and Roman Ruegg.  The purpose of the adjustment is to try to reduce the amount of beam lost as it passes through the T2 area (presumably being blocked by the collimator because it is not possible to steer the beam through all three devices in their current configuration).  Keeping in mind that changing the elevation of Collimator A would be prohibitively time consuming it was decided that the protect monitor and target both be adjusted upwards 3.0mm compared to where they were during the 2016 running period.  This will result in the following elevations (all w.r.t. the T2 target flange).  The T2 profile monitor elevation will also be adjusted to match the target elevation.  

T2 Protect Monitor:  1833.5mm (1833.2mm from protect monitor flange)

T2 Target: 1836.5mm 

T2 Collimator A:  1834.7mm  (1834.5mm from collimator flange;  This is the current collimator elevation, measurement taken by Tom Lyth in Feb 2009)

These values are similar to what were used in 2009 during which time TNF neutron flux rates were acceptable (>800), and centered beam spots were achieved.  During the 2016 running period the beam spot on target was also roughly centered.  At both times the target was approximately 3mm lower than the protect which seems to indicate the beam is travelling at a downwards angle, possibly not correctable with available tuning.  This is a departure from the strategy in the last few years when the target and protect monitor were adjusted downwards to bring them closer to the beam plane (they had risen due to movement of the T2 monolith).  This is now thought to have been a mistake because it made it difficult to steer beam through the collimator.

Beam will be started with T2 in Position 2 (new target, never seen beam), then after a few days the target assembly will be moved to the hot cell to check the beam spot to confirm alignment and make changes if the strategy needs adjusting.

A PDF copy of Yi-Nong's summary is also attached.

 The following changes were made to the T1/T2 PLC control system:

- De-glitching system investigated and Tony discovered that it was programmed incorrectly:  the de-glitch time was observed only when a PV changed from bad state to good, not the opposite (meaning there was actually no de-glitching in place since system start-up in 2016!).  This was reversed and the following de-glitch times were programmed:  100ms for trip thresholds except 300ms for flow meter trips, and 2s de-glitch on all warning thresholds.  The new system was tested with the PLC in simulation mode and seems to be working correctly.  This should eliminate or significantly reduce nuisance warnings and trips from the system.

- PLC system code was modified so that the target version displays '???' when no target is installed instead of defaulting to 'MK1'  (all target positions registered as 'plugged' with no target installed, so there was no danger to the system, but this was still confusing to operators)

The 18' shielding block which lies west of the T1 cooling package was removed on Friday Mar 24 so that vacuum equipment could be craned into the 1A tunnel.  During replacement, the flexible SS hose for T1 target return water got caught under the block and was ripped apart.  Workers in the area did not realize this had occurred.  The cooling package tripped off when the hose ripped (likely due to low water flow or low supply pressure).  In total approximately 6.5L was lost which flowed to the 1A tunnel below.  The hose connections at the target and cooling package do not appear to have sustained damage.  Plans are under way for a replacement hose to be fabricated and installed.

Update March 30, 2017:  The damaged hose has been replaced with a new 20' long braided SS hose with 3/4" male NPT fittings both ends.  Both air supply lines for T1 profile monitor actuation (1/4" copper) were also damaged during the Mar 24 block move and they have now been replaced.  All hoses, cables, etc on the west side of the T1 monolith have been re-routed or secured to reduce the chance of this happening again in the future.

 On March 13th in the morning the Meson Hall Hot Cell Lab active sump reached the high level sensor.  The system responded as expected:  high level panel warning light illuminated, siren sounded, and MCR notified via "B1 RCR1 RM6 WARN".  A 1L sample was delivered to RPG for analysis.  After approval, the sump was pumped out over the following two days.  The procedure in Document #64834 was followed for sampling and pump-out.

The following work was completed on the T1 cooling package:
- water drained to BL1A holding tank
- all resevoir o-rings changed
- all cuno filters changed:
     M15 permanent magnet filter < 5uSv/hr at 0.5m, dark grey colour
     Target inlet filter 210uSv/hr at 0.5m, light brown colour
     Resin filter, <5uSv/hr, looks clean and new
- package re-filled and started
- demin circuit flow tuned to 1.0gpm
- #6 female Hansen fittings at target end of flexible hose swapped for previously used SS models (zinc coated steel models (PO# TR207041) were accidentally ordered and installed last shutdown and had black corrosion on the inside)
 

The T2 cooling package was also drained to the 1A holding tank.
After approval from RPG the holding tank water was released.

Note: the appearance of the used cuno filters is consistent with previous years.

 The beam spot on the 10cm beryllium target in Position 1 on T2-MK2 (Target ID: "OCT.06#3") was measured and photographed yesterday.  A fairly well centered spot was measured to be approximately 4.6mm wide x 5.2mm tall on the entrance window and 5.7mm wide x 7.6mm tall on the exit window using the Nikon micrometer level.  This target first saw beam on June 16, 2016.

A PDF scan of the beam spot measurements and photographs of the beam spots are attached.

 The T2-MK2 target assembly was moved from the beam line to the hot cell today.  A maximum field of 203mSv/hr was measured at approximately 0.5m

A steel shielding plug has been installed in the T2 hole to reduce shine and allow the T2 volume to be pumped down.

 At approximately 11am today the T2 cooling package heat exchanger secondary side water flow suddenly dropped from ~11gpm to 0.  This was confirmed to be a real reading by observing that with no secondary cooling both the inlet and outlet target water temperatures rose steadily with beam on the target.  While standing on the blocks in the T2 area and having a controls expert (wither "superuser" access mode) actuate the secondary side solenoid independently it was confirmed that no "click" was heard when power was supplied to the valve.  No other changes were observed with other parts of the CuALCW system, therefore it was concluded that the secondary side solenoid valve (B1A:T2CS:SVSEC) had failed.

Blocks were removed above the T2 cooling package, the CuALCW lines were valved off and drained, and the failed valve was removed.  A new valve was installed.  A section of 3/4" SS tubing was replaced with flexible 21" long 3/4" diameter SS hose to allow re-installation (alignment could not be achieved with rigid tubing).  Water supply and return were restored, and the system was started.  Because of the new section of tubing, the flow through the secondary side dropped from ~11gpm to ~9gpm.  This should not have a significant effect on operation.  No leaks were observed.

Total dose for Isaac: 0.15mSv, for Keith: 0.03mSv

 An expansion tank high level warning has been added to the T1/T2 control system under Work Request #4011.  The purpose for this new warning is to alert staff of an unexpected water increase in the unlikely scenario that a leak develops in the heat exchanger between the two fluids and CuALCW on the secondary side enters the target water system.  The new thresholds for high level warn, low level warn, and low level trip have been set at 40cm, 30cm, and 25cm respectively.  The nominal water level is 35cm.  All thresholds were tested by adding water to the tank, or removing it.  All responded as expected.

 The T1 cooling system inlet pressure transducer (EPICS ID: B1A:T1CS:PGIN,  Omega part # PX315-100GI) began failing in August (exhibited gradual decline in pressure with no corresponding change to flow rate or outlet pressure).  A new transducer was installed on Sept 30 2016 which fixed the problem.  Cyclotron Fault Report #9238 was returned and the temporary bypass on the warning level was removed.

 The following work was performed on the T1-MK2 target assembly during the September 2016 Mini-shutdown:

Monday Sept 26:
- Target moved from beamline to hot cell (28mSv/hr at 0.5m)
- Position 4 beam spot on entrance and exit windows photographed (attached)
- Target flushed with fresh water, positions 0, 4, and 5 blown out with air, then left overnight with air blowing through position 0 at 10psi supply
 

Tuesday Sept 27:
- Pumped on target water system using lab leak detector:  reached 1.0E-3 Torr and E-8 Torr*L/s range in < 5 mins
- Initial leak check performed: TPP < 1.0E-3 Torr, Base leak rate: Low E-9 range, 3psi helium supply, ~0.5s bursts
- Response up to 3.6E-6 at lower left side delta seal (when facing ladder), up to 4.9E-7 for lower right seal  (note: T1 area at good vacuum level prior to mini-shutdown),  All other positions no response, or response in low E-8 range (assumed drift or insignificant helium leak)
- Plugs at positions 1, 2, and 3 removed.  All threads inspected, brushed with wire brush, rinsed with acetone, and blown dry with compressed air
- New targets installed:  Serial #101 at Position 1, #102 at Position 2, and #103 at Position 3  (Position 1 threads significantly stiffer than others, but no major issue)
- Leak check repeated: similar response from all locations including all beryllium target windows  (concluded suitable for use in beamline)
 

Wednesday Sept 28:
- Profile monitor actuation checked in hot cell:  movement starts at 27psi, fully actuated at 48psi, limit switches functioned properly
- All target ladder micro-switches checked, ok
- Vertical and horizontal target position measurements taken using procedure in Document #50194  (PDF attached)
 

Thursday Sept 29:
- Target returned to beam line and all services connected
- Target info sheet and elevations delivered to Ops, XTPAGE 7L updated
- T1/T2 PLC information updated with new cassette information
 

Friday Sept 30:
- Vacuum pumped on the T1 volume.  Quickly 200mTorr (threshold for T1 Vac OK signal).  Edi reports pump-down slower than usual, will evaluate on Monday.
- T1 package started (no change in BL1A vacuum)
- Profile monitor actuation checked from control room (actuated normally)
- T1 profile monitor and protect monitors checked by Mike Russel and Victor Verzilov:  profile monitor normal, protect monitor signals low but Victor expects it is fine and give ok to cover

 

To address the deficiencies identified in E-Logs #204 and #205 the old active sump high level switch was replaced with Omega LV164SS model (316SS stem and float, 4' length, standard mount) using adaptor flange TRH1316 to mount to the top flange of the sump.  A new electrical box was also installed (circuit diagram attached in PDF format).  The circuit was designed by Don Jackson and wired by Travis Cave.  Connection to the RCR Lab PLC was done by Dave Morris.

The switch was tested using the following procedure:

1.  The switch was triggered by submerging the base in water until the float lifted.  This was confirmed to be equivalent to disconnecting the CPC-4 connector for the switch from the new electrical box
2.  Switch connector disconnected:  Local alarm sounds in HC Lab, red "Sump High" LED light illuminates, green "Sump Good" LED turns off, RCR PLC reports high level (cannot be reset), main control room "MHESA B1 RCR1 RM 6 WARN" displays NOT OK  (this will sound every 3 minutes until the RCR PLC high level alarm is reset)
3. "Alarm Silence" button pressed on new electrical box:  Local alarm stops, LEDs on electrical box do not change, RCR PLC still cannot be reset
4. Switch connector reconnected:  "Sump High" LED turns off, "Sump Good" LED turns on, RCR PLC can be reset
5. Power to the electrical box removed by switching off the breaker:  All LEDs turn off, no alarm, RCR PLC reports high level

Pump out of water from the MHESA Lab active sump to the HC Lab active sump was tested while observing the HC Lab sump.  The water entering the HC lab sump did not splash on the new switch excessively, and did not cause any false alarms.

The new system will continue to be tested annually according to the calibration schedule.

 A new version of the T1/T2 PLC code has been uploaded to the PLC CPU.  The main change in the new version is a fix to a bug related to the target MK# which may have caused the previous CPU crashes.  The EPICS interface now needs to be updated so that no target information is displayed on the EPICS screens when there is no target installed, or jumpers for both MK1 and MK2 are detected.  This must be tested during the next maintenance day.  The appropriate tests are Section 6.4.1 Test #5 and #6 from Document #118467

Further tests were done on the active sump high level float switch today.  The switch was triggered 1 time by filling the sump, and approximately 8 times by tilting the sensor manually.  A local alarm sounded in the hot cell lab each time, but only one time in the RCR Lab and MCR (this occurred on the final trial).  Fault report #9099 has been submitted to Controls Group.  Although the existing float sensor did cause a local alarm when the sump was filled, it's operation did not seem ideal:  it only triggered when completely horizontal, which wasn't until water was several inches above the base of the switch;  in addition it appeared that it may be possible for the switch to become stuck under the horizontal portion of the sump (possibly even in an untriggered orientation).  For these reasons a replacement level switch will also be investigated.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

 The high level sensor for the Meson Hall Hot Cell Lab Active Sump was tested today and worked as expected.  When the sensor is tilted (as occurs when the water level rises above it) an audible alarm sounds in the hot cell lab, and the Safety Panel in the MCR also alarms with "MHESA B1 RCR1 RM 6 WARN" displaying NOT OK.  This alarm sounds every 3 minutes in the MCR until the PLC outside of the RCR lab is reset.  When the sensor was tested on the RCR lab PLC there was a message indicating that the MH Hot Cell Lab Active Sump is full.

It is still unclear why the operators did not receive this alarm when the sump filled and overflowed last night (or possibly they did receive it, but it was ignored or silenced).  The operators from that shift are currently off until this Saturday, so they will be contacted next week to try to learn more.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

 A call was received from the MCR at approximately 21:30 on Monday June 27th to report that active drains in the east end of the Meson Hall were backing up and overflowing.  I came in and found the leak coming from a broken spray nozzle in a sink in the rad waste area.  The leak was stopped at which point there were puddles between 1-2m diameter around the active sump top flange, the drain directly south of the warm cell water windows, and the drain in the counting room south of the clean room.  There was lots of water on the floor in the rad waste area as well.  No water was found in the TR-13 area.  A swipe was taken from the puddle around the active sump top flange, which was not above background level.  Robin Swanson (ops) contacted Joe Mildenberger who gave permission for the sump to be pumped out after a water sample was taken.  Robin took a water sample and left it for analysis by RPG, the sump pump was started and ran overnight.  At approximately 8am the next morning the sump was almost empty and the pump was stopped.  Floor swipes were taken where there had been puddles which were now dry - no contamination found.  The areas in the hot cell lab which had been wet were mopped.

 With-beam commissioning tests were completed today by Tony Tateyama and Isaac Earle.  The test results are attached in PDF format.  These tests are described in Section 7 of the Commissioning Plan (Document-118467 Release #2).  These tests were not part of the official commissioning, but intended as an additional check of system functionality with beam on, and to confirm that the CCS current protection system performs as expected.

Official commissioning of the T1/T2 EPICS system was completed in April 2016, with the results released in Document-131503.

 Position #4 on the T2-MK2 target has now reached the maximum beam time.  The target was moved to Position #1 at the nominal elevation.  This target should be used for the remainder of the 2016 running period.

The T2-MK2 target at Position 4 has been adjust to approximately +1.5mm offset.  This causes the ladder to be moved downwards, raising the beam spot location on the target window.  The potentiometer ratio at this offset is approximately 0.810 (> 0.8059 nominal ratio for position 4, and therefore closer to Position 5)

The purpose of the offset is to better center the beamspot on the target window.  Target window photographs showed the beam spot being slightly below center (See E-Log #197), and target ladder elevations taken in Spring 2016 showed the Position 4 target approximately 1.0mm above the profile monitor center.

MCR has been instructed to continue running with this offset, and a message with details on how to do so was posted to their Plone site.

The Remote Handling Hot Cell Lab Active Sump has been released to city sewer following the procedure in Document-64834.  The water was close to the sump high limit switch (approximately 5800L).  Before release the water was sampled, tested, and approved for release by RPG.  The tritium analysis form has been filled out and returned to RPG,.

It took approximately 5 hours to empty the sump with all valves fully open and the water flowing through the 10" Cuno filter.