After the monitor assembly was moved to the hot cell, electrical checks were performed by Bill Rawnsley which showed the same results as when tested in the beam line.  A new monitor was installed (supplied by Probes Group), the tests were repeated, with all results appearing normal.  The monitor assembly was transported to the beam line this morning and installed at the T2 monolith.  All cables were attached and the flange was bolted down.

Bill was notified by e-mail and it was suggested that he perform one final electrical check from the mezzanine before the monolith is covered.

On Monday Oct 5th a decision was made to uncover and remove the T2 Protect Monitor due to faulty readings.  The monitor had been behaving strangely since the September shutdown, and had been getting worse over time.

- Blocks were removed over T2 starting Monday afternoon.  It is necessary to remove the narrow 12' block and two 6' blocks west of the T2 plug block in order to fit the flask frame in for protect monitor removal

- The monitor was moved to the hot cell by 7pm on Monday.  Pierre was operating the crane and there were no issues with the move.  (monitor measured 15mSv/hr at 0.5m)

Oct 6:

- Monitor inspected: a thin piece of foil was found jammed into the entrance side of the monitor (see photos) & a heat or burn mark was seen on the exit side above and to the south of the plate gap.  It is suspected that this material traveled down the beamline at high speed when there was a vacuum burst during the September shutdown.  In light of this information it was decided to remove the T2 target and T1 target as well (see following e-logs)

- Monitor elevation measured: 1835.8mm (april 2015 measurement was 1836.1 --> ok)

- Old monitor cassette removed, new one installed.  Elevation checked: 1832.2mm - did not match, removed, decontaminated, and returned to Probes for adjustment

- Adjusted monitor installed, elevation checked: 1835.8mm --> ok

- Electrical check done by Probes group at hot cell: found left plate shorted to ground

- Cassette removed, decontaminated, and repaired by Probes Group

- Cassette reinstalled, and electrical check repeated --> ok

Oct 7:

- Monitor elevation re-checked: 1836.2mm --> ok

- T2 protect returned to beamline, bolts installed, cables connected

Oct 8:

- Vacuum pumped on T2 volume after T2-MK2 target returned (see next e-log):  Only reached 2 Torr and a leak was heard coming from T2 protect flange.

- T2 protect was unbolted and lifted ~1in for cleaning of the flange base and o-ring

- The T2 volume then pumped down normally

- A helium leak check of all flanges on the T2 monolith was done by Vacuum Group --> no leaks found

 The T2 Protect Monitor was transported from the beam line to the hot cell this morning using the transport flask.  The move went smoothly.  The only difficulty was locating the yellow flask alignment frame to the T2 monolith as the T2 protect holes on the yellow frame are tighter than for other devices.  It was necessary to remove four screws that secure the top flange of the T2 protect monitor assembly to the flange on the monolith.  These screws are stored on the top of the monolith in a labelled bag.  The T2 protect o-ring was in good condition, but was replaced anyways, and a blank-off was installed over the hole so that vacuum can be pulled on the T2 volume.  A maximum field of 15mSv/hr was measured at 0.5m from the monitor while lifting.

Probes Group may now begin testing / repair of the monitor.  Bill Rawnsley has been notified.

Photos attached.

Update (March 19, 2014):  Bill Rawnsley thinks that the damage was caused by mis-steered beam (not from water).  He points to the evidence of heating in the attached photo.

When beam was started in mid-April, it was observed by Operations Group that with the beam centered vertically within the protect monitor, the profile monitor showed the beam spot to be approximately 6mm high.  The protect monitor was changed during the shut-down due to a failure in November 2013, so it was suspected that the new protect monitor was not aligned properly.  The beam spot appeared to be dangerously close to the top weld of the target window so it was decided the target assembly would be removed for inspection to visually confirm the suspected misalignment before increasing beam current.  Since start-up, a maximum beam current of approximately 30 micro-amps had been used on the 10cm beryllium target in position 4 which previously had not seen beam.

On Monday April 28th, the T2-MK2 target was transported from the beam line to the hot cell.   The target assembly was measured to be 298mSv/hr at 0.5m.  A clear beam spot was visible on the entrance and exit windows of the target approximately 6mm high of vertical center (photos attached).  Target and profile monitor elevation measurements were conducted, and confirmed those previously taken on the target by Tom Lyth.  This exercise confirmed that the position 4 target and profile monitor were aligned properly, and that misalignment of the protect monitor was the cause of the beam being high on the target.  It was then decided that the protect monitor assembly would be removed and a new monitor cassette installed to remedy the problem.

On Wednesday April 30th the T2-MK2 target was returned to the beam line and the protect monitor assembly was transported to the hot cell.  A maximum field of 15mSv/hr at 0.5m from the protect monitor was measured.  Elevation of the monitor cassette was checked, and was found to be 6.8mm above the elevation of the position 4 target.  The monitor cassette was removed from the assembly.  The cassette was approximately 100μSv/hr at 0.5m and 2mSv/hr on contact, with 200cpm measured from a swipe of the outside surfaces (not the inside plates).  The cassette was returned to Probes Group for possible use in the future.  A new monitor assembly was supplied by Scott Kellog on Friday morning with the 6.8mm adjustment.   The monitor was installed on the protect monitor assembly.  Elevation was checked and found to be within 0.5mm of the target elevation.  After installation an electrical check was performed by Dave Cameron and Mike Russel.

The protect monitor assembly is ready for installation in the beam line on Monday morning.

- T2 Protect Monitor moved from the beam line to the hot cell on Monday April 20th
- Elevation of the plate center measured at 1839.0mm from the base of the vacuum flange
- Old monitor cassette photographed (attached).  Heat mark noticed on exit side approximately 10mm above and 5mm south of plate centers (cause unknown, and no matching mark on entrance side)
- Old monitor cassette removed, new one installed.  New elevation 1836.1mm (2.9mm upward shift - 3.0mm was requested)
- Electrical check performed by Probes Group on April 22 - They reported that it looks normal
- T2 Protect Monitor returned to the beamline April 22 in the evening.  Cables plugged in and flange bolts installed
- Vacuum pumped down normally on April 23 in the morning
- Probes group repeated the electrical check after installation from the 1A mezzanine - They reported that it looks normal
- The old monitor cassette will be left in the hot cell for possible future use

T2 Profile Monitor actuation was tested today from the control room and all is working properly.  The position (IN / OUT) also displays correctly.
To my knowledge nobody has looked at this system, so it is a mystery why it suddenly started working.  This e-log will be updated if any explanation is found.

Actuation of the M20 beam blocker and T1 Profile Monitor were also tested and are working as expected.

The T2 monolith may now be covered.

Update:  A blown fuse in the control racks was replaced yesterday by Dave Cameron, which explains the change.

The T2 profile monitor cannot be actuated from the control room.  When the solenoid valve that supplies air to the air cylinders is powered directly, the monitor moves in normally, therefore a controls issues is suspected.

Additionally the monitor position shows neither IN nor OUT.  This does not change when the monitor is at either limit.  Proper actuation of the target micro-switches has been confirmed.  The T2-MK2 target is currently installed, but this was also an issue in Nov/Dec 2013 with the T2-MK1 target installed.

Fault report #7276 has been filed, targeted at Cyclotron Controls Group.

The T2 plug block was replaced and the M9 and M20 beam blockers were tested.  The plug block did not interfere with the travel of the beam blockers.  There was no change in the T2 vacuum during BB actuation.

Initially there was some confusion about how to actuate the beam blockers.  The following was learned:

M9BB:
- The M9 beam blocker is no longer part of a safety system because there are separate blockers for the M9A and M9B channels
- The M9 beam blocker does not need to be 'enabled' before actuation
- The M9 beam blocker can be actuated from the M9B ASU panel located on the B2 level
- The M9 beam blocker cannot be actuated from the control room.

M20BB:
- There are no additional blockers in the M20 beam line, so the M20BB functions as a safety system and must be lowered before the M20 area can be accessed
- The M20BB must be 'enabled' by controls before it can be actuated
- The M20BB can be actuated from the control room or the M20 ASU on the B2 level

The T2 P1 pressure transducer (Omega PX315-100GI) and T2 Q1 Proteus flow meter (0150SSTF3) have been replaced with new units to address issues with noisy signals.

 While leak checking components in the M20 front end on May 25th, the top of the T2 monolith was also checked by flooding the top of the monolith with helium while it was covered with plastic sheeting.  There was no response on the Varian leak detector indicating no significant leaks from the flange seals, shaft seals, and seals for upper services on the T2 target, T2 protect monitor, M9BB, M20BB, and M8BB/Col.A

The following work was performed at the T2 cooling package and target between Jan 25 - Feb 3, 2016:

- Cuno water filters removed (2), and new filters installed
- Target water solenoid, H.E. secondary side solenoid, Collimator A solenoid, and Collimator B solenoids removed and replaced with new 24V DC versions
- All thermistors removed and replaced with Type K thermocouples
- All Proteus paddle wheels, o-rings, and paddle wheel pins replaced (5 sensors total)
- Target side #6 Hansen QD fittings replaced
- All air QD fittings (6 total) for profile monitor actuation were replaced, some of the flexible hosing was replaced (cracking), and the flexible hose assemblies were checked for gross air leaks
- Top surface of the cooling package resin can support platform was painted (was rusty)
- Resin can swapped (previous resin swap was during Shutdown 2014)
- Target water check valve was removed and tested (working as expected), then reinstalled

- T2-MK2 target (currently in the T2 monolith) purged with air

- All cooling package reservoir o-rings changed
- Target water line #6 Hansen o-rings changed at both ends
- M8 BB / Col. A supply hose Hansen o-rings changed at both ends

- Target water and demin circuit Cuno filters changed
- Used filters double-bagged, labeled, and stored on 1A blocks north of boot box area

- The following Swagelok ball valves removed for service:  Tgt return (X2), Sec return (X2), Demin supply, Demin return (X2), Col A return, Col B return

*** Four #4 Hansen o-rings on package panel still require changing (size was not available from stores)

The lower amplifier for the T2 target station air supply was removed for teardown, inspection, and rebuild. 

The goal of this operation is to understand wear development in the air amplifier over ~13 years of operation, and potentially determine a recommeded service interval.

---------------------------------------------------------------

The T2 volume was vented for an unrelated maintenance operation during this time.

During testing, prior to removing the lower amplifier, both regulators were set to ~20 psi. 

The upper air amplifier resulted in ~125 psi at the outlet while the lower air amplifier resulted in ~120 psi at the outlet.

The upper air amplifier had more audible air flowing out from the muffler than the lower amplifier. 

'Scratching' sounds in both amplifiers were comparable.

The following cycle times were recorded with the M20 BB raised/out (min:sec):

    UPPER: 1:22 / 1:00 / 2:06

    LOWER: 2:30 / 1:46 / 2:12

The following times were recorded to raise the M20 BB (sec): 

    UPPER: 8.36 / 8.76

    LOWER: 8.56 / 9.10

These times will be compared against after completing the teardown and rebuild of the lower amplifier, at which point this ELOG will be updated.

-----------------------------------------------------------------

UPDATE (Feb 23, 2024):

The lower air amplifier was serviced (photos in 'S:\Albert Kong\Shutdown Files\2024\Feb20 T2 Lower Amplifier Cleanup'):

1. Full assembly cleaning.
2. Piston and barrel was cleaned and lubricated, o-rings and plastic components replaced (with new lubricated ones).
3. Pilot valve components replaced (except plug).
4. Check valves (4x) cleaned and plastic/rubber components replaced (with new lubricated ones). 
5. Muffler cleaned (blown out with compressed air).
6. Spool and sleeve assembly cleaned, o-rings replaced (with new lubricated ones), and rubber stopper replaced.
7. Clamping rods tightened to ~17 ft-lbs. 

Note: the piston o-rings were difficult to seat on the piston body/teflon ring. During assembly, we instead seated the o-ring in the barrel on the piston plates (see picture), which allowed the oring to be seated properly, before placing the piston body onto the piston rod.

After servicing, the amplifier was returned to the station, air connections reconnected, and tested. 

Note: it is recommended to do torque-ing of the clamping rods as a final step to simplify mounting of the amplilfier and re-doing connections to the rest of the compressed air system.

The first observation we made was how silently the lower amplifier now operates when cycled: only the exhaust sound from the muffler can be heard.

Note that the piston's motion can be heard when listening ~5cm away from the amplifier barrel. 

The following times were recorded to raise the M20 BB (sec):

    UPPER: ~8.5 

    LOWER: ~7.7s

The following cycle times were recorded with the M20 BB in the out/raised position (min:sec):

    UPPER: 1:05 / 1:04

    LOWER: 1.22 / 1:45

The outlet pressure from the amplifier registers 120 psi with the regulator set only to 15 psi (improvement from previous performance as well as the upper air amplifier's performance).

We will check in on the amplifier next week to see if it still operates silently and can actuate the beam blockers/profile monitor without issue, at which point this ELOG will be updated. 

-----------------------------------------------------------------

UPDATE (Feb 26, 2024):

The lower amplifier was inspected this morning, higher volume sound could be heard from the drum in concert with the motion of the piston, but still much quieter than before servicing.

 At approximately 13:50 today the T2 cooling system secondary side solenoid valve (T2CS:SVSEC) failed to a closed position.  This was the same location where the valve failed on July 10th (E-Log #224), and also in Nov 2016 (E-Log #211).  This repeated failure does not seem like a coincidence, so instead of replacing the valve with a new unit, it was bypassed temporarily until we can determine the cause of the failures.  A 3/4" FNPT coupling was installed in the place of the valve, and a CPC-4 stub was plugged into the control cable with pins 1 and 2 shorted in order to prevent the control system from timing out.  The cooling system has now been restarted and water is flowing properly through both sides of the system.  These failures must be investigated further during the fall shutdown, and ideally a solenoid valve should be reinstalled at this location (currently to shut off water supply to the secondary side the 1A tunnel must be entered to access the CuALCW supply valve).

Prior to shutting off the system for repairs some tests were performed by Tony Tateyama who provided the following summary:  "I wanted to see if there was any possibility of anomalous signals for T2CS:SVSEC drive & loop-back voltages at the PLC BOP.  Approximately 23.8 VDC was seen for the drive signal & slightly less for the loop-back signal.  In addition, about 70 mVAC ripple was also observed.  The same measurements were done for the T1CS:SVSEC voltages for comparison.  Approximately 23.9 VDC measured for the drive signal, and again, slightly less (~23.8 VDC) for the loop-back signal.  About 100 mVAC @ ~8 kHz was also seen. "

The T2 heat exchanger secondary side solenoid failed on July 9 at approximately 19:30 (EPICS ID: B1A:T2CS:SVSEC).  The valve was replaced with a spare and the cooling package is now running normally.

The valve at this location also failed in November 2016:  https://elog.triumf.ca/TIS/RH-Meson+Hall/211  Further investigation will be attempted to determine if there is a root cause to the failure, or if it is a coincidence.

The attached OneNote PDF contains details of the work.  Cyclotron Fault #10119 has been returned.

 At approximately 11am today the T2 cooling package heat exchanger secondary side water flow suddenly dropped from ~11gpm to 0.  This was confirmed to be a real reading by observing that with no secondary cooling both the inlet and outlet target water temperatures rose steadily with beam on the target.  While standing on the blocks in the T2 area and having a controls expert (wither "superuser" access mode) actuate the secondary side solenoid independently it was confirmed that no "click" was heard when power was supplied to the valve.  No other changes were observed with other parts of the CuALCW system, therefore it was concluded that the secondary side solenoid valve (B1A:T2CS:SVSEC) had failed.

Blocks were removed above the T2 cooling package, the CuALCW lines were valved off and drained, and the failed valve was removed.  A new valve was installed.  A section of 3/4" SS tubing was replaced with flexible 21" long 3/4" diameter SS hose to allow re-installation (alignment could not be achieved with rigid tubing).  Water supply and return were restored, and the system was started.  Because of the new section of tubing, the flow through the secondary side dropped from ~11gpm to ~9gpm.  This should not have a significant effect on operation.  No leaks were observed.

Total dose for Isaac: 0.15mSv, for Keith: 0.03mSv

- The new tube array was picked up from Kaltech on Feb 8th (observed vessel holding 60psi on hydro test at their shop)
- Vacuum leak checked at TRIUMF: would not pump down
- Applied 20psi air, and found leaks on both tube sheets using Snoop

- Dropped off tube array at Kaltech on Feb 13th for repairs
- To fix the welds on the manifold side it was necessary to cut open the manifold, then weld on a cap after re-doing the welds
- Calculations were performed to determine the stress on the welds of the new cap.  The weld was found to have a factor of safety of 27.4 (notes attached in PDF format)

- Repaired tube array was pressurized to 65psi with air at Kaltech and placed in a water bath: no bubbles.  Also checked welds with Snoop: no leaks
- Transported tube array to TRIUMF on Feb 20th
- Helium leak checked assembly, no leaks

- Old heat exchanger assembly was disassembled: old o-rings were stiff, but not cracking
- Small active metal chunks and flakes were found in the tube side outlet manifold (~1mSv/hr on contact)
- The old tube array was measured to be 8µSv/hr at 0.5m, and 500µSv/hr on contact at specific locations, which indicates that active chunks of metal may be stuck at certain places in some tubes
- Shell side of old tube array had  brownish/orange buildup, tube side appeared fairly clean (see photo below)

- The new tube assembly was inserted into the H.E. shell (required some filing on baffle welds)
- The welds between the baffle strips and large tube sheet were found to interfere with the flange on the shell as shown below (this was an oversight on the drawings, not machine shop error)

- The tube sheet face was skimmed on the lathe by Maico and Dan Wright to remove the interfering weld material
- Repeated the vacuum leak check after machining: no leaks
- The modified tube array fit properly within the shell as shown below

- All parts were cleaned using Windex, followed by methanol, with special attention to sealing surfaces
- The heat exchanger was reassembled using new o-rings, new fasteners, and NeverSeez compound on all fasteners
- A static water test at city supply pressure (~70psi) was performed on the vessel, and a leak was found coming from the tube stub seal (#319 o-ring)
- An o-ring calculation was performed which indicated that the current configuration only compressed the o-ring 15.4%
- 1.50" OD, 1.080" ID shims were machined from 6061 aluminum at 0.017", 0.022", and 0.029" thicknesses as shown
- The addition of a 0.029" shim was calculated to increase the o-ring compression to 27.7%

- The 0.029" thick shim was installed as shown below

- The vessel was filled with water and pressurized to city supply pressure (~70psi): no leaks
- The pressure was increased to 150psi (1.5X the working pressure) using a Reed hydrostatic test pump supplied by Beamlines Group
- No water leaks were observed
- The vessel was valved off while under pressure downstream of the pump hose
- The following pressure drop was observed over the course of a weekend:
     Mar 1st @ 2:35pm:  151psi
     Mar 1st @ 4:30pm:  142psi
     Mar 1st @ 5:30pm:  139psi
     Mar 4th @ 9:00am:  70psi

- This indicates a slow leak either through an o-ring seal on the vessel, the valve used for pressure testing, or one of the fittings used for pressure testing
- The vessel was drained and pressurized with air to house pressure (~100psi)
- Snoop was applied around all o-ring seals: no leaks found
- The leak check with Snoop indicates that the leak must have been through a valve, on a pipe fitting, or of too low a rate to be of any concern

- The heat exchanger was tipped on its side to allow the shell side to be fully filled with water
- After filling with city water, a ball valve and solenoid valve were installed on the shell side ports to prevent water leaking during transport (see photo below)

- The heat exchanger was bagged and transported back to the T2 cooling package
- Testing of the heat exchanger will be done when the T2 target is reinstalled and the CuALCW system is back up (in approximately 2 weeks time)

Friday Jan 25:
- Both sides of heat exchanger drained, disconnected, and capped
- Heat exchanger transported to warm cell (field from H.E. approximately 15µSv/hr at 0.5m)

Monday Jan 28:
- Both sides of heat exchanger flushed with city water
- Heat exchanger lifted and tilted to drain as much water as possible 

The water level in the T2 cooling system expansion tank dropped to below the trip treshold of 20 cm (ref Cyclotron fault #16123), requiring the beam to be 'defined' off (see attached image).

Some calculations (see attached .html) show that over the period in which the tank level was dropping (from July 12 - Aug03, 2023) approximatley 11L of water was lost. At the end of the 'drop period' the leak rate was at a maximum of approximately 1L/day or 40mL/hour. 

The tank was filled up to 39 cm and we will continue to monitor the water level in the coming days.

We will also plan to enter the BL1A tunnels in the coming maintenance day (Tuesday, Aug 08) to check for poolig water. 

Alternative to a leak, entrapped air in the system may have escaped/been displaced to allowed 11L of water from the expansion tank to drop into the cooling loop.

If the lost 11L indeed occured due to leak, at least the leak will be outside of the vacuum volume since vacuum levels remained stable.

A likely source for a leak would be one of the exchanged valves from last shutdown.

Alternatively one of the not-exchanged valves may have started leaking due to aging/radiation damage.

--------------------------------------------------------------------------------------------------------------------------------------------

 UPDATE: Aug 08, 2023

The expansion tank at T2 was topped up to ~44cm on Friday Aug 04 ~12pm.

Over the long weekend (Friday Aug 04 ~12pm to Tuesday Aug 08 ~7am), ~14cm of water was lost and the water level in the expansion tank went below the low level warning limit (30cm).

Looking at the water level trend, the leak rate seems to be increasing (see attached - rate approximately doubled/trippled to 3L/day or 120 mL/hour).

Maico and Albert entered the BL1A tunnels at Aug 08 ~1pm and found trickling water underneath the T2 cooling package (see attached). Fortunately, no pooling water was found in the BL1A tunnels.

A plan must be developed to decide the appropriate course of action to remedy this problem.

Tentatively we have two choices:

1) attempt to remedy the leak during a maintenance day by uncovering the T2 cooling station. 

2) accept the leak until the mini shutdown in October (requiring ~2-3 expansion tank 'top-ups' every week for 2 months or so).

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 11, 2023

The expansion tank was topped up again to ~44cm on Thursday Aug 10, ~8am.

Since then the water level has dropped to ~34.5cm on Friday Aug 11, ~9am. ~10cm of water was lost within the span of a day, equating to ~6L/day or ~250mL/hour (see attached calculations).

From the data, it seems like the leak has stabilized to this value. 

An SAS job request has been filed to uncover the cooling package during the mini shutdown to fix the leak. Leading up to this, it would likely be a good idea to regularly enter the BL1A tunnel and assess the condition of the leak. 

Additionally, it will be necessary to fill the expansion tank to the brim every two days or so to keep it from tripping the beam.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 15, 2023

We took advantage of an unexpected maintenance window this week to uncover the blocks surrounding T2 and identified that the source of the leak was a 'pinched' o-ring on the Q2 flow meter (demin water return - see attached pictures). 

The bottom set of screws on the flow-meter o-ring plate was loose when we took it apart. The o-ring may have been pinched when it was assembled back in 2022, making it difficult to establish even loading on all screws.

The bottom screws then creeped loose over time, creating the leak. 

The o-ring was successfully replaced, the pump was turned back on with no immediate leaking at the service flow-meter, and the expansion tank was filled to 39cm.

We will monitor the water level overnight and inspect the cooling package for leaks before deciding the next steps tomorrow morning (if no leaks found, we will proceed with closing up the T2 area).

We will specifically asess whether the puddle underneath the main tank (see picture) dries up in addition to tracking the expansion tank water level.

--------------------------------------------------------------------------------------------------------------------------------------------

UPDATE: Aug 16, 2023

The water level remained stable overnight, up-close visual inspection of the serviced flow-meter and the cooling package in general confirms that the leak has been fixed.

The fill rate of the active sump in XTpage P2 also leveled.

See attached html document (updated calculations and notes) for relevant information.

Work to re-place the blocks started after confirmation of the fix.

While checking operation of the M9 and M20 beam blockers earlier this week, a strange noise was noticed coming from the T2 cooling package area.  It is a squealing sound that starts when M9BB, M20BB, or T2 profile monitor are actuated and continues for approximately 30 seconds, then stops.  The source was eventually narrowed down to somewhere on the air supply line coming from the air amplifiers on the south mezanine.  When the air amplifiers are adjusted below ~95psi, the noise stops (the standard setting is 110psi).  A 20psi Nupro check valve on the line was suspected to be causing the noise.  It was removed, rebuilt, and reinstalled, but there was no improvement.

A few of the blocks directly east of the T2 cooling package must be removed so we can have access to the lines in that area.  We will continue our investigation next week.