I have filled the water package on T2 with DI water. The expansion tank is at 38.1cm. This may change when the pump is turned on.

 We moved M20 back to the T2 monolith and re did the connections for air + limit switches + T2 profile monitor air.

Note: T2 profile monitor electronics sparked when we moved M20 out initially. (Update: Shengli Liu from probes group performed a test on the T2 profile monitor electronics on Mar24 and found that they are working properly)

M20 started leaking when we tried to bring it to vaccum - from the monolith o ring seal and not the main shaft seal - will look to correct on monday.

Verified through measurement that the replacement o ring was the correct size (#268 0.139" diameter compared to the 0.131-0.137" that was remved - likely shrunk over use).

Update (Mar 06, 2023): M20 was lifted ~5 in above the monolith flange and we identified that the leak was caused due to damage to the o ring (image attached).

The o ring likely damaged when M20 was brought into contact wit the monolith flange, lifted up again, and brought down for a final time before vaccum check.

The above procedure was done because the flask tends to stall close to the lowermost position and prevents unlatching unless lifted systems are lowered fully.

This will be noted in updated procedures.

 T1/T2 valves were not leaky still (only a few droplets) - pending T2 cooling system pump on for final leak verification.

 We tested the actuation of the M9/M20 beam blockers on the T2 monolith (3-4pm).

Vacuum levels remained steady throughout so the M20 o-ring replacement that was performed this shutdown was successful (see strip of 1ACG4 - vacuum gauge for T2 systems interlock).

Note: M20 was actuated by contacting the control room, while M9 (labeled T2 blocker) was actuated through the physical ASU on the ground level of the meson hall (see picture).

For future shutdown work: the air supply valve must be kept open (tab lifted up), otherwise the solenoids won't see pressurized air. 

We ran into issues because the tab on the valve broke early into the shutdown and we didn't realize that it had to stay in the open position. 

Note: The flow on the T2 demin line went up to 1.5 gpm yesterday when we were on the blocks. The needle valve was likely nudged on accident.

Update:

1) the T2 demin flow was corrected and the air supply tab was replaced - see picture M20

2) T2 BB actuated successfully with replacement tab - vacuum remained stable

3) slow leaking valve connections were tightened again

4) water level in expansion tank corrected to just below 40 cm 

We tested the air amplifiers by actuating the beam blockers and profile monitors on the T1 and T2 target stations. 

The blockers and profile monitors were each actuated (brought up/out and down/in fully) > 5 times and vacuum remained stable throughout (see attached).

The cycle rate of the air amplifiers (time between 'puffs') under steady conditions (target devices kept in the out position) was measured to be between 60-90 seconds, with 90 seconds corresponding to when the 'flow control' ball valve is fully shut and ~60 seconds when the valve is opened (either partly or fully). 

When a target device is brought up, a cycle event will occur shortly after the device reaches the out position.

This cycle time is much longer than last reported in 2014 (see elog #119), by a factor of 2-3. 

The top amplifier sounds different from the bottom amplifier (cycles between being 'rattly' and 'quite' between cycles).

Seeing how the beam blockers and profile monitors were actuated smoothly with the air amplifiers, we will keep the system running off the lower air amplifier with the 'flow control' ball valve kept open (not shown).

At some point we will look into disassembling the top amplifier to see why it sounds differently between cycle events.

For reference, both amplifiers read 20psi at the regulator and 120 psi to the target station. 

The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops. 

The HEPA filter in the nuclear ventilation system for the Meson Hall Hot Cell was changed and tested by mechanical services under work permit C2023-04-25-14.

They have given approval to operate again. The system is re-energized.

The level sensor for the active sump in the RH lab was tested.
The level sensor is working properly and is operational.
 

Today we re-lubricated the o-rings on the top air amplifier for T1/T2, and replaced the o-rings on the bottom air amplifier with new and also lubricated o-rings. 

We used Haskel lubricant #50866.

The bottom air amplifier spool assembly was noticably dirtier than the top (see attached). and we noticed some worn spots on the o-rings. This was not the case with the top air amplifier spool (it has not been used outside of testing since being fully serviced in 2012).

Replacing the o-rings did not result in any perceptible change to the cycle rate and sound of either air amplifier (~50 sec/cycle, same sounds made by the piston before and after servicing). 

We initially planned to also remove the sleeve on the amplifiers (see image) but could not do so easily. We will contact Haskel to get some input regarding how best to remove the sleeve.

The plan is to perform a full teardown of the lower air amplifier next shutdown (2024).

 The following tasks were performed:

• All electrical, water, and pneumatic connections to the T1 target was reconnected.
• 3x and 2x CUNO filters for T1 and T2 respectively were changed (O-ring for the housings were not changed).
• All hansen fittings on the front mounting plates of the T1 and T2 cooling packages were inspected.
• All proteus paddle wheels (Q1-5) for both T1 and T2 were replaced. 
• The main reservoir drain valves on both cooling packages were closed in preparation for re-filling.
• All other ball valves in the cooling packages were returned to open, also in preparation for re-filling.

Two buckets containing (primarily) active water from the CUNO filters were brought closer to the boot up area, to be moved to the active sink in the MH hot cell lab in the coming days (pending ELOG update). 

Spent shafts, paddle wheels, and o-rings from the proteus flow meters will be checked and appropriately tagged/disposed of, also in the coming days.

------------------------------------------------

Some observations for future shutdown work:

• The drain port at the bottom of the CUNO filter housings can be used to verify that the filter line has been successfully closed off.
• Prepare several lengths of flathead screws to service the proteus flowmeters.
• Prepare spare locknuts and screws for the proteus flowmeters in case they become stripped or a locknut is dropped.
• Stick to well-fitting gloves to make using screwdrivers, picking screws, etc. easier.
• Possibly a pipe or coupling could be used with the tool for the CUNO filters to make it easier to loosen the CUNO filter nut in hard to reach areas.
• If possible, it may be a good idea to replace the o-rings on the CUNO filter housings in future shutdown operations (dash # to be identified). 

The lower amplifier for the T2 target station air supply was removed for teardown, inspection, and rebuild. 

The goal of this operation is to understand wear development in the air amplifier over ~13 years of operation, and potentially determine a recommeded service interval.

---------------------------------------------------------------

The T2 volume was vented for an unrelated maintenance operation during this time.

During testing, prior to removing the lower amplifier, both regulators were set to ~20 psi. 

The upper air amplifier resulted in ~125 psi at the outlet while the lower air amplifier resulted in ~120 psi at the outlet.

The upper air amplifier had more audible air flowing out from the muffler than the lower amplifier. 

'Scratching' sounds in both amplifiers were comparable.

The following cycle times were recorded with the M20 BB raised/out (min:sec):

    UPPER: 1:22 / 1:00 / 2:06

    LOWER: 2:30 / 1:46 / 2:12

The following times were recorded to raise the M20 BB (sec): 

    UPPER: 8.36 / 8.76

    LOWER: 8.56 / 9.10

These times will be compared against after completing the teardown and rebuild of the lower amplifier, at which point this ELOG will be updated.

-----------------------------------------------------------------

UPDATE (Feb 23, 2024):

The lower air amplifier was serviced (photos in 'S:\Albert Kong\Shutdown Files\2024\Feb20 T2 Lower Amplifier Cleanup'):

1. Full assembly cleaning.
2. Piston and barrel was cleaned and lubricated, o-rings and plastic components replaced (with new lubricated ones).
3. Pilot valve components replaced (except plug).
4. Check valves (4x) cleaned and plastic/rubber components replaced (with new lubricated ones). 
5. Muffler cleaned (blown out with compressed air).
6. Spool and sleeve assembly cleaned, o-rings replaced (with new lubricated ones), and rubber stopper replaced.
7. Clamping rods tightened to ~17 ft-lbs. 

Note: the piston o-rings were difficult to seat on the piston body/teflon ring. During assembly, we instead seated the o-ring in the barrel on the piston plates (see picture), which allowed the oring to be seated properly, before placing the piston body onto the piston rod.

After servicing, the amplifier was returned to the station, air connections reconnected, and tested. 

Note: it is recommended to do torque-ing of the clamping rods as a final step to simplify mounting of the amplilfier and re-doing connections to the rest of the compressed air system.

The first observation we made was how silently the lower amplifier now operates when cycled: only the exhaust sound from the muffler can be heard.

Note that the piston's motion can be heard when listening ~5cm away from the amplifier barrel. 

The following times were recorded to raise the M20 BB (sec):

    UPPER: ~8.5 

    LOWER: ~7.7s

The following cycle times were recorded with the M20 BB in the out/raised position (min:sec):

    UPPER: 1:05 / 1:04

    LOWER: 1.22 / 1:45

The outlet pressure from the amplifier registers 120 psi with the regulator set only to 15 psi (improvement from previous performance as well as the upper air amplifier's performance).

We will check in on the amplifier next week to see if it still operates silently and can actuate the beam blockers/profile monitor without issue, at which point this ELOG will be updated. 

-----------------------------------------------------------------

UPDATE (Feb 26, 2024):

The lower amplifier was inspected this morning, higher volume sound could be heard from the drum in concert with the motion of the piston, but still much quieter than before servicing.

The following maintenance tasks were completed on 2024-11-29 by A. Newsome, A. Kong, A. Tam, M. Dalla Valle:

• Warm cell right window water level topup... the water level was a couple inches lower from the top. Using a tube connected to the nearby sink, the level was topped up fully. See attached before/after picture.
• Hot cell oil level checks:
  • Right side OK. See attached picture.
  • Left side very low compared to July 2022 measurement (suspect internal leak?). The oil was topped up. See attached before/after picture.
• Hot cell scissor table pump oil level check - M. Dalla Valle notes this was topped up approximately July 2022. The table was operated and functioned normally, did not sound concerning. The oil level was visible and acceptable.

Still to do, next maintenance check-up:

• Check HC atmospheric pressure differential gauge
• Lubricate HC turntable (planned for 2025 winter shutdown when target assembly not present)
• Lubricate all telemanipulators

All CUNO filters at T1 was exchanged on Jan 17.

At the time, access to the CUNO filters at T2 was obstructed by a shielding block, we will exchange these filters once the block is moved.

All Hansen fittings on the target station were inspected, all o-rings found to be in good condition.

No leaks identified in system.

The noisy T2FGSEC proteus paddle wheel flow sensor electronics was replaced.

While exchanging the sensor board, we decided to compared the wear on the paddle wheels of this sensor and another sensor (Q1, << sees less flow rate than T2FGSEC).

We found that both paddle wheels had similar levels of wear and decided to exchange both while we had the sensors taken apart.

UPDATE JAN 24:

The two CUNO filters on the T2 cooling package was serviced, all 5 spent filters transfered to MH RH HC lab for temporary storage, kept in boot box area by the hot cell tool port (attached photo only shows 4, 5th spent filter underneath rest).

It was noticed that the -235 O-ring beside T2 pump was no longer circular, new maintenance item added to checklist for all CUNO O-rings to be replaced, will be done in coming shutdown.

All rubber components on the T1 T2 water solenoid valves were exchanged on Jan 22, 2025.

To service the heat exchange water and all collimator solenoid valves, the low active copper supply line in the 1A tunnel had to be shut off (see pictures). Note that the shut off for the T2 secondary water did not work perfectly, it is possible that there either the valve was not shut off properly or there is another shutoff valve in the 1A tunnel that needed to be turned (see picture of 'T' on T2 supply). When exchanging the rubber seal and o-rings on the T2 parker heat exchanger solenoid, approximately 1-2L of active water flowed out from the valve opening onto the blocks. When returning into the 1A tunnel at the end of the valve servicing job, no dripping water or pools of water was found.

The solenoid bodies and connectors were not exchanged, but we will check that they are working properly after re-filling the systems near the end of shutdown.

The compressed air ASCO valves for the M9BB, M20BB, T1 Profile Monitor (2x 8342C1 [rebuild kit 306-191] + 1x 8320G216 [rebuild kit 314-949] respectively), and T2 Profile Monitor (8320G216) were serviced with their rebuild kits.

To service these valves, we isolated them from the compressed air source by shutting off the outlet valves on the air amplifiers and venting the compressed air through the shared inlet lines (see photo).

When reinstalling the T2 profile monitor valve, there was some difficulty turning one of the Swagelok nuts. It would be a good idea to prepare spare tube fittings to replace any that no longer seal properly.

We can re-energize the air amplifier and check for leaks in the valve in the coming days (tube of soap solution brought to red toolbox on blocks)

The actuation of the T1 profile monitor and T2 beam blockers (M9 & M20) can be checked shortly after the air amplifiers are re-energized.

When the T2 target is returned to the station, we can check the actuation of the T2 profile monitor.

--------------------------------------------------------------------------------------------------

UPDATE Feb 19:

The air amplifiers were re-started, supply valves to the T1 and T2 valves were re-opened, and the T1 profile monitor solenoid valve was tested by actuating the profile monitor.

The T1 profile monitor moved 'in' and 'out' successfully without timing out, but the control room operator noted that there was some odd status on EPICS for the T1 'harp' and that it did not correspond to the limit switch status for the profile monitors.

At T2, the M20 blocker solenoid valve does not hold pressure when supplied with air (toggle on cooling package panel turned on) and vented through the open port.

• Troubleshooting results:
  • M20 solenoid, when connected to the M9/T2 blocker solenoid electrical connector, 'opens' successfully and does not leak, causing the M20 blocker to go up.
  • M9 solenoid does not 'vent' when disconnected from the connector, successfully holds pressure in the 'rest' state.

We believe we made a mistake when re-assembling the M20 solenoid valve, this will have to be troubleshooted at a later date and possibly review the 'harp position' status on EPICS for the T1 profile monitor.

--------------------------------------------------------------------------------------------------

UPDATE Feb 24:

The cause of the 'leaky' M20 valve was identified: it was installed in the reverse orientation, small arrows were drawn on the M9 and M20 valve bodies to denote the direction of airflow to prevent future installation issues.

The air amplifier was venting through the muffler, which is not the way that it normally operated prior to being turned off. We will look at this issue at the end of shutdown when testing the beam blocker actuation.

blah blah blah.... fjda;jfeowifjoajglfa;nafg

Mike Mouat enabled recording for T1 and T2 expansion tank warning and trip level sensors on XTPAGE/XSTRIP.

XSTRIP output is in discreet integers as follows:
2 = Both sensors OK
1 = Warning level NOT OK
0 = Trip level NOT OK

Mike is now working on having a warning message show up in the control room when the sensors read NOT OK.

 Mike Mouat and Juan Pon have modified the control system so that the control room will receive a message when the warning or trip level sensors for the T1 & T2 expansion tanks read "NOT OK".  The message also displays the current water height in the tank.  This will help with earlier detection of water leaks from the cooling package or from the target ladder in the vacuum volume when the vacuum interlock has been defeated.

All sensors were checked by depressing the level sensor float and the correct warnings came up on the message reader.

 Installation of the seismic restraint clamps has been completed for the MK1 and MK2 target flasks in the Remote Handling lab (photos attached).  Two clamps on the MK1 flask required 3/8" shims due to warpage of the flask baseplate (see photo IMG_0429).

Dragan Mitrovic has been notified so that he can inspect the installation and close the work request.

Installation, wiring, and commissioning of the high temperature limit switches for the T1 and T2 cooling packages was completed today as part of SAS project SASP0120.  The relay output from the switches was wired in series with the Central Safety System signals for T1 and T2 (cable number 13250 and 12250 respectively) in break-out panel #2 on the Meson Hall south mezzanine.  There were already 3 other circuits wired in series with each signal for various other interlock requirements.

After wiring, commissioning of the switches was performed according to the following procedure:

- With the limit switches in a non-tripped state, it was confirmed that the safety signals were reaching the CSS in the main control room as expected (had to jumper T2 circuit to achieve this as the T2 target is not currently installed)
- The limit switch was tripped by adjusting the set point on the switch, it was confirmed that the signal was lost
- The set point was changed to a non-trip level, and the switch trip cleared, confirmed that the signal in MCR returned
- The enable signal from each cooling package was removed by shutting off the cooling package (for T2 just removed jumper), confirmed that safety signal lost as expected
- Reversed this condition and confirmed that the signal returned
- Checked that the thermocouples are performing properly, and the switches actually trip on high temperature by directly applying heat to the thermocouple area on the cooling package using a heat gun;  the switches tripped as expected when the set-points were exceeded

The results were discussed with John Drozdoff (Safety Systems group leader) who approved the method.

The final task remaining for SASP0120 is to route the PRV outlets to the 1A tunnel active drain using flexible hosing.

Yesterday (April 18th) the T1 and T2 cooling package PRV outlet lines were routed to the active drain in the 1A tunnel.  All items for the safety and standards compliance upgrade for T1 and T2 as specified in Document-68861 are now complete.