The new T1/T2 level sensors (Omega LVCN414) were tested in the RH HC lab today using the Omega software on a laptop and positioning the level sensor flange above a bucket of water with similar height as the expansion tank.
The level sensors behaved as expected with varying water level.

The level sensor was configured with the following parameters:

- Loop Fail-Safe: Empty
- Output at Empty: 4mA at Bottom
- Start-up condition: Empty
- Sensor Height: 50.1cm (Based on TBP1682)
- Fill Height: 45.0cm (Based on TBP1682)

This results in 4mA output at 0cm tank height, and 20mA output at 45.0cm tank height.

This test will be repeated next week with the sensor wired directly to the PLC.

The new T1/T2 level sensors (Omega LVCN414) were hooked up to the PLC and the test performed on June 11 was repeated.

The water level above the base of the bucket was measured using a tape measure, as well as with the level sensor at various intervals as the bucket was filled and then drained:

Tape Measure Level (cm) --- Reading from Ultrasonic Level Sensor (cm)

Empty --- 0
3.3 --- 3.3
6.7 --- 6.7
12.2 --- 12.2
20.5 --- 20.3
29.2 --- 28.5
36.6 --- 36.3  (Last measurement on June 17, left overnight, see graph below)
36.6 --- 35.6 (First measurement on June 18)
31.3 --- 30.2
22.0 --- 21.9
13.1 --- 12.3
7.3 --- 6.5
Empty --- 0

Data was recorded while the sensor was left overnight (see attached plot).  After approximately 2 hours, the sensor stabilized and then fluctuated less than 0.1cm for the next 12 hours.  Overall these tests proved that this sensor will be suitable for the intended application and the parameters used for configuration are appropriate.

 The following instructions were provided by Graham Waters:

From diag5pc bootup

Password: cfzx29mkdr

Bring up a terminal (click on "terminal icon" left side of screen)
> su
> xpfk34yzjc
> chmod o+rw /dev/ttyS0
> chmod o+rw /dev/ttys1

Bring up terminal emulator GTKTerm
   From booton left of screen
  Click on Applications->Accessories->GTKTerm 

 Preliminary testing of the new PLC control system and EPICS interface was completed today for the T1 target, T1 cooling package, and T1 collimator.  All systems behaved as specified.  This test did not include testing of any T1 thermocouples which will be included as part of the T2 test program to follow.  This is not considered system commissioning, which will follow after all preliminary testing is complete and the commissioning plan document has been released.

A scanned copy of the T1 test plan with notes is attached.

Preliminary testing of the new PLC control system and EPICS interface was completed today for the T2 target, T2 cooling package, T2 collimators, and T1 thermocouples.  All systems behaved as specified.

Review of the commissioning plan is currently underway.

A scanned copy of the test plan is attached.

The T1/T2 Control System "Assembly Tests"  (Document 118467 Release #1, Section 5) have been completed.  The expected result, as written in the document, occurred for every test.  The T2 Shield Thermocouple #3  (B1A:T2:TC3SHLD:RDTEMP) showed slightly erratic behaviour when viewing archived data.  It is uncertain at this point whether or not it will need to be bypassed in the B1A:T2:TGTRDY interlock.

The official commissioning tests (Section 6) are currently on-going.

 With-beam commissioning tests were completed today by Tony Tateyama and Isaac Earle.  The test results are attached in PDF format.  These tests are described in Section 7 of the Commissioning Plan (Document-118467 Release #2).  These tests were not part of the official commissioning, but intended as an additional check of system functionality with beam on, and to confirm that the CCS current protection system performs as expected.

Official commissioning of the T1/T2 EPICS system was completed in April 2016, with the results released in Document-131503.

 A new version of the T1/T2 PLC code has been uploaded to the PLC CPU.  The main change in the new version is a fix to a bug related to the target MK# which may have caused the previous CPU crashes.  The EPICS interface now needs to be updated so that no target information is displayed on the EPICS screens when there is no target installed, or jumpers for both MK1 and MK2 are detected.  This must be tested during the next maintenance day.  The appropriate tests are Section 6.4.1 Test #5 and #6 from Document #118467

 An expansion tank high level warning has been added to the T1/T2 control system under Work Request #4011.  The purpose for this new warning is to alert staff of an unexpected water increase in the unlikely scenario that a leak develops in the heat exchanger between the two fluids and CuALCW on the secondary side enters the target water system.  The new thresholds for high level warn, low level warn, and low level trip have been set at 40cm, 30cm, and 25cm respectively.  The nominal water level is 35cm.  All thresholds were tested by adding water to the tank, or removing it.  All responded as expected.

 The following changes were made to the T1/T2 PLC control system:

- De-glitching system investigated and Tony discovered that it was programmed incorrectly:  the de-glitch time was observed only when a PV changed from bad state to good, not the opposite (meaning there was actually no de-glitching in place since system start-up in 2016!).  This was reversed and the following de-glitch times were programmed:  100ms for trip thresholds except 300ms for flow meter trips, and 2s de-glitch on all warning thresholds.  The new system was tested with the PLC in simulation mode and seems to be working correctly.  This should eliminate or significantly reduce nuisance warnings and trips from the system.

- PLC system code was modified so that the target version displays '???' when no target is installed instead of defaulting to 'MK1'  (all target positions registered as 'plugged' with no target installed, so there was no danger to the system, but this was still confusing to operators)

An attempt was made to actuation the T2 profile monitor, however it showed up in EPICS as being not present. Furthermore, the "Remote" indicator light on the physical controller was not lit up despite the switch being in remote mode. The limit switch indicator light was working.

Upon troubleshooting with Erwin Klassen, it was determined that a jumper was missing on the profile monitor control connector (on the target end, rear of bulkhead). The jumper shorts pins A and G on the mil-spec connector on the controller end, which is for 120 VAC supply, to indicate that the device is connected. T1-MK1 was checked - this jumper was present on T1-MK1, across pins E and F of the mil-spec connector, but was not present on T2-MK1. Otherwise the connectors were identical. It is suspected that it was accidentally omitted during maintenance in the past. Regardless, a new jumper was soldered in-place on T2-MK1 across pins E and F to replicate T1-MK1. The system functioned normally after this.

Attached photo shows the completed solder job - the current state of the connector on the rear side of the target bulkhead panel.

 We tested the actuation of the M9/M20 beam blockers on the T2 monolith (3-4pm).

Vacuum levels remained steady throughout so the M20 o-ring replacement that was performed this shutdown was successful (see strip of 1ACG4 - vacuum gauge for T2 systems interlock).

Note: M20 was actuated by contacting the control room, while M9 (labeled T2 blocker) was actuated through the physical ASU on the ground level of the meson hall (see picture).

For future shutdown work: the air supply valve must be kept open (tab lifted up), otherwise the solenoids won't see pressurized air. 

We ran into issues because the tab on the valve broke early into the shutdown and we didn't realize that it had to stay in the open position. 

Note: The flow on the T2 demin line went up to 1.5 gpm yesterday when we were on the blocks. The needle valve was likely nudged on accident.

Update:

1) the T2 demin flow was corrected and the air supply tab was replaced - see picture M20

2) T2 BB actuated successfully with replacement tab - vacuum remained stable

3) slow leaking valve connections were tightened again

4) water level in expansion tank corrected to just below 40 cm 

The T1 motor controller hearbeat tripped (noticed this morning). 

This does not seem prevent beam delivery or target ladder actuation (see screenshot, target still ready for beam).

An improperly functioning hearbeat led on a controller may indicate that the controller is beginning to fail.

Interestingly, the profile monitor 'out' status registers a warning - the two may be related: i.e. some electrical work interrupted the two signals. 

This problem will be looked at in the coming days.

(Update) 

After consulting with Tony Tateyama from Cyclotorn Controls group, the two trips were re-set and the controller heartbeat is now 'green' again.

The motor controller is likely still healthy, seeing how it was installed only a few years back (~2015). The cause of the trip should be some electrical work on the mezzanine.

(Update - May 08, 2023)

The controller HB tripped again (noticed in the morning). Coordinated with operators to have it re-set. This was done automatically by driving the target ladder to position 2 then back to position 0.

(Update - May 16, 2023)

The controller HB on T2 tripped, and re-set by ops. 

As of 2024-04-03, the BL1A Holding Tank water level is approximately 580L. It is recommended to drain the tank prior to the start of the operating year.

The RH active sump was drained (over Jul 31-Aug 01), in total approximately 5500 L of water was sent to city sewage.

We also tried to divert water from the BL1A holding tank to the active sump, and learned that we cannot run back-flow through the sump pump. The valving configuration used here was V10, V1, SV1, V6 open - V2, V3 closed. (Note solenoid valve SV1 does not serve a purpose and energizing/powering-down does not affect the observed flow.)

In order to divert water from the holding tank to the sump, we likely need to add a 'T' after V6 to send the flow through the opening port into the sump.

The BL1A holding tank was subsequently drained and is now ready again to receive water from T1/T2/TNF.

The lower amplifier for the T2 target station air supply was removed for teardown, inspection, and rebuild. 

The goal of this operation is to understand wear development in the air amplifier over ~13 years of operation, and potentially determine a recommeded service interval.

---------------------------------------------------------------

The T2 volume was vented for an unrelated maintenance operation during this time.

During testing, prior to removing the lower amplifier, both regulators were set to ~20 psi. 

The upper air amplifier resulted in ~125 psi at the outlet while the lower air amplifier resulted in ~120 psi at the outlet.

The upper air amplifier had more audible air flowing out from the muffler than the lower amplifier. 

'Scratching' sounds in both amplifiers were comparable.

The following cycle times were recorded with the M20 BB raised/out (min:sec):

    UPPER: 1:22 / 1:00 / 2:06

    LOWER: 2:30 / 1:46 / 2:12

The following times were recorded to raise the M20 BB (sec): 

    UPPER: 8.36 / 8.76

    LOWER: 8.56 / 9.10

These times will be compared against after completing the teardown and rebuild of the lower amplifier, at which point this ELOG will be updated.

-----------------------------------------------------------------

UPDATE (Feb 23, 2024):

The lower air amplifier was serviced (photos in 'S:\Albert Kong\Shutdown Files\2024\Feb20 T2 Lower Amplifier Cleanup'):

1. Full assembly cleaning.
2. Piston and barrel was cleaned and lubricated, o-rings and plastic components replaced (with new lubricated ones).
3. Pilot valve components replaced (except plug).
4. Check valves (4x) cleaned and plastic/rubber components replaced (with new lubricated ones). 
5. Muffler cleaned (blown out with compressed air).
6. Spool and sleeve assembly cleaned, o-rings replaced (with new lubricated ones), and rubber stopper replaced.
7. Clamping rods tightened to ~17 ft-lbs. 

Note: the piston o-rings were difficult to seat on the piston body/teflon ring. During assembly, we instead seated the o-ring in the barrel on the piston plates (see picture), which allowed the oring to be seated properly, before placing the piston body onto the piston rod.

After servicing, the amplifier was returned to the station, air connections reconnected, and tested. 

Note: it is recommended to do torque-ing of the clamping rods as a final step to simplify mounting of the amplilfier and re-doing connections to the rest of the compressed air system.

The first observation we made was how silently the lower amplifier now operates when cycled: only the exhaust sound from the muffler can be heard.

Note that the piston's motion can be heard when listening ~5cm away from the amplifier barrel. 

The following times were recorded to raise the M20 BB (sec):

    UPPER: ~8.5 

    LOWER: ~7.7s

The following cycle times were recorded with the M20 BB in the out/raised position (min:sec):

    UPPER: 1:05 / 1:04

    LOWER: 1.22 / 1:45

The outlet pressure from the amplifier registers 120 psi with the regulator set only to 15 psi (improvement from previous performance as well as the upper air amplifier's performance).

We will check in on the amplifier next week to see if it still operates silently and can actuate the beam blockers/profile monitor without issue, at which point this ELOG will be updated. 

-----------------------------------------------------------------

UPDATE (Feb 26, 2024):

The lower amplifier was inspected this morning, higher volume sound could be heard from the drum in concert with the motion of the piston, but still much quieter than before servicing.

 On Wednesday November 21 at approximately 9am the level of the Remote Handling active sump was inspected and found to be at approximately the 7000L level.  The high level alarm sounds at approximately 5800L.  The high level alarm had been silenced by somebody, and RH staff had not been notified.  It is unknown when the high level alarm was reached.  The total sump volume is 7200L (which if exceeded will cause active water to spill on the lab floor).  The sump was previously inspected about 3 weeks prior, and the level was observed to be well below the high level sensor.  The recent increase in volume was from the RCR lab in the MHESA basement.

The sump contents were sampled and approved for release by RPG.  Sump contents were released on November 22nd using the procedure written in Document #64834.

A sign was put up next to the high level alarm silence button indicating that RH staff must be contacted if the alarm is silenced.

The high level sensor was tested and found to be working properly.

A notification was received that the RH lab active sump level sensor is due for annual inspection.  The sensor was tested and is working properly.

The water level was observed to be only a few inches below the sensor level.  Ken Buckley was notified and someone from his group will sample and drain the tank according to the procedure in Document-64834

A filter housing and filter were installed by Dan McDonald on November 14 in the piping section between the Remote Handling Hot Cell Lab active sump and the city sewer drain as shown in the attached picture.  The purpose of the new filter is to prevent pieces of active material from being released to the drain when the sump is pumped out.  If small pieces of active material enter the sump they may sink to the bottom, and not be captured in a water sample which is taken from the top of the sump.

The filter housing is a Waterite HP1034CLUR-KIT, and the installed filter is a 10" Cuno Microwynd 25micron filter.

This modification was approved by Curtis Ballard, Joe Mildenberg, and Grant Minor prior to installation.

Release of the Remote Handling Hot Cell Lab active sump water was completed today by Hua Yang.  During the procedure fields of up to 0.50 micro Sv/hr were measured around the newly installed filter housing.  After completion the filter was removed and replaced.  The filter measured 450cpm on the RM-14 monitor and ~5000cpm on the 44-2 monitor (both on contact).  RPG has been notified and the filter has been bagged, labeled, and left for pick-up by RPG.