A call was received from the MCR at approximately 21:30 on Monday June 27th to report that active drains in the east end of the Meson Hall were backing up and overflowing.  I came in and found the leak coming from a broken spray nozzle in a sink in the rad waste area.  The leak was stopped at which point there were puddles between 1-2m diameter around the active sump top flange, the drain directly south of the warm cell water windows, and the drain in the counting room south of the clean room.  There was lots of water on the floor in the rad waste area as well.  No water was found in the TR-13 area.  A swipe was taken from the puddle around the active sump top flange, which was not above background level.  Robin Swanson (ops) contacted Joe Mildenberger who gave permission for the sump to be pumped out after a water sample was taken.  Robin took a water sample and left it for analysis by RPG, the sump pump was started and ran overnight.  At approximately 8am the next morning the sump was almost empty and the pump was stopped.  Floor swipes were taken where there had been puddles which were now dry - no contamination found.  The areas in the hot cell lab which had been wet were mopped.

 The high level sensor for the Meson Hall Hot Cell Lab Active Sump was tested today and worked as expected.  When the sensor is tilted (as occurs when the water level rises above it) an audible alarm sounds in the hot cell lab, and the Safety Panel in the MCR also alarms with "MHESA B1 RCR1 RM 6 WARN" displaying NOT OK.  This alarm sounds every 3 minutes in the MCR until the PLC outside of the RCR lab is reset.  When the sensor was tested on the RCR lab PLC there was a message indicating that the MH Hot Cell Lab Active Sump is full.

It is still unclear why the operators did not receive this alarm when the sump filled and overflowed last night (or possibly they did receive it, but it was ignored or silenced).  The operators from that shift are currently off until this Saturday, so they will be contacted next week to try to learn more.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

Further tests were done on the active sump high level float switch today.  The switch was triggered 1 time by filling the sump, and approximately 8 times by tilting the sensor manually.  A local alarm sounded in the hot cell lab each time, but only one time in the RCR Lab and MCR (this occurred on the final trial).  Fault report #9099 has been submitted to Controls Group.  Although the existing float sensor did cause a local alarm when the sump was filled, it's operation did not seem ideal:  it only triggered when completely horizontal, which wasn't until water was several inches above the base of the switch;  in addition it appeared that it may be possible for the switch to become stuck under the horizontal portion of the sump (possibly even in an untriggered orientation).  For these reasons a replacement level switch will also be investigated.

UPDATE (Sept 8, 2016):  Further testing revealed that the electronics for the high level switch were wired incorrectly which resulted in the RCR PLC only seeing sump level high when the "silence" button on the electronics box for the switch was pushed.  This was not discovered during previous tests because the "silence" button was pushed each time shortly after triggering the switch.  The new electronics for the switch will address this issue.

 On Thursday May 11th operators attempted to move the T2 profile monitor (T2-MK2) to the IN position but were unsuccessful.  The monitor would leave the OUT limit, but would not reach the IN limit.  This was repeated 2 times. The air amplifier was inspected and nothing out of the ordinary was noted.  The T1 profile monitor was tested and was working properly.  At the time it was suspected that the monitor cassette had become dislodged as occurred earlier this running period (see E-Log #221).

Today monitor actuation was attempted again and it moved to the IN position normally, this was done a total of 3 times..  An electronic check was performed by Mike Russell who reported that it looked the same as when he last checked, meaning the monitor cassette was definitely not unplugged. The monitor was then used with beam and Eric Chapman reported that monitor was functioning normally.  With the beam spot centered vertically on the profile monitor the protect monitor showed more beam on the lower plate, which is expected because the protect monitor center is 2-3mm higher than the profile monitor center (see E-Log #220).

It is unknown why it would not actuate properly last week, but seems to be fine now.  Cyclotron fault #9928 has been returned.

 The three failed T2 cooling system secondary side solenoid valves were tested yesterday to try to determine the problem.  The test steps are detailed below:

• New spare valve (NSV) produced audible "click" and valve was seen to physically open when 24V DC driving voltage was applied
• Failed valves of July 25, 2017 (J25), July 10, 2017 (J10), and November 2016 (N16) all did not click or open when driven
• The solenoid of NSV was removed from its valve and installed on each of the failed valves in turn.  The NSV solenoid succesfully opened each of J25, J10, and N16 (the valve bodies and pistons of the failed valves appear to be fine, and the solenoids all faulty)
• Each solenoid's coil resistance was measured: NSV = 187kOhm, J25 = 200kOhm, J10 = O/C, N16 = O/C

On the next maintenance day Tony Tateyama is planning on performing continuity tests on the T2CS:SVSEC control cable and comparing it to the control cable for identical solenoids in the system.

Randomly checked on the nuclear ventilation for the Meson Hall Hotcells and discovered they were not running again (both magnahelic gauges displayed zero). Attempted to restart the fan manually and the control box seems to not be working. Matt McClean has been contacted and they will look into the problem.

The nuclear ventilation is currently not running in the RHMH hot cells.

 The second (and final) 55 gallon drum containing active items from the Meson Hall East Hot Cell was transported today from the Warm Cell to the Cyclotron East Vault Tunnel.   The updated form is attached.

Diagnostics group (Shengli Liu) noted a lack of connectivity between control panel and T2 Protect Monitor via the high voltage cable. Upon inspection it was noted:

• The cable was very brittle due to radiation exposure and it was cracking in multiple places
• In the SHV connector, the pins were slightly bent (bending them back into position seemed to rectify the connectivity issue).
• The cable was bent at the strain relief junction

RH Group has advised that Diagnostics replace the entire cable as soon as possible. For now it appears to be working.

See fault 14966: https://web.accel.triumf.ca/ncr/dbfault.pl?faultgroup=CYCLOTRON&faultno=14966&btn_submit=Getfault

The flow rate sensor for T2 target cooling was exhibiting some instantaneous drops below the warning threshold, causing nuisance warnings. It was also dropping quite close to the trip limit. The reason for this unknown and this is unexpected since the sensor was recently replaced this winter shutdown.

Until the problem can be figured out and resolved, Ops has lowered both the warn and trip threshold values to 3.0 gpm and 2.8 gpm, respectively. This was done mid-day on June 7. This has been done in the past and is deemed safe to do as it is still within normal operating parameters (and a defeat is not required). The main reason for doing this is to reduce the chance of an accidental trip if the flow rate were to drop below the previous trip threshold of 3.0 gpm. These noisy blips do not seem to be dropping below 3.0 gpm, although they are close (see attached screenshot of June 6-8 data). The sensor history will be examined weekly, and if there are no drops below 3.0 gpm, these thresholds should be adjusted back to their regular values.

Some oil was noticed around the hot cell window, on the control panel, and on the ground below the control panel. A leak is suspected.

Suggest tightening all bolts to specified torque along the window, and monitoring overflow oil level (position today was marked). If the problem persists, the gasket must be replaced. Note that the gasket should likely be replaced anyway as it has not been done for some time.

The sump level was checked by Robin Swanson on Dec. 31, 2022. It was essentially empty - see attached photo.

Between 1:00 pm and 6:00 pm on July 17, 2023, the T1 target cooling water flow rate sensor exhibited some noise which resulted in it dropping below the warning/trip limit. The control room requested that the threshold values be lowered - permission was granted. The thresholds were set to 2.3 gpm for both warn and trip levels. See attached screenshot from EPICS sensor data. The occasional spikes and dips continued throughout the night fairly consistently.

While the cooling water temperature (TCOUT) fluctuates, there does not seem to be an obvious correlation between the temperature and the flow rate, which suggests that the flow rate is still adequate.

It is worth noting that overall, the FGTGT signal has been slightly trending downward since around mid-July (see attached pic).

It is suggested to replace this sensor as soon as possible - perhaps the mini shutdown.

As of 2024-04-03, the BL1A Holding Tank water level is approximately 580L. It is recommended to drain the tank prior to the start of the operating year.

The primary transfer flask was inspected on the 26th of June, 2024:

1. Electrical cabling and connections externally mounted on the flask + on the hook/latch were inspected for wear, damage, proper connections,etc.
   • It was learned that the controls+power cable that drives the latch also runs the height indicator pulley+lead weight system, this should be reworked in the future and replaced with proper load bearing rope/cable.
   • The cable connecting to the main control box was duct-taped heavily, it may be a good idea to replace it with proper reinforcement in the future.
   • The power cable connection to the grey junction box is not terminated properly and should be re-terminated (some other cables may need to be re-terminated properly).
2. Various markings on the exterior of the flask were checked:
   • Marks on the flask flange were still visible
   • Marks on the flask body for reference flask orientation were still visible
   • The weight of the flask (30,000 lbs) had faded and was written over with a sharpie (it may still be a good idea to paint a larger version of the weight so that it is visible by the crane operator)
3. Limit switches throughout the flask was inspected
   • The chain hoist limit switches were tested and both functioned properly, some wear was observed on the spring toggle for the upper limit switch but it does not need to be replaced yet.
   • The limit switches on the hook/latch were tested and functioned properly, with the exception of the 'unlatch' limit switch which we could not test because the unlatch indicator light on the control was broken (this should be re-tested in the future).
4. Rolling/rotating components (pulleys for the door and level indicator, guide wheels inside the flask,  door pin/hinge, main lifting eye, hook latch) were inspected
   • A retaining ring was missing from one of the level indicator pulleys which was promptly replaced
   • All other pins/shafts/rollers were in visually in good condition and rolled properly (despite some being rusty) and had proper retaining components (cotter pins, retaining rings, etc.).
   • Note that for future inspections, it may be a good idea to manually move the various rollers to check by hand if the pins/shafts need to be lubricated
   • Also, the lead weight driving the level indicator should be painted over, and a sacrificial contact pad should be used to prevent the load from wearing the outside of the flask
5. Electrical assemblies inside the control and grey junction boxes were inspected and found to be in good condition
   • *Except for a broken LED for the unlatch limit switch in the main control box (the broken LED issue was sourced to a broken relay that was promptly replaced, the unlatch status light shows up now)
   • All hook/latch actuating switches on the control box seemed to be working (note that the finger-related switches were not tested)
   • Note that there is an 'override' button on the control box whose purpose is unclear, we should investigate this to determine its purpose in the future
   • Note that to access the internals of the control box, only the two central screws along the white line need to be removed
6. The chain hoist and cable pulley system for flask door were inspected
   • The chain lubrication seemed dry, it would be a good idea to re-lubricate the chain properly and perform some form of maintenance on the hoist in the future
   • The chain length is retracted and dropped from a bucket and currently causes the chain to rub against a corner on top of the flask, in the future it may be a good idea to make some changes to avoid this
   • The flask door cables used duct tape to prevent the crimped cable loops from fraying and was still in good condition
   • In the future, we may want to look into the cables' channel to check for fraying or damage along their length.
7. Inspect main flask structure
   • Welds on and around the main lifting eye were in good condition without chipping paint
   • There was visible deformation on the lifting eye from contact with the hook, but this should be no cause for concern, and besides this, no signs of wear/damage was found on the lifting structure
   • The external welds on the main flask body were in good condition
   • The welds on the hook/latch that we could see were in good condition and still fully painted
   • There was some rusting on the platform
   • The welds on the ladder and platform were of lesser quality than the lifting structure but no cracks or broken welds were found
   • In the future, we should determine a weight limit for the platform based on some analysis
8. The aluminum tray was inspected and found to be in good condition, however we should replace the old rubber pieces with new ones at some point
9. The drive mechanism for the doors/hook/latch was tested and we were able to travel the full distance without abnormal sounds from the chain hoist.
10. Seismic clamps were inspected and the bolts + threaded inserts were re-tapped and greased for smoother disassembly/installation.

In addition to the items above, the four 'fingers'/actuators that are holstered by the base of the flask was inspected visually, we will confirm their function before determining whether they need regular inspection/maintenance.

Also, at some point we should inspect the target alignment frames, their camera systems, as well as hot cell flask controls systems.

Secondary flow sensor B1A:T2CS:FGSEC:RDFLOW on the T2 cooling station suddenly became very noisy at around 11 pm, 01 September 2024.

The sensor reading would fluctuate from 0-150 gpm which is beyond typical noise levels for these sensors (~2gpm).

The noise spontaneously ended at around 4 am, 05 September 2024.

The noise likely originated from some stuck debris preventing the paddle wheel from turning normally that got spontaneously dislodged.

Alternatively, changes in the environment temperature could have broken some electrical contact that recovered when the temperature cooled over the weekend. We will make a note to look into this sensor in the coming shutdown and perform preventative replacement of components.

A safety walkaround was completed for the Meson Hall Hot/Warm Cells.

The resulting spreadsheet can be found on DocuShare as Document-242733.

Main deficiencies identified:

• Hot Cells:
  • Pressure gauge reading is suspect
  • Operator station phone not working (resolved now)
• General:
  • Lifting equipment has overdue inspection

Action has been taken on all deficiencies.

  Moved Kaon Target from storage pit to remote handling hot cell using flask as training exercise.  Safety surveyor scanned storage pit area (wide flanges: clean,  pit floor: 50cpm).  Returned flask to hot cell lab floor and covered storage pit.

Replaced T1-MK1 target ladder with the new ladder design which has an integrated upper slide plate (2 less sealing locations) and no c-seal grooves.  C-seals replaced with aliuminum delta seals.Transfered all target cassettes to new ladder in the same positions.  Performed helium leak check at interface with bellows flanges and all cassette locations.  No leaks detected.

Target still requires installation of new protect monitor.  Beam-spot and elevations procedure must still be performed.

 T1-MK1 target transferred from hot cell to storage pit position #3 to free up the hot cell in preparation for shut-down activities.

Current device locations in storage pit:

Pos 1: TNF Resin Can
Pos 2:    empty
Pos 3:  T1-MK1 Target
Pos 4:  T2-MK1 Target
Pos 5:     empty 
Pos 6:     empty 
Pos 7:     empty 
Pos 8:     empty 
Pos 9:     empty 
Pos 10:   empty 
Pos 11:   empty 
Pos 12:  Liquid Deuterium Target
Pos 13:  Kaon Target
 

Drained T2 cooling package water, collected sample and delivered to RPG.  Swapped resin can, refilled, and restarted cooling package.

After 6 hours can inlet at 10.3MΩ *cm, can outlet at 10.5MΩ*cm electrical resistivity.